Roman Cheplyaka

Рейтинги шахових розрядів і звань

Sun, 05 Apr 2026 20:00:00 +0000

Якщо ви переважно граєте в шахи онлайн, ви могли задаватися питанням, якому шаховому розряду або званню відповідає ваш рівень гри.

Отже, я провів невеличкий аналіз, щоб визначити приблизний діапазон рейтингів FIDE, який відповідає кожному розряду і званню в Україні. Оцінити свій рейтинг FIDE можна, скориставшись chessmonitor.com або chess-analysis.org.

Розряд/звання	Від (10%)	Медіана	До (90%)
III розряд	1410	1440	1470
II розряд	1430	1500	1600
I розряд	1480	1640	1890
Кандидат у майстри спорту	1820	2000	2230
Майстер спорту	1970	2190	2320
Міжнародний майстер	2080	2330	2440
Міжнародний гросмейстер	2240	2480	2600

Джерела даних:

Рейтинг-лист шахістів України на 1 березня 2026 року Федерації шахів України
Рейтинг-лист шахістів Прикарпаття до 18 років на жовтень 2025 року, зібраний Михайлом Кепещуком

Якщо ви маєте доступ до інших масивів даних (зокрема, даних по розрядах в інших регіонах України), зв’яжіться зі мною, це допоможе зробити аналіз більш повним та точним.

Більше про присвоєння спортивних звань та розрядів з шахів в Україні можна дізнатися тут: Додаток 116 до Кваліфікаційних норм та вимог Єдиної спортивної класифікації України з неолімпійських видів спорту.

Alice and Bob flipping coins puzzle

Mon, 18 Mar 2024 20:00:00 +0000

Daniel Litt on Twitter suggests a puzzle with a surprising answer:

Flip a fair coin 100 times—it gives a sequence of heads (H) and tails (T). For each HH in the sequence of flips, Alice gets a point; for each HT, Bob does, so e.g. for the sequence THHHT Alice gets 2 points and Bob gets 1 point. Who is most likely to win?
— Daniel Litt (@littmath) March 16, 2024

Like many others, I intuitively assumed the answer would be Alice, since her winning segments can overlap, while Bob’s can’t. And yet the right answer is Bob:

The correct answer is “Bob.” Congrats to the 10% who got it right — those few brave dreamers.

— Daniel Litt (@littmath) March 17, 2024

To figure out the answer, let’s represent the game as the following Markov chain:

We start at $S_1$, and every time a coin is flipped, we move to another state according to the diagram. When we reach state $S_3,$ Alice gets a point, and when we reach $S_4,$ Bob gets a point.

The transition matrix for this Markov chain is

\[ \begin{pmatrix} 0.5 & 0.5 & 0 & 0 \\ 0 & 0 & 0.5 & 0.5 \\ 0 & 0 & 0.5 & 0.5 \\ 0.5 & 0.5 & 0 & 0 \end{pmatrix} = 0.5 \cdot \begin{pmatrix} 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 1 \\ 1 & 1 & 0 & 0 \end{pmatrix}. \]

While raising this matrix to the $n$th power would allow us to compute the probabilities of ending up in different states after $n$ steps, this is not enough to keep track of Alice’s and Bob’s scores. For that, let’s multiply the transition probabilities by $x$ if Alice gets a point after a transition to that state and by $x^{-1}$ if Bob does. Our matrix is then

\[A = 0.5 \cdot \begin{pmatrix} 1 & 1 & 0 & 0 \\ 0 & 0 & x & x^{-1} \\ 0 & 0 & x & x^{-1} \\ 1 & 1 & 0 & 0 \end{pmatrix}. \]

Transition probabilities computed using such a matrix are Laurent polynomials in $x$, $\sum a_k x^k$, where $a_k$ is the probability of arriving at the given state and having Alice earn $k$ more points than Bob. So to answer the original question, we need to see whether the sum of the coefficients of $x^k$ for $k>0$ is more, less, or equal to the sum of the coefficients of $x^k$ for negative $k$ after 100 steps. Since we start at $S_1$ and can finish in any state, the polynomial of interest is the sum of the first row of $A^{100}$.

To summarize, if

\[ P_{100}(x) = \begin{pmatrix} 1 & 0 & 0 & 0 \end{pmatrix} \cdot A^{100} \cdot \begin{pmatrix} 1 \\ 1 \\ 1 \\ 1 \end{pmatrix} = \sum_{k > 0} a_k x^k + \sum_{k > 0} b_k x^{-k} + c, \]

then $\sum a_k$ is the probability of Alice’s win, $\sum b_k$ is the probability of Bob’s win, and $c$ is the probability of a tie.

For the record, $P_{100}(x)$ is:

0.5^100 (x^-50 + 1325 x^-49 + 294050 x^-48 + 26617290 x^-47 + 1330255815 x^-46 + 42919935835 x^-45 + 983156559210 x^-44 + 17033316749820 x^-43 + 233400608147735 x^-42 + 2614381239414735 x^-41 + 24550574713787010 x^-40 + 197150946519643550 x^-39 + 1375679410685414450 x^-38 + 8451073972164127108 x^-37 + 46210399961145343413 x^-36 + 227004362149489184022 x^-35 + 1009860516432421484187 x^-34 + 4096712177858121360399 x^-33 + 15247749308938974542421 x^-32 + 52350824044788156054216 x^-31 + 166607171077709190756060 x^-30 + 493645720376366009163516 x^-29 + 1367143433188619109053352 x^-28 + 3551954012109459164577376 x^-27 + 8686109884277283187997106 x^-26 + 20055225149893348816333554 x^-25 + 43844627549786945958589680 x^-24 + 91001422499460503626585260 x^-23 + 179763829541066577971612952 x^-22 + 338757354861894875655388932 x^-21 + 610315290987050592852426198 x^-20 + 1053386609896119310058220228 x^-19 + 1745118407135736558029089726 x^-18 + 2780043435628079169902430486 x^-17 + 4265870818552696769971996446 x^-16 + 6315241299298077313067374248 x^-15 + 9033468660829178932648222012 x^-14 + 12503156034451523603337781244 x^-13 + 16767474523347110492502105992 x^-12 + 21814598828347171620337357152 x^-11 + 27566098692592563661052910714 x^-10 + 33871849156423377148712374730 x^-9 + 40513231768576837533202703640 x^-8 + 47215173793461455702590625004 x^-7 + 53666145312218517883076816204 x^-6 + 59543892735365660957139789376 x^-5 + 64543704258920714616943515626 x^-4 + 68405570616760641713507980732 x^-3 + 70936793516426715075493432331 x^-2 + 72027343751505931141583428951 x^-1 + 71656412569848144755925222552 + 69889902879368773013173398330 x^1 + 66869833498941600008977020285 x^2 + 62797584077826886590138297533 x^3 + 57913469629702003919294540728 x^4 + 52475267875032276436535696432 x^5 + 46738074687809409100604254222 x^6 + 40937334350174219405131043774 x^7 + 35276210892780037700873584024 x^8 + 29917762478918476882474277364 x^9 + 24981757635389467855166986572 x^10 + 20545499627554709440186102728 x^11 + 16647734116451790331493872286 x^12 + 13294601803478278796908527364 x^13 + 10466633422619434205715329714 x^14 + 8125926604510988498320158730 x^15 + 6222846658893522578565699494 x^16 + 4701814476526100204432324608 x^17 + 3505952179066875448077681428 x^18 + 2580529813081249595335291068 x^19 + 1875284165409224711691937404 x^20 + 1345762480733919638881619816 x^21 + 953884541799647548467422492 x^22 + 667924788912881555961042316 x^23 + 462101645635764580234757556 x^24 + 315933195502895341298309504 x^25 + 213484478528509628644574020 x^26 + 142597606903299702697618140 x^27 + 94165295328566214405929076 x^28 + 61483190066811250498536216 x^29 + 39697162852338816024840079 x^30 + 25348214742043061522139419 x^31 + 16008996325093659482849398 x^32 + 10001161522302185560020758 x^33 + 6180791128771176344008963 x^34 + 3779023811996709991594019 x^35 + 2286066208917365558991896 x^36 + 1368359999967026032354752 x^37 + 810476154998045342538654 x^38 + 475041466638122452084414 x^39 + 275546274649221072591904 x^40 + 158178293697206665493492 x^41 + 89867648637382149651024 x^42 + 50533126782451168142548 x^43 + 28123845113051129416418 x^44 + 15491932054735130750508 x^45 + 8446422906616520626494 x^46 + 4558036012110548547798 x^47 + 2434549498994866786034 x^48 + 1287036656999057467424 x^49 + 673422983571685912648 x^50 + 348737888851294898280 x^51 + 178735377499734476360 x^52 + 90658265801360274016 x^53 + 45506194887963205806 x^54 + 22603387487011330494 x^55 + 11109477277752161064 x^56 + 5402634939088716388 x^57 + 2599348281853560988 x^58 + 1237186710532642568 x^59 + 582504495903896934 x^60 + 271258374586669636 x^61 + 124915774672520357 x^62 + 56888561881025689 x^63 + 25615767980815224 x^64 + 11399249487823014 x^65 + 5014926877132547 x^66 + 2180915576957059 x^67 + 936384359299208 x^68 + 397188128932528 x^69 + 166612052646818 x^70 + 68893011316114 x^71 + 28088731372888 x^72 + 11348888534124 x^73 + 4513130425164 x^74 + 1759354998368 x^75 + 683049732010 x^76 + 261506110220 x^77 + 96634341361 x^78 + 35818805501 x^79 + 13299438184 x^80 + 4606342574 x^81 + 1599386433 x^82 + 586908453 x^83 + 189264458 x^84 + 58562524 x^85 + 21967781 x^86 + 6714397 x^87 + 1655690 x^88 + 664210 x^89 + 206498 x^90 + 32856 x^91 + 14727 x^92 + 5330 x^93 + 390 x^94 + 198 x^95 + 101 x^96 + 2 x^97 + x^98 + x^99)

Tallying up the coefficients, Alice wins with probability $580127949239420834381088427404/2^{100} ≈ 0.4576$, Bob wins with probability $615866238418960422359689555420/2^{100} ≈ 0.4858$, and there is a tie with probability $71656412569848144755925222552/2^{100} ≈ 0.0565$.

Here are the outcome probabilities for different numbers of tosses (cf. the simulation results here).

The Haskell code for computing the probabilities is available here.

Sum of random cards puzzle

Sat, 23 Dec 2023 20:00:00 +0000

Here’s a nice puzzle, via Christian Lawson-Perfect and Tanya Khovanova:

There are 100 cards with integers from 1 to 100. You have three possible scenarios: you pick 18, 19, or 20 cards at random. For each scenario, you need to estimate the probability that the sum of the cards is even. You do not need to do the exact calculation; you just need to say whether the probability is less than, equal to, or more than 1/2.

Solution

For a card with the number $k$, construct the term $a(k) = 1 + (-1)^kx$, and consider the product of these terms for all the cards:

\[ A=a(1)a(2)\cdots a(100)=(1-x)^{50}(1+x)^{50}. \]

The expansion of this product consists of $2^{100}$ monomials of the form $(-1)^{k_1+k_2+\cdots+k_n}x^n$, and each such monomial corresponds to a unique combination of $n$ cards $k_1, k_2, \ldots, k_n$. The coefficient of this monomial is $+1$ when the sum of the cards $k_1+\cdots+k_n$ is even and $-1$ when it’s odd.

Therefore, once we reduce $A$ by adding up all monomials of the same degree, the coefficient in front of $x^n$ will be the number of $n$-card combinations with an even sum minus the number of $n$-card combinations with the odd sum.

And the simplified form of $A$ is

\[ A=(1-x)^{50}(1+x)^{50}=(1-x^2)^{50} = \sum_{m=0}^{50} {50 \choose m} (-1)^m x^{2m}. \]

The fact that all terms here have even powers means that for odd values of $n$, the numbers of even-sum and odd-sum $n$-card combinations are exactly equal, and so the probability to draw an even sum is exactly 1/2. And for even values of $n=2m$, the probability is greater than 1/2 when $(-1)^m > 0$, i.e. when $n$ is divisible by 4, and less than 1/2 otherwise.

Bonus: calculating the probabilities

One way to calculate these probabilities is by a recursive calculation. Here’s a Haskell function that does that.

{-# LANGUAGE LambdaCase #-}
import Data.Ratio
import Data.MemoUgly (memo)

prob_even
  :: ( Integer {- number of even cards -}
     , Integer {- number of odd cards -}
     , Integer {- number of cards to draw -}
     )
  -> Rational {- probability of drawing an even sum -}
prob_even = memo $ \case
  (_, _, 0) -> 1 -- no cards to draw; the sum is 0
  (_, 0, _) -> 1 -- all even cards
  (0, _, to_draw) -> fromInteger $ 1 - (to_draw `mod` 2) -- all odd cards
  (num_even, num_odd, to_draw) ->
    let
      prob_1st_odd = num_odd % (num_even + num_odd)
      if_1st_even = prob_even (num_even-1, num_odd, to_draw-1)
      if_1st_odd = 1 - prob_even (num_even, num_odd-1, to_draw-1)
    in
      prob_1st_odd * if_1st_odd + (1-prob_1st_odd) * if_1st_even

E.g. for $n = 20$, this gives

\[p(\text{even})=\frac{26088826721}{52177653441} = \frac{1}{2} + \frac1{104355306882}.\]

Notice how close this is to 1/2.

Alternatively, we can derive the probabilities from the proof itself. For $n = 2m$, the number of even combinations minus the number of odd combinations is $(-1)^m {50 \choose m}$, and their sum is the total number of $n$-card combinations, $100 \choose 2m$.

Then the probability of drawing an even sum is

\[p(\text{even})=\frac{\frac{1}{2}\left((-1)^m {50 \choose m} + {100 \choose 2m}\right)}{100 \choose 2m}=\frac12+(-1)^m\frac{50 \choose m}{2 {100 \choose 2m}}.\]

And sure enough, for $n=20$,

\[p(\text{even}) = \frac12+\frac{50 \choose 10}{2 {100 \choose 20}} = \frac12 + \frac1{104355306882}. \]

AeroPress Go measurements. What volume do AeroPress marks correspond to?

Sun, 23 Jul 2023 20:00:00 +0000

As a new owner of an AeroPress Go coffee brewer, I was surprised not to be able to find (either in the manufacturer’s manual or online) what the circle marks on the AeroPress chamber correspond to, or how much ground coffee fits into the included scoop. So here are my own measurements. They all assume the non-inverted brewing method with a single standard paper filter.

Chamber volume

For each of the three circle marks, I include three measurements: the bottom, the middle, and the top of the circle. I measured these both experimentally (pouring and weighing the water) and mathematically (based on the diameter of the chamber and the height of the marks).

These measurements are of the empty chamber. In reality, you will have some coffee grounds in it, so you have to account for their volume. (Here I assume that when you pour the water initially, it doesn’t mix much with the grounds, and so the combined volume should be approximately the sum of the coffee volume and the water volume. I didn’t attempt to verify this experimentally yet.)

The volume of the AeroPress Go scoop is 30 ml, so the last column in the table below contains the adjusted volume assuming you have 30 ml of grounds in the chamber. The numbers here are rounded to the nearest 5 ml.

Mark	Side	Volume of the empty chamber (ml)	Volume with 1 scoop (30 ml, ≈13 g) of coffee in the chamber (ml)
1	low	70	40
1	mid	85	55
1	high	95	65
2	low	130	100
2	mid	140	110
2	high	150	120
3	low	185	155
3	mid	200	170
3	high	210	180

Coffee grounds weight

As I mentioned above, the volume of the included scoop is 30 ml. In my measurements, I found 1 level scoop to contain around 13 g of ground coffee, more or less independent of the grind size (which was a bit surprising).

Another number for the density of ground coffee that I found online is 0.32 g/ml, resulting in 10 g per level scoop.

Brewed coffee yield

How much water should you pour to get X ml of brewed coffee? I found the output/input ratio to vary from 80% for a small portion to 90% for a large portion, assuming 1 scoop (13 g) of ground coffee and 2 minutes of brewing time.

Water (ml)	Coffee output (ml)	Ratio (%)
100	80	80
140	124	89
200	182	91

How much current does an active bass guitar draw from a 9V battery?

Sun, 14 Nov 2021 20:00:00 +0000

The 9V battery in my bass guitar tends to die rather often, and so I wanted to measure how much current the bass draws.

My bass is a Squier Jazz Bass with an active on-board preamp but passive pickups (there are only two wires going to the pickups, as shown below). Some basses have active pickups, and so their measurements may not be comparable to mine.

The electronics inside my bass

So here are my findings.

How much current does my active J-Bass preamp draw? Around 11.2mA.

Does the current remain constant over time? Once the battery is connected, the current quickly drops from 11.3–11.4mA to 11.24–11.25mA and then slowly drops further to 11.21mA over a couple of minutes. It never seems to go below 11.21mA.

Does the current increase when a string is plucked? No, it stays exactly the same regardless of whether one is playing.

Does the current depend on the potentiometer positions? No, it doesn’t depend on any of the 5 knobs: tone, volume, pickup pan (bridge vs neck mix), or bass/treble boost.

Does the current depend on the voltage supplied by the battery? Within a reasonable range, no. I tried a few batteries lying around (rechargeable and non-rechargeable) that happened to output 8.98V, 8.47V, and 8.31V, and the initial current consistently stayed at 11.24–11.25mA. When I plugged in a discharged battery that could only output 6.54V, the current jumped around 6-8mA, but the bass doesn’t sound good with such a battery.

Does it matter whether the amp is turned on? Nope, turning the amp off or even unplugging the cable from the amp does not affect the current drawn, as long as the other end of the cable stays plugged into the bass. Curiously, when I plugged in a bare TRS jack I had lying around (as opposed to TS jacks normally used in instrument cables), the current fell to 6.75mA.

Does the bass draw any current when it’s unplugged? Fortunately, it does not.

The measurement setup

The only other discussion of the active bass current draw I could find is this TalkBass thread from 2012. Even though it’s about active pickups, the author also observes the same 11mA current and complains that it’s too high (compared to 1mA that their other bass draws).

If you do similar measurements for your own bass or just happen to know more on this subject, please let me know.

A data structure puzzle

Mon, 24 May 2021 20:00:00 +0000

Jarno Alanko offered this puzzle on twitter:

Fun but difficult data structure puzzle with an elegant solution: Suppose we have an oracle to a permutation f on n elements. Describe a data structure taking O(sqrt(n)) space that can compute the inverse f^(-1)(x) for any x in O(sqrt(n)) time.
— Jarno N. Alanko (@jnalanko) May 22, 2021

Here’s my solution. It’s a programmer’s solution: it assumes that integers take $O(1)$ space and hash maps allow $O(1)$ lookups. If you know a solution that doesn’t make these assumptions, please let me know!

So, consider the decomposition of $f$ into disjoint cycles. For every cycle that has $k > \sqrt{n}$ elements, arbitrarily pick $\lceil k / \sqrt{n}\rceil$ elements—“breakpoints”—such that, when moving along the cycle, the distance between any two breakpoints is at most $\sqrt{n}$. Since there are fewer than $\sqrt{n}$ such cycles, the total number of breakpoints is

\[ \sum_i \lceil k_i / \sqrt{n}\rceil < \sum_i (k_i / \sqrt{n}+1) < 2\sqrt{n}. \]

Store all the breakpoints in a hash map $h$ that maps each breakpoint to the previous breakpoint in the same cycle. Then, to invert an element $x$, use the following algorithm:

x1 := x
x2 := f(x)
while (x2 != x):
  if h contains x1:
    x1 := h(x1)
  else:
    x1 := x2
  x2 := f(x1)
return x1

This algorithm finds the inverse element by iteratively applying $f$ until it comes back to $x$. The hash map $h$ provides a shortcut to instantaneously traverse a large part of the cycle. After less than $\sqrt{n}$ iterations, the loop either finds $x$ or finds a point $x_1$ for which $h(x_1)$ is defined. In the latter case, $x$ lies (cycle-wise) in the interval $(h(x_1), x_1)$, and once the search is restarted from $h(x_1)$, it will take less than $\sqrt{n}$ additional loop iterations to locate $x$.

Understanding modern Linux routing (and wg-quick)

Sat, 27 Feb 2021 20:00:00 +0000

Back in the old days, I could just type route (or, later, ip route) in my Linux terminal and get an accurate picture of all my routes. This is no longer the case.

For instance, the machine where I’m writing this is connected to the Mullvad VPN via the Wireguard protocol using the wg-quick script. I’m pretty sure all my traffic goes through Mullvad, yet you wouldn’t be able to tell this from my ip route output:

% ip route
default via 192.168.1.1 dev enp34s0 proto static metric 100
192.168.1.0/24 dev enp34s0 proto kernel scope link src 192.168.1.121 metric 100
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Note that the default route seemingly directs all traffic through my physical network interface, not the virtual VPN interface.

So let’s figure out how this all works.

Routing tables (`ip route`)

In reality, there isn’t the routing table in Linux (and hasn’t been for more than 20 years, since around Linux-2.2). Instead, there are multiple routing tables — and a set of rules that tell the kernel how to choose the right table for each packet.

(By the way, do not confuse routing tables with iptables. To simplify a bit, routing tables specify how to deliver a packet, whereas iptables specify whether to deliver it at all. They are completely different and unrelated.)

What you see when you run ip route without specifying a table is the contents of one particular table, main. Tables are identified by integer numbers (from 1 to 2³²−1) but can be also given textual names, which are listed in the file /etc/iproute2/rt_tables. The default one will look something like this:

#
# reserved values
#
255 local
254 main
253 default
0   unspec
#
# local
#
#1  inr.ruhep

(Are you wondering what inr.ruhep is? This is just an example, likely added by Alexey Kuznetsov, who worked on these parts of the Linux kernel and iproute tools. It stands for “Institute for Nuclear Research / Russian High Energy Physics”, the place where Alexey worked at the time, and probably refers to their internal network. There was also an old-school Russian computer network/ISP called RUHEP/Radio-MSU.)

You can view the contents of any table like this:

% ip route list table local
% ip route list table 13

Routing policies (`ip rule`)

So how does the kernel know which routing table to apply? It uses the “routing policy database”, which is managed by the ip rule command. In particular, ip rule without any arguments will print all existing rules. These are mine:

% ip rule
0:  from all lookup local
32764:  from all lookup main suppress_prefixlength 0
32765:  not from all fwmark 0xca6c lookup 51820
32766:  from all lookup main
32767:  from all lookup default

The numbers you see on the left (0, 32764, …) are rule priorities: the lower the priority, the higher the priority. That is to say, rules with lower numbers are processed first.

Apart from the priority, each rule has also a selector and an action. The selector tells us whether the rule applies to the packet at hand. If it does, the action is executed. The most common action is to consult a particular routing table (see the previous section). If that routing table contained a route for our packet, then we’re done; otherwise, we proceed to the next rule.

The rules with priorities 0, 32766 and 32767 above are created automatically by the kernel. To quote the ip-rule(8) man page:

Priority: 0, Selector: match anything, Action: lookup routing table local (ID 255). The local table is a special routing table containing high priority control routes for local and broadcast addresses.
Priority: 32766, Selector: match anything, Action: lookup routing table main (ID 254). The main table is the normal routing table containing all non-policy routes. This rule may be deleted and/or overrid‐ den with other ones by the administrator.
Priority: 32767, Selector: match anything, Action: lookup routing table default (ID 253). The default table is empty. It is reserved for some post-processing if no previous default rules selected the packet. This rule may also be deleted.

The other two rules have been created by the wg-quick script. If you want to understand how they work, read on.

Understanding wg-quick

Let’s look at the two rules that are added by wg-quick:

32764:  from all lookup main suppress_prefixlength 0
32765:  not from all fwmark 0xca6c lookup 51820

At first sight, these are quite cryptic: what does suppress_prefixlength do, what is 0xca6c, and how can a packet be “not from all”?

Rule 32764

Let’s start from the 32764 rule: as it has a lower number, it’s considered first.

32764:  from all lookup main suppress_prefixlength 0

The rule has no selector, making the kernel consult the main table for every single packet.

If this was the whole rule, every packet would be routed by the main table, never reaching the VPN. This is why the action also contains a suppressor: suppress_prefixlength 0. From the ip-rule(8) man page

suppress_prefixlength NUMBER
    reject routing decisions that have a prefix length of NUMBER or less.

Here “prefix” refers to the address or range of addresses matched in the routing table. So if you have a route for 10.2.3.4, its prefix length is 32 (bits); if you change it to 10.0.0.0/8, the prefix length will be 8.

What is a prefix of length 0 or less? It’s the empty prefix, 0.0.0.0/0, corresponding to the default route. So if the packet was routed by the default route from main, that routing decision is ignored; otherwise, it’s respected.

To summarize, the effect of this rule is to respect all manual routes that the administrator might have added to the main table. However, if the packet didn’t match any of the specific routes, then instead of applying the default route, we’re proceeding to the next rule.

Rule 32765

32765:  not from all fwmark 0xca6c lookup 51820

The “not from all” bit is just a quirk of how ip rule formats its rules. A better way to express it would be

32765:  from all not fwmark 0xca6c lookup 51820

It’s just that when no “from” prefix (address or range) is present in the rule’s selector, ip rule prints “from all”.

51820 is a routing table, also created by wg-quick, containing a single role:

% ip route list table 51820
default dev mullvad scope link

So the effect of the rule is to route everything that reached it through the VPN, with one exception: the mysterious not fwmark 0xca6c.

0xca6c is just a numerical label (“firewall mark”) that wg-quick asked wg to mark all of the packets that it emits. These are packets that already encapsulate other packets and are targeted to your VPN peer/server. If these packages were routed back to wireguard, that would create an infinite loop of wrappers on top of wrappers.

So the selector ensures that packets that have already been encapsulated can escape through your normal internet connection. Since these packets are ignored by this rule, they proceed to the rule

32766:  from all lookup main

But now there is no suppressor, so these packages are free to use the default route.

Fun fact: wg-quick uses the same numbers for the table and the fwmark: 0xca6c is just 51820 in hexadecimal.

Overall, this setup works quite well. Older VPN scripts used to override your default route in the main table when connecting to the VPN and restore it when disconnecting. Sometimes this wouldn’t work, and after disconnecting from the VPN you would be left without any default route at all. wg-quick doesn’t have this problem, as it never messes with your main routing table. All it has to do when disconnecting is delete its two rules, and your default route is active again, Or you could even do that yourself with ip rule del.

Félix Baylac Jacqué says:

If you want to go further on advanced linux-based routing:

man ip-vrf <= routing tables attached to a particular netdev.

man ip-netns <= network namespaces. There’s a nice trick using them to prevent any packet leakeage w/ wireguard https://www.wireguard.com/netns/

My podcast editing workflow in REAPER

Sun, 21 Feb 2021 20:00:00 +0000

I divide podcast post-production into two parts:

Editing: cutting and moving the audio around. This includes reducing pauses (or sometimes adding them), removing parts that didn’t turn out interesting, and cutting out some excessive filler sounds. The goal here is to make the conversation concise, engaging, and easy to listen to.
Mixing: figuring out which plugins/effects to apply so that the resulting audio sounds good.

This is not to say that editing and mixing happens in this strict order; on the contrary, while editing, I notice things that could be improved and adjust the plugin chains accordingly. But it’s useful to distinguish these two concepts.

Here is my podcast editing check-list for REAPER, which I’ll elaborate on below:

Create the project; import and align the tracks
Normalize each track to 0dB and adjust their gain
Apply dynamic split/noise gate to each track
Edit at item boundaries
Re-listen and make final edits

Create the project; import and align the tracks

To create a project, it’s convenient to use a template that contains a standard set of tracks (including the intro/outro music) and effects. Here is what mine looks like:

Normalize

When recording, I set the gain very conservatively to prevent clipping. As a result, an individual raw track may peak at as low as -15dB. So the first thing I do is normalize each track to peak at 0dB. This has little to do with the final volume of the track (which will be controlled by the volume faders/knobs), but normalizing tracks achieves a few nice things:

It makes the waveform easier to read.
When setting up the compressor and other plugins, it makes the threshold a more sane and standard level (e.g. -25dB instead of -40dB).
By default, volume faders in reaper go up to +20dB, so if raw track peaks at -15dB, the fader may not have enough gain to bring the audio to the right level.

Dynamic split

The dynamic split (or more precisely, the way I use it) acts like a noise gate; that is, its main goal is to silence the parts where the person is not talking. However, it has two big advantages over simply applying a noise gate effect:

I can visually see which parts are silenced, and adjust them by dragging the item edges.
I can easily move the non-silent parts around — whether I need to rearrange parts of the conversation or just fine-tune the pause between different speakers.

Here are my typical settings for the dynamic split.

There are some obvious settings like min slice and silence lengths and the threshold/hysteresis; and then there are somewhat less obvious settings that are worth paying attention to.

Leading and trailing pads are simply extra bits of sound that are left untouched even after the audio goes under the threshold (trailing pad) or before it emerges from below the threshold (leading pad). Without the pads, I would either have to set the threshold too conservatively, or manually adjust all the items, un-silencing the parts where a person starts or finishes talking.

You’ll also notice that I left the ‘Fade pad’ box unchecked. This is because a fade would span the whole pad, and a 300ms or even a 70ms fade is too large.

However, I do want the fades to smooth the edges of the items. So after a dynamic split, I select all the items, go to Item Properties, and manually set the fades to 0.03 (i.e. 30ms). This affects all the selected items.

Tip: make sure to unselect the items after this. If you start editing (e.g. resizing) items, all selected items will be affected, creating a big mess.

Edit at item boundaries

At this stage I just go through all the places where one person stops and another starts talking. These are the places that require the most editing, so I like to take care of them first, without having to listen to the whole recording.

Re-listen and make final edits

At this point, I listen to the full length episode, and make the final edits as needed. Sometimes I do this in REAPER, but mostly I like to render the episode and take it for a walk, writing down the timestamps where an edit is needed.

Tip: when applying edits according to a list of timestamps, move through them in the direction from the end towards the beginning of the episode. Otherwise, earlier edits will affect the timestamps that occur later in the episode.

StateT vs. IORef: a benchmark

Tue, 29 Dec 2020 20:00:00 +0000

Sometimes I’m writing an IO loop in Haskell, and I need some sort of a counter or accumulator. The two main options are to use a mutable reference (IORef) or to put a StateT transformer on top the IO monad.

I was curious, though, if there was a difference in efficiency between these two approaches. Intuitively, IORefs are dedicated heap objects, while a StateT transformer’s state becomes “just” a local variable, so StateT might optimize better. But how much of a difference does it make?

So I benchmarked the four functions, all of which calculate the sum of numbers between 1 and n = 1000.

base_sum simply calls sum from the base package; state_sum and stateT_sum maintain the accumulator using the State Int and StateT Int IO monads, respectively, and ioref_sum uses an IORef within the IO monad. And here are the results, as reported by criterion.

I’m not sure how stateT_sum manages to be faster than state_sum and base_sum (this doesn’t appear to be a statistical fluke), but what’s clear is that ioref_sum is significantly slower of them all.

So if 3ns per state access matter to you, go for StateT even when you are in IO.

(Update: also check out the comments on reddit, especially the ones by u/VincentPepper.)

Here’s the full benchmark code. It was compiled with -O2 by GHC 8.8.4 and run on AMD Ryzen 7 3700X.

import Criterion
import Criterion.Main

import Control.Monad.State
import Data.IORef

base_sum :: Int -> Int
base_sum n = sum [1 .. n]

state_sum :: Int -> Int
state_sum n = flip execState 0 $
  forM_ [1..n] $ \i ->
    modify' (+i)

stateT_sum :: Int -> IO Int
stateT_sum n = flip execStateT 0 $
  forM_ [1..n] $ \i ->
    modify' (+i)

ioref_sum :: Int -> IO Int
ioref_sum n = do
  ref <- newIORef 0
  forM_ [1..n] $ \i ->
    modifyIORef' ref (+i)
  readIORef ref

main = do
  let n = 1000
  defaultMain
    [ bench "base_sum" $ whnf base_sum n
    , bench "state_sum" $ whnf state_sum n
    , bench "stateT_sum" $ whnfAppIO stateT_sum n
    , bench "ioref_sum" $ whnfAppIO ioref_sum n
    ]

Laptop vs. desktop for compiling Haskell code

Tue, 22 Dec 2020 20:00:00 +0000

I’ve been using various laptops as daily drivers for the last 12 years, and I’ve never felt they were inadequate — until this year. There were a few things that made me put together a desktop PC last month, but a big reason was to improve my Haskell compilation experience on big projects.

So let’s test how fast Haskell code compiles on a laptop vs. a desktop.

Specs

	Laptop	Desktop
CPU	Intel Core i7-6500U	AMD Ryzen 7 3700X
Base clock	2.5 GHz	3.6 GHz
Boost clock	3.1 GHz	4.4 GHz
Number of cores	2	8
Memory speed	2133 MT/s	4000 MT/s

Methodology

I picked four Haskell packages for this test: pandoc, lens, hledger, and criterion. An individual test consists of building one of these packages or all of them together (represented here by a meta-package called all).

The build time includes the time to build all of the transitive dependencies. All sources are pre-downloaded, so just the compilation is timed.

The compilation is done using stack (current master with a custom patch), GHC 8.8.4, and the lts-16.26 Stackage snapshot, with the default flags.

The build time of each package (including the all meta-package) is measured 3 times, with all tests happening in a random order. There is a 2 minute break after each build to let the CPU cool down.

The CPU frequency governor is set to performance while compiling and to powersave during the cooling breaks.

To calculate the average level of parallelism achieved on each package, I divide the user CPU time by the wall-clock time (as reported by GNU time’s %U and %e, respectively), using the data from the desktop benchmark (as it has more potential for parallelism).

The full benchmark script is available here.

I also measured the average power drawn by both computers, both when running the benchmark and in the idle state. As my power meter only reports the instantaneous power and cumulative energy, I measured the cumulative energy (in W⋅h) at several random time points and fitted an ordinary least squares linear regression to find the average power.

Results

The first result is that I had to take the laptop outside the house (0°C) to even be able to finish this benchmark; otherwise the computer would overheat and shut down. While the laptop was outside, the CPU temperature would rise up to 74°C. The desktop, on the other hand, had no issue keeping itself cool (< 60°C) under the room temperature with only the stock coolers.

And here are the timings.

Mean compile times (minutes:seconds) and their ratio
package	desktop	laptop	ratio
lens	01:50	02:53	1.57
criterion	03:49	06:05	1.59
hledger	04:28	07:51	1.75
pandoc	14:07	22:30	1.59
all	15:20	26:48	1.75

We can also see how well the desktop/laptop speed ratio is predicted by the parallelism achieved for each package.

The average power (where averaging also includes the cooling breaks) drawn during the benchmark was 19W for the laptop and 65W for the desktop.

The average idle power was 3W for the laptop and 37W for the desktop.

Conclusions

The overheating laptop issue is real and has happened to me numerous times while working on real projects, forcing me to limit the number of threads and making the compilation even slower. This alone was worth getting a desktop PC.
There’s a decent increase in the compilation speed, but it’s not huge. The average time ratio (1.65) is much closer to the ratio of clock frequencies (1.42–1.44) than to the difference in the combined power of all cores. Also, the laptop/desktop ratio grows slowly with the level of parallelism. My interpretation of this is that the (dual-core, 4 threads) laptop is capable of exploiting most of the parallelism available when building these packages.

So the way things are today, I’d say a quad-core or probably even a dual-core CPU is enough for a Haskell developer to compile code.

That said, I hope that our build systems become better at parallelism over the coming years.
In terms of power efficiency, the laptop is a clear winner: twice as power-efficient for compilation (after adjusting for the speed difference) and 13 times as power-efficient when idle.
I also played a bit with overclocking the desktop’s CPU. I’m not an experienced overclocker and didn’t dare to go to the extreme settings, but moderate overclocking (raising the clock speed to 3.8 GHz or enabling MSI Game Boost) actually resulted in longer compile times. My understanding is that overclocking affects all cores, while CPU’s default “boosting” logic (which is disabled by overclocking) can significantly boost the clock frequency of one or two cores when needed. The latter seems to be a much better fit for a compilation workload, where most of the cores are idle most of the time.

Acknowledgments

Thanks to Félix Baylac-Jacqué for educating me about the modern PC parts.

system2 in R considered inadequate

Fri, 11 Dec 2020 20:00:00 +0000

When you need to run another program (executable) from your program, there are two different ways to go about it: to run the executable directly, or to go through the shell.

“Going through the shell” means that instead of running the executable you want to run, your program will run the shell, /bin/sh, which in turn will be told (via the -c option) to run the executable you want.

Why would one want to go through the shell? There are a couple of advantages:

When going through the shell, you can use a familiar syntax to write the arguments, e.g. system("cp -r dir1 dir2"). On the other hand, when running the executable directly, you have to split the command into the executable name (cp) and the array of arguments "-r", "dir1", "dir2".
You get for free the various shell features: input/output redirection, pipes, variable expansions etc.

But there are downsides, too:

There is a bit of overhead for launching the shell and having the shell parse the command and do other housekeeping. Often this overhead is negligible compared to all the other stuff that your program is doing, but of course it depends on what exactly it’s doing and how many times it has to run the shell/executable.
More importantly, now you have to worry about how exactly the shell will parse your command. If dir1 and dir2 are variables, you might construct your shell command as
```
paste("cp -r", dir1, dir2)
```
(R’s paste() function will by default add spaces between its arguments.)

But what if dir1 or dir2 contain spaces or other special shell characters?

For these reasons, my rule of thumb is to always run the executable directly, unless there’s a specific reason why a shell is needed, which in my experience is very rare.

Most programming languages offer both options:

The C standard defines the system function, which executes a command through the command processor (i.e. the shell on POSIX systems). On POSIX systems, one can instead execute the command directly by using fork and execvp (or another function from the exec* family).
In Python, there’s a boolean shell parameter to the subprocess.run() function that specifies whether the call should go through the shell.
In Haskell’s System.Process, there’s callCommand, which goes through the shell, and there’s callProcess, which doesn’t.
In Perl, there’s just one function, system, which does both things depending on how it’s invoked. If it gets more than one argument, it runs the executable directly, treating the first argument as the executable name and the rest as arguments for the executable.

If there’s just one argument to system, then it’s considered as a shell command.

This is still less explicit than I prefer (what if you need to run an executable without arguments, but the executable name may contains spaces or special characters?), but it’s at least something.

Now on to R. In R, there are two functions, system and system2. system takes the command as a single string, so it makes sense that it would go through the shell. system2 is described as “a more portable and flexible interface than ‘system’”, and it takes the command and the list of arguments as separate arguments:

   system2(command, args, ...)

So one might hope that it doesn’t go through the shell. Sadly, not only does it goes through the shell, it doesn’t even attempt to quote the args list so that it corresponds to the arguments actually received by the command. A single argument in args may end up as several arguments to command — or even vice versa!

> system2("printf", c("%s\\\\n" , "a b"))
a
b
> system2("printf", c("%s\\\\n" , "'", "a", "b", "'"))
 a b

Looking at the source code for system2, the argument list is simply concatenated with spaces — what’s the point of having it as a list at all?

system2 <- function(command, args = character(), ...) {
    ...
    command <- paste(c(env, shQuote(command), args), collapse = " ")
    ...
    .Internal(system(command, intern, timeout))
}

As far as I can tell, there’s currently no standard way in R to run an executable directly, bypassing the shell, which is a shame. In order to avoid issues with spaces and special characters in arguments, you have to quote each one using shQuote:

> system2("printf", c("%s\\\\n" , shQuote("a b")))
a b

Sort directories by the newest file

Fri, 04 Dec 2020 20:00:00 +0000

If you need to clean up a file system (especially your $HOME directory from some ancient dot-files), it may be useful to sort the top-level directories by the last modified time of any file inside them.

Here’s the command I came up with:

find -printf '%T@/%P\n' | awk -F/ \
  '{t[$2] = t[$2] > $1 ? t[$2] : $1}
   END {
     PROCINFO["sorted_in"] = "@val_num_asc";
     for (key in t) {
       printf "%s\t%s\n", strftime("%F", t[key]), key
     }
   }'

And here’s how the output looks like:

2007-07-30      .gkrellm2
2007-08-04      .gobby
2007-08-17      .imgseek
2007-08-25      .kderc
2007-09-12      .ftp_banner
2007-09-18      .BitchX
...

Thank you to numerous StackOverflow users from whose questions and answers I benefited; in particular the ones about printing mtime from find and sorting an AWK array.

Don't buy tempered glass PC cases

Tue, 03 Nov 2020 20:00:00 +0000

For my new desktop PC, I got the PURE BASE 500 Window Black case from “be quiet!”. On the first day of trying to use it, here’s what happened after a small bump of the edge against the floor:

It may not be obvious from the picture, but it’s not like the panel was usable in this state: it started to crumble even while I was carrying it to the trash can.

Moreover, the Ukrainian dealership of “be quiet!” doesn’t have any replacement, glass or otherwise.

Admittedly, it was a mistake to try to put on the side panel vertically — I should have laid the case on its other side, put the glass panel on top, and driven the screws while the panel was resting on the case and could not fall. Expensive lesson learned. But I certainly did not expect the glass for PC cases to be that fragile.

Here are some other shattered glass PC cases I found on reddit:

Avoid the months() and years() functions from lubridate

Sun, 30 Aug 2020 20:00:00 +0000

lubridate is an R/tidyverse package for dealing with dates and times.

One of its nice features is the date arithmetics: you can take a date, such as ymd("2020-08-30"), add an interval (such as days(10)), and get a new date ("2020-09-09").

In addition to days(), you can modify dates with months() and years(). However, I would argue you almost never want to use these two functions.

I am writing this on August 30, 2020. I’d like to do some forecasting 6 months ahead. So I would do

> today() + months(6)
[1] NA

The reason is that, strictly speaking, 6 months for now would be February 30, 2021, which does not exist.

Here’s the relevant section of the vignette:

If anyone drove a time machine, they would crash

The length of months and years change so often that doing arithmetic with them can be unintuitive. Consider a simple operation, January 31st + one month. Should the answer be

February 31st (which doesn’t exist)

March 4th (31 days after January 31), or

February 28th (assuming its not a leap year)

A basic property of arithmetic is that a + b - b = a. Only solution 1 obeys this property, but it is an invalid date. I’ve tried to make lubridate as consistent as possible by invoking the following rule if adding or subtracting a month or a year creates an invalid date, lubridate will return an NA. This is new with version 1.3.0, so if you’re an old hand with lubridate be sure to remember this!

If you thought solution 2 or 3 was more useful, no problem. You can still get those results with clever arithmetic, or by using the special %m+% and %m-% operators. %m+% and %m-% automatically roll dates back to the last day of the month, should that be necessary.
jan31 <- ymd("2013-01-31")
jan31 + months(0:11)
#>  [1] "2013-01-31" NA           "2013-03-31" NA           "2013-05-31"
#>  [6] NA           "2013-07-31" "2013-08-31" NA           "2013-10-31"
#> [11] NA           "2013-12-31"
floor_date(jan31, "month") + months(0:11) + days(31)
#>  [1] "2013-02-01" "2013-03-04" "2013-04-01" "2013-05-02" "2013-06-01"
#>  [6] "2013-07-02" "2013-08-01" "2013-09-01" "2013-10-02" "2013-11-01"
#> [11] "2013-12-02" "2014-01-01"
jan31 %m+% months(0:11)
#>  [1] "2013-01-31" "2013-02-28" "2013-03-31" "2013-04-30" "2013-05-31"
#>  [6] "2013-06-30" "2013-07-31" "2013-08-31" "2013-09-30" "2013-10-31"
#> [11] "2013-11-30" "2013-12-31"
Notice that this will only affect arithmetic with months (and arithmetic with years if your start date it Feb 29).

Now, can you think of some applications where this behavior — returning an NA — would be the desired return value of the expression today() + months(6)?

These functions are especially dangerous because they work most of the time. So when you are testing your code, you’ll likely see it working. And then once a couple of months it will fail.

So I am trying to get into the habit of writing days(6*30) instead of months(6). Alternatively, you can try to remember to use %m+% instead of +, although I’m sure you will slip up sooner or later.

The date arithmetic in lubridate is very neat and simple syntactically, and I think it was a mistake to make the most natural-looking construct fail 2% of time. Even removing those functions from the API would be better — as it would make it impossible to write the subtly-incorrect code.

Configuring PulseAudio for Audient EVO 4

Fri, 10 Jul 2020 20:00:00 +0000

For about a month now, I’ve been using the Audient EVO 4 audio interface. One of its features is “audio loopback”, where it can capture your computer audio and send it as if it were coming from your microphone.

Unfortunately, when used with its default PulseAudio configuration on Linux, this feature is turned on by default. As a result, your peers on a video call will be hearing themselves (unless an echo-cancellation algorithm of the video call service interferes).

Toon Claes ran into the same problem and published his solution. It was a great starting point for me, but it didn’t quite work in my case, so here I am publishing a rather simpler configuration that does work for me. (Note that this configuration does not support the audio loopback feature. If needed, it can be set up directly in PulseAudio by using its loopback module.)

Put the following in /etc/udev/rules.d/91-pulseaudio-custom-profiles.rules:

SUBSYSTEM=="sound", SUBSYSTEMS=="usb", ACTION=="change", KERNEL=="card*", ENV{ID_VENDOR}=="Audient", ENV{ID_MODEL}=="EVO4", ENV{PULSE_PROFILE_SET}="/etc/pulse/profile-sets/audient-evo4.conf"

Put the following in /etc/pulse/profile-sets/audient-evo4.conf:

[General]
auto-profiles = no

[Mapping mic-input]
description = Mic
device-strings = hw:%f
channel-map = mono,mono,aux0,aux1
paths-input = analog-input-mic
direction = input
priority = 1

[Mapping stereo-output]
description = Stereo output
device-strings = front:%f
channel-map = front-left,front-right,aux3,aux4
paths-output = analog-output
direction = output
priority = 1

[Profile output:stereo-output+input:mic-input]
description = Stereo Output + Mic Input
output-mappings = stereo-output
input-mappings = mic-input
priority = 100
skip-probe = yes

Put the following in ~/.config/pulse/daemon.conf:
```
remixing-use-all-sink-channels = no
```

Apply the settings:

sudo udevadm control --reload
sudo udevadm trigger --subsystem-match=sound
pulseaudio --kill

How I integrate ghcid with vim/neovim

Wed, 08 Jul 2020 20:00:00 +0000

ghcid by Neil Mitchell is a simple but robust tool to get instant error messages for your Haskell code.

For the most part, it doesn’t require any integration with your editor or IDE, which is exactly what makes it robust—if you can run ghci, you can run ghcid. There’s one feature though for which the editor and ghcid have to talk to one another: the ability to quickly jump to the location of the error.

The “official” way to integrate ghcid with neovim is the plugin. However, the plugin insists on running ghcid from within nvim, which makes the whole thing less robust. For instance, I often need to run ghci/ghcid in a different environment than my editor, like in a nix shell or a docker container.

Therefore, I use a simpler, plugin-less setup. After all, vim/nvim already have a feature to read the compiler output, called quickfix, and ghcid is able to write ghci’s output to a file. All we need is a few tweaks to make them play well together. This article describes the setup, which I’ve been happily using for 1.5 years now.

ghcid setup

ghcid passes some flags to ghci which makes its output a bit harder to parse.

Therefore, I build a modified version of ghcid, with a different default set of flags.

(There are probably ways to achieve this that do not require recompiling ghcid, but this is what I prefer—so that when I run ghcid, it simply does what I want.)

The patch you need to apply is very simple:

--- src/Ghcid.hs
+++ src/Ghcid.hs
@@ -97,7 +97,7 @@ options = cmdArgsMode $ Options
     ,restart = [] &= typ "PATH" &= help "Restart the command when the given file or directory contents change (defaults to .ghci and any .cabal file, unless when using stack or a custom command)"
     ,reload = [] &= typ "PATH" &= help "Reload when the given file or directory contents change (defaults to none)"
     ,directory = "." &= typDir &= name "C" &= help "Set the current directory"
-    ,outputfile = [] &= typFile &= name "o" &= help "File to write the full output to"
+    ,outputfile = ["quickfix"] &= typFile &= name "o" &= help "File to write the full output to"
     ,ignoreLoaded = False &= explicit &= name "ignore-loaded" &= help "Keep going if no files are loaded. Requires --reload to be set."
     ,poll = Nothing &= typ "SECONDS" &= opt "0.1" &= explicit &= name "poll" &= help "Use polling every N seconds (defaults to using notifiers)"
     ,max_messages = Nothing &= name "n" &= help "Maximum number of messages to print"
--- src/Language/Haskell/Ghcid/Util.hs
+++ src/Language/Haskell/Ghcid/Util.hs
@@ -47,7 +47,8 @@ ghciFlagsRequiredVersioned =
 -- | Flags that make ghcid work better and are supported on all GHC versions
 ghciFlagsUseful :: [String]
 ghciFlagsUseful =
-    ["-ferror-spans" -- see #148
+    ["-fno-error-spans"
+    ,"-fno-diagnostics-show-caret"
     ,"-j" -- see #153, GHC 7.8 and above, but that's all I support anyway
     ]

Alternatively, you can clone my fork of ghcid at https://github.com/UnkindPartition/ghcid, which already contains the patch.

Apart from changing the default flags passed to ghci, it also tells ghcid to write the ghci output to the file called quickfix by default, so that you don’t have to write -o quickfix on the command line every time.

vim/neovim setup

Here are the vim pieces that you’ll need to put into your .vimrc or init.vim. First, set the errorformat option to tell vim how to parse ghci’s error messages:

set errorformat=%C%*\\s•\ %m,
               \%-C\ %.%#,
               \%A%f:%l:%c:\ %t%.%#

Don’t ask me how it works—it’s been a long time since I wrote it—but it works.

Next, I prefer to define a few keybindings that make quickfix’ing easier:

map <F5> :cfile quickfix<CR>
map <C-j> :cnext<CR>
map <C-k> :cprevious<CR>

When I see any errors in the ghcid window, I press F5 to load them into vim and jump to the first error. Then, if I need to, I use Ctrl-j and Ctrl-k to jump between different errors.

Which dynamic mic should you buy?

Sat, 13 Jun 2020 20:00:00 +0000

Shawn Milochik has published a cool blind test of dynamic microphones. Go take it yourself and choose which ones you prefer.

But also, if you don’t have time for that or would like to know what I think, here’s my rating of Shawn’s dynamic mics based on his recordings.

Favorite

Shure SM7B ($400)

Very good

Samson Q2U ($60)
Electro-Voice RE20 ($450)

Good

Behringer XM8500 ($23)
Sennheiser MD 441U ($900)
Rode Procaster ($230)

Least favorite

Sennheiser MD 421-II ($380)
Heil PR40 ($330)
Sennheiser e 835-S ($100)
Rode PodMic ($100)

I’m not publishing the correspondence between the microphone models and Shawn’s codenames so that you can still take the test. I added the retail prices of these microphones in USD (mostly based on Sweetwater) for a quick reference.

The usual disclaimer is that the way a mic sounds depends on the voice, but I would think these results are generalizable at least to some extent. Shure SM7B once again has proven its dominance, and Samson Q2U its value for money.

Visualizing Haskell heap profiles in 2020

Thu, 14 May 2020 20:00:00 +0000

Heap profiling is a feature of the Glasgow Haskell Compiler (GHC) that lets a program record its own memory usage by type, module, cost center, or other attribute, and write it to a program.hp file.

Here I review the existing tools—and introduce a new one—for visualizing and analyzing these profiles.

hp2ps

hp2ps is the standard heap profile visualizer, as it comes bundled with GHC.

Run it as

hp2ps -c benchmark.hp

(where -c makes the output colored), and it will produce the file benchmark.ps, which you can open with many document viewers.

Here’s what the output looks like:

The example shows a heap profile by the cost center stack that allocated the data. As I mentioned, there are many other types of heap profiles, but this is what I’ll be using here as an example.

As you see, the cost centers on the right are truncated. I usually like to see them longer. They are actually truncated by the profiled program itself, not by the visualizer, so to get longer profiles, rerun your program with +RTS -hc -L500 to increase the maximum length from the default 25 to, say, 500.

However, hp2ps doesn’t deal well with long cost center stacks (or other long identifiers) by default: the whole page would be filled with identifiers, and there would be no room left for the graph itself. To work around that, pass -M to hp2ps. It produces a two-page .ps file, with the legend on the first page and the graph on the second one.

I found that viewers like Okular and Evince only display the second page of the two-page .ps file, but it works if you first convert the output to pdf with ps2pdf. Here’s what the output looks like:

Two-page output from hp2ps -M

hp2pretty

hp2pretty by Claude Heiland-Allen has a few advantages over hp2ps: a nicer output with transparency and grid lines, truncation of long cost center stacks, and the ability to write the full cost center stacks to a file using a --key option.

Run it simply as

hp2pretty benchmark.hp

and it will produce a file named benchmark.svg.

hp/D3.js

hp/D3.js by Edward Z. Yang is an online tool to visualize Haskell heap profiles. There’s a hosted version at heap.ezyang.com, and there is the source code on GitHub.

I wasn’t able to build the source code due to the dependency on hp2any (see below), but the hosted version still works. The disadvantage of the hosted version is that you have to upload your heap profile to the server, and it becomes public—consider this when working on proprietary projects. (The profile files do not contain any source code, but even the function names and call stacks may reveal too much information in some cases.)

hp/D3.js offers a choice of three different styles of pretty graphs shown below. You can also browse this profile yourself. There are some cool interactive features, like the entry’s name or call stack being highlighted when you hover the corresponding part of the graph.

Perl & R

Sometimes a quick look at the heap profile graph is all you need to understand what to do next. Other times, a more detailed analysis is required. In such cases, my favorite way is to convert an .hp file to csv and load it into R.

To convert an .hp file to csv, I wrote a short Perl script, hp2csv. (Unlike many tools written in Haskell, there’s a good chance it’ll continue working in 10 years.) Put it somewhere in your PATH, make it executable (chmod +x ~/bin/hp2csv), and run

hp2csv benchmark.hp > benchmark.csv

The CSV has a simple format:

time,name,value
0.094997,(487)getElements/CAF:getElements,40
0.094997,(415)CAF:$cfoldl'_r3hK,32
0.094997,(412)CAF:$ctoList_r3hH,32
0.094997,(482)match/main/Main.CAF,24
0.094997,(480)main/Main.CAF,32

where time is the time in seconds since the program start, name is the name of the cost center/type/etc. (depending on what kind of heap profiling you did), and value is the number of bytes.

Now let’s load this into R and try to reproduce the above graphs using ggplot.

library(tidyverse)
library(scales) # for a somewhat better color scheme

csv <- read_csv("benchmark.csv") %>%
  # convert bytes to megabytes
  mutate(value = value / 1e6) %>%
  # absent measurements are 0s
  complete(time,name, fill = list(value = 0))
  

# find top 15 entries and sort them
top_names <- csv %>%
  group_by(name) %>%
  summarize(sum_value = sum(value)) %>%
  arrange(desc(sum_value)) %>%
  head(n=15) %>%
  mutate(name_sorted = str_trunc(name,30),
         name_sorted = factor(name_sorted, levels=name_sorted))
top_entries <-
  inner_join(csv, top_names, by="name")

# Create a custom color palette based on the 'viridis' palette.
# Use 'sample' to shuffle the colors,
# so that adjacent areas are not similarly colored.
colors <- function(n) {
  set.seed(2020)
  sample(viridis_pal(option="A",alpha=0.7)(n))
}

theme_set(theme_bw())
ggplot(top_entries,aes(time,value,fill=name_sorted)) +
  geom_area(position="stack") +
  discrete_scale(aesthetics = "fill",
                 scale_name = "viridis modified",
                 palette = colors) +
  scale_y_continuous(breaks=function(limits) seq(0, floor(limits[[2]]), by=10)) +
  labs(x="seconds", y="MB", fill = "Cost center")

But these stacked plots are not always the best way to represent the data. Let’s see what happens if we try a simple line plot.

top_entries %>%
  ggplot(aes(time,value,color=name_sorted)) +
    geom_line() +
    scale_y_continuous(breaks=function(limits) seq(0, floor(limits[[2]]), by=5)) +
    labs(x="seconds", y="MB", color = "Cost center")

This looks weird, doesn’t it? Do those lines merge, or does one of them just disappear?

To disentangle this graph a bit, we can add a random offset for each cost-center.

set.seed(2020)
top_entries %>%
  group_by(name) %>%
  mutate(value = value + runif(1,0,3)) %>%
  ungroup %>%
  ggplot(aes(time,value,color=name_sorted)) +
    geom_line() +
    scale_y_continuous(breaks=function(limits) seq(0, floor(limits[[2]]), by=5)) +
    labs(x="seconds", y="MB", color = "Cost center")

So it’s not a glitch, and indeed several cost centers have identical dynamics. It’s not hard to imagine why this could happen: think about tuples whose elements occupy the same amount of space but are produced by different cost centers. As these tuples are consumed and garbage-collected, the corresponding lines remain in perfect sync. But this effect wasn’t obvious at all from the stacked plot, was it?

Another thing that is hard to understand from a stacked plot is how different cost centers compare, say, in terms of their maximum resident size. But in R, we can easily visualize this with a simple bar plot:

top_entries <- csv %>%
  group_by(name) %>%
  summarize(max_value = max(value)) %>%
  filter(max_value >= 1) %>%
  arrange(max_value) %>%
  mutate(name = str_trunc(name, 120), name = factor(name, levels=name))
ggplot(top_entries, aes(name,max_value)) + geom_col(fill=viridis_pal(alpha=0.7)(5)[[4]]) +
  geom_text(aes(name,label=name),y=0,hjust="left") +
  labs(x="Cost center", y="Memory, MB") +
  scale_x_discrete(breaks=NULL) +
  scale_y_continuous(breaks=function(limits) seq(0, floor(limits[[2]]), by=5)) +
  coord_flip()

Finally, in R you are not limited to just visualization; you can do all sorts of data analyses. For instance, a few years back I needed to verify that, in a server process, a certain function was not consuming increasingly more memory over time. I used this technique to load the heap profile into R and verify that with more confidence that I would have had from looking at a stacked graph.

hp2any

One issue with big Haskell projects is that, if not actively maintained, they tend to bitrot due to the changes in the compiler, the Haskell dependencies or even the C dependencies.

One such example is Patai Gergely’s hp2any. It no longer builds with the current version of the network package because of some API changes. But even when I tried to build it with the included stack.yaml file, I got

glib          > Linking /tmp/stack214336/glib-0.13.6.0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.2.0.1/setup/setup ...
glib          > Configuring glib-0.13.6.0...
glib          > build
glib          > Preprocessing library for glib-0.13.6.0..
glib          > setup: Error in C header file.
glib          >   
glib          > /usr/include/glib-2.0/glib/gspawn.h:76: (column 22) [FATAL] 
glib          >   >>> Syntax error!
glib          >   The symbol `__attribute__' does not fit here.
glib          >

I’m guessing (only guessing) that this issue is fixed in the latest versions of the glib Haskell package, but we can’t benefit from that when using an old stack.yaml. This also shows a flaw in some people’s argument that if you put prospective upper bounds on your Haskell dependencies, your projects will build forever.

(At this point, someone will surely mention nix and how it would’ve helped here. It probably would, but as an owner of a 50GB /nix directory, I’m not so enthusiastic about adding another 5GB there consisting of old OpenGL and GTK libraries just to get a heap profile visualizer.)

Compile and link a Haskell package against a local C library

Tue, 07 Apr 2020 20:00:00 +0000

Let’s say you want to build a Haskell package with a locally built version of a C library for testing/debugging purposes. Doing this is easy once you know the right option names, but finding this information took me some time, so I’m recording it here for the reference.

Let’s say the headers of your local library are in /home/user/src/mylib/include and the library files (*.so or *.a) are in /home/user/src/mylib/lib. Then you can put the following into your stack.yaml (tested with stack v2.2.0; instructions for cabal-install should be similar):

extra-include-dirs:
  - /home/user/src/mylib/include
extra-lib-dirs:
  - /home/user/src/mylib/lib
ghc-options:
  "$locals": -optl=-Wl,-rpath,/home/user/src/mylib/lib

Here "$locals" means “apply the options to all local packages”.

Binomial coefficients via integer division

Sun, 22 Mar 2020 20:00:00 +0000

Daniel Lemire explains how to compute binomial coefficients using only integer multiplications. The binomial coefficient $n \choose k$ is defined as

\[ {n \choose k} = \frac{n!}{k! (n-k)!} = \frac{n(n-1)\cdots(n-k+1)}{1\cdot 2\cdots k}. \]

Daniel’s strategy of computing binomial coefficients consists of two steps.

First, he expresses the binomial coefficient as an iterative computation

\[ \begin{align} a_1 &= n - k + 1 \\ a_z &= \frac{a_{z-1} (n-k+z)}{z}\label{ii}\hspace{2em}(z=2,\ldots,k)\\ {n \choose k} &= a_k\label{iii} \end{align} \]

Then he replaces integer division in $\eqref{ii}$ with integer multiplication, following the Granlund-Montgomery-Warren algorithm.

What wasn’t immediately obvious to me is why we can treat the division in $\eqref{ii}$ as integer division in the first place. Indeed, the equality ${n \choose k} = a_k$ obviously holds if we treat division in $\eqref{ii}$ as exact rational division. But what happens if we replace it with integer division, throwing away the fractional part at each iteration? Then we might potentially arrive at the wrong answer at the end.

However, it turns out that there never is a fractional part in $\eqref{ii}$: all $a_k$ are integers even if we interpret the division as a proper (not truncated or rounded) division of rational numbers.

To see that, expand the definition of $a_z$:

\[ \begin{equation} \begin{aligned} a_z &= \frac{a_{z-1} (n-k+z)}{z}\\ &= \frac{a_{z-2} (n-k+z)(n-k+z-1)}{z(z-1)}\\ &= \ldots \\ &= \frac{(n-k+z)(n-k+z-1)\cdots (n-k+1)}{z(z-1)\cdots 1} \\ &= {n-k+z \choose z}. \end{aligned}\label{expanded} \end{equation} \]

It turns out that the thing computed at each step of Daniel’s iterative algorithm is itself a binomial coefficient, and is therefore an integer. Notice how $\eqref{expanded}$ reduces to $\eqref{iii}$ if we let $z = k$.

Equation $\eqref{ii}$ can also be written as

\[ {n-k+z \choose z} = {n-k+z-1 \choose z-1}\cdot \frac{n-k+z}z, \]

which is an instance of the general identity

\[ {p \choose q} = {p-1\choose q-1}\cdot \frac{p}{q}. \]

Does micro-USB cable affect charging speed?

Sun, 08 Mar 2020 20:00:00 +0000

How much does a micro-USB cable affect the charging speed of your phone? Should you pay extra for a premium-quality cable? To find this out, I ran a small experiment.

For this experiment, I tried all combinations of four micro-USB cables I had lying around, two USB chargers, and three devices to be charged: a Sony smartphone running Android, a Samsung tablet also running Android, and a Mimacro power bank. As a proxy for charging speed, I used the power drawn by the charger measured by a Kill A Watt-like electricity monitor. To avoid batch effects, I tested combinations in a pre-determined random order.

It turns out that I had one particularly bad cable (“Cable 1”) that was 3-4 times less efficient at charging:

I do not remember how I got Cable 1—whether I bought it separately or it was included with some gadget—but superficially it looks better built than some of the other cables, so I was surprised by this result.

Other than that one cable, all cables were performing similarly, drawing around 9–11 watts, which at 5V corresponds to the current of 1.8–2.2A.

There also were no practically significant differences between different devices or chargers as the graphs below show (where Cable 1 measurements are excluded):

None of my devices are new, and none of my cables are particularly high-end, so your mileage may vary. But my conclusions from this are:

Do not buy the cheapest cable or charger, but it’s not worth investing extra either.
Do a quick test to see whether the cable/charger is drawing around 10W or 5A of electricity—do not rely on the “high-quality” look of it.

Break on NaN in gdb

Sat, 11 Jan 2020 20:00:00 +0000

Recently I had to debug a case where, somewhere during a numerically-intensive computation (solving an ordinary differential equation), a value would become NaN (“not a number”). This happens, for example, when taking a logarithm or a square root of a negative number, dividing 0 by 0, stuff like that. However, I had no idea where and why NaNs appeared in this particular program.

So here I’ll show how to detect this using gdb, the GNU debugger. Here is the program that we will be debugging:

/* nan.c */
#include <math.h>
#include <stdio.h>
#include <stdlib.h>

void f1(double *y) {
  *y -= 2.0;
}

void f2(double *z) {
  *z = sqrt(*z);
  *z += 1.0;
}

int main() {
  double *x = malloc(sizeof(double));
  *x = 5e4;
  int i;
  for (i = 0; i < 1000; i++) {
    f1(x);
    f2(x);
  }
  printf("%f\n", *x);
  free(x);
}

Compile it with

gcc -g -lm -Wall -pedantic -onan nan.c

and confirm that it produces a NaN:

% ./nan
-nan

Now, start gdb and proceed to the point where the storage for our double value is allocated:

Reading symbols from ./nan...
(gdb) start
Temporary breakpoint 1 at 0x4011cd: file nan.c, line 15.

Temporary breakpoint 1, main () at nan.c:15
15    double *x = malloc(sizeof(double));
(gdb) next
16    *x = 5e4;

At this point we can do two useful things.

First, we turn on the program execution log, so that we can go back in time once we encounter a NaN.

(gdb) set record full stop-at-limit off
(gdb) record full

Second, we set a watchpoint that will trigger once *x becomes NaN. But how do we express that?

If we were programming in C, we would use the isnan() function. But isnan() is not a C function, it is a preprocessor macro and is by default not available in gdb. And even if I make it available (by compiling the program with -g3), it still doesn’t work, at least on my system:

(gdb) macro expand isnan(1.0)
expands to: __builtin_isnan (1.0)
(gdb) print isnan(1.0)
No symbol "__builtin_isnan" in current context.

Luckily, there are a few workarounds. Perhaps the simplest and the most reliable one is to exploit the fact that NaN is the only floating-point value not equal to itself. Therefore we can set a conditional watchpoint like this:

(gdb) watch *x if *x != *x
Hardware watchpoint 2: *x

Some other options are:

Figure out the underlying C function used to implement the isnan macro. On my system, this seems to work:
```
(gdb) p ((int (*)(double))__isnan)(sqrt(-1.0))
$1 = 1
(gdb) p ((int (*)(double))__isnan)(sqrt(1.0))
$2 = 0
```
However, it may differ on your system/compiler/standard library.
Wrap the standard isnan macro in a C function in your program to make it available inside gdb:
```
int myisnan(double x) {
  return isnan(x);
}
```
(Naming your wrapper function isnan might work too. In particular, it doesn’t necessarily lead to an infinite recursion because the inner isnan call will be expanded by the preprocessor. However, the C standard explicitly says (section 7.1.3) that the standard macro names are reserved identifiers, and redefining these identifiers results in undefined behavior.)
You can try to inspect the bit pattern of the floating-point number directly. See e.g. this answer by Paul Pluzhnikov. Note, however, that IEEE 754 NaNs do not have a fixed bit pattern as Paul appears to assume (they may have an arbitrary fraction part), so unless you know exactly what NaN you are expecting, you have to be more careful.

In any case, once we’ve started recording the execution log and set a watchpoint, we are ready to restart the program and wait for the condition to trigger:

(gdb) continue
Continuing.

Hardware watchpoint 2: *x

Old value = -0.19219211672498604
New value = -nan(0x8000000000000)
f2 (z=0x4052a0) at nan.c:11
11    *z += 1.0;

Now we know that NaN was produced in f2 right before line 11. If this is not enough to diagnose the bug, we can use the execution log to go back in time. Let’s say we want to find out what the value of *x was before the last f1 call.

(gdb) tbreak f1
Temporary breakpoint 3 at 0x40115e: file nan.c, line 6.
(gdb) reverse-continue
Continuing.

Temporary breakpoint 3, f1 (y=0x4052a0) at nan.c:6
6     *y -= 2.0;
(gdb) print *y
$1 = 1.807807883275014

Significance level vs. the type I error chance, or how to interpret conditionals

Sat, 21 Dec 2019 20:00:00 +0000

One of the 12 p-value misconceptions states:

With a $P = .05$ threshold for significance, the chance of a type I error will be 5%.

I find this to be a particularly counter-intuitive one. Recall that the type I error occurs when we reject the null hypothesis if the null hypothesis is in fact true. Therefore, the chance of committing a type I error is $p(\text{reject } H_0 | H_0\text{ is true})$—which is exactly the definition of the significance level!

The confusion here stems from the ill-defined mapping from the English language to the language of the probability theory. Consider these semantically equivalent ways to state when a type I error occurs:

Type I error occurs when we reject $H_0$, given that $H_0$ is true.
Type I error occurs when $H_0$ is true, given that we have rejected it.
Type I error occurs when $H_0$ is true and we reject it.

If we try to map these descriptions to their probabilities by replacing “given” with the conditional probability “|”, we will get three distinct probabilities:

$p(\text{reject } H_0 | H_0\text{ is true})$
$p(H_0\text{ is true} | \text{reject } H_0)$
$p(H_0\text{ is true} \wedge \text{reject } H_0)$, where $\wedge$ means the logical “and”.

However, in this particular case there is no ambiguity. This is because a “type I error” is an event (whose probability is the subject of the misconception), but $\text{reject } H_0 | H_0\text{ is true}$ and $H_0\text{ is true} | \text{reject } H_0$ are not events. (You can find a proof that “|” cannot be an operator constructing new propositions/events in Richard Jeffrey’s “Subjective Probability: The Real Thing”, section 1.5.)

Therefore, the only possible interpretation is the third one.

Here’s another scenario where interpreting conditionals may be ambiguous. You say “I bet you that if Bernie Sanders becomes the Democratic candidate in the 2020 election, he’ll defeat his opponent”. I take the bet, and we each put $10 on the respective outcomes. Now imagine Sanders does not win the Democratic primaries. Who wins the $10?

One way to resolve this is to say that our whole bet was conditional on Sanders becoming the Democratic nominee in 2020. Since he didn’t, the bet is called off and no one pays. Under this interpretation, the probability of you winning the bet is

\[p(\text{Sanders becomes president } | \text{ Sanders is nominated for president}).\]

The fact that we had to recall the bet remidns us that this a conditional probability, and is not an unconditional probability of the “event”

\[\text{Sanders becomes president } | \text{ Sanders is nominated for president}.\]

Alternatively, you could say you were proven correct. Your statement essentially was

\[\text{Sanders is nominated for president } \Rightarrow \text{ Sanders becomes president}.\]

Since the antecedent is false (Sanders was not nominated), the rules of logic say that the whole implication is true. And since I took the other side of the bet, essentially betting on you being wrong, I lost.

The probability of you winning the bet under such an interpretation is

\[p(\neg\text{(Sanders is nominated for president) } \vee \text{ Sanders becomes president}),\]

(where $\neg$ and $\vee$ mean the logical “not” and “or”, respectively) because only in the case when Sanders is nominated but loses the election can I claim that your statement was wrong.

Bioinformatics events in Europe in 2020

Sun, 15 Dec 2019 20:00:00 +0000

Here are some conferences and summer schools that will take place in Europe in 2020 and may be of interest to fellow bioinformaticians.

Previous editions: 2018, 2019.

Event	Dates	Location	Early registration price	Early registration deadline
Data Structures in Bioinformatics	February 3–5	Rennes, France	Free	January 20
Woodstock.Bio	February 13–14	Tel-Aviv, Israel	Free
Applied Bioinformatics in Life Sciences	February 13–14	Leuven, Belgium	€500	December 20, 2019
Manchester’s Molecular and Genome Evolution Symposium	March 20	Manchester, UK	Free
RECOMB	May 10–13	Padova, Italy	€855	March 3
London Calling	May 20–22	London, UK	£699 + 20% VAT	February 28
CENTURI Summer School: From data to biology and back	June 16–25	Marseille, France	€75	May 4
#NGSchool2020: Statistical Learning in Genomics	July 22–31	Białobrzegi, Poland		April 17
NGSymposium in Computational Biology	July 31–August 1	Warsaw, Poland	€75	March 27
Summer course in analyses of genotyping and NGS data in medical and population genetics	August 3–7	Copenhagen, Denmark	$200	March 30
ECCB	September 5–9	Sitges, Barcelona, Spain	€775	March 16
Workshop on Algorithms in Bioinformatics (WABI)	September 7–9	Pisa, Italy

Long-term reminders

Fri, 22 Nov 2019 20:00:00 +0000

By long-term reminders I mean things that are not actionable right now, but at some point in the future become important to take care of: cancel a subscription or a credit card, renew a passport/visa/driver’s license etc.

At one point I used nudgemail for this. However, I ran into a pretty serious date parsing/scheduling bug: I had two reminders, one for October 6 and one for October 20, and both arrived on October 1. I never received a response to the email I sent to support@nudgemail.com, so I don’t consider them reliable anymore.

That got me thinking: what is a reliable way to send a message to the future me?

Email reminders seem perfect for this, if only there existed a service that has proved to be dependable (I’m not aware of one).
Telegram allows to schedule messages in the future and send them to oneself. But since telegram is tied to my phone number, there is a high chance that in a year or two I’ll have a different phone number and will lose my reminder.
There are probably many mobile apps for this, but then I need to remember to transfer them to any new phone I may have.
I considered writing or installing an existing self-hosted email scheduling solution, but it would become a maintenance burden.
I also considered doing it UNIX-way, using atd and mail, but:
- What if I forget to transfer my atd files to the next system/device?
- How to ensure reliable delivery of mail from localhost? (I don’t set up smtpd/smarthosts these days.)

Here’s what I decided in the end, inspired by the birthday program:

I write my reminders in a simple text file structured like this:
```
YYYY-MM-DD<tab>description
```
For instance:
```
2021-01-03   Renew ro-che.info
```
I wrote a small Haskell program that parses the file and prints any reminders that are due.
I added remind ~/my/reminders.txt to ~/.zshrc.

I figured that in several years I’ll still be using terminals and zsh, and I tend to back up and transfer these files to new systems, so this is one of the most reliable way to send a message to the future for me.

A blind microphone comparison

Wed, 04 Sep 2019 20:00:00 +0000

Like many other podcasters, I like obsessing over different microphones and watching mic reviews. Unfortunately, most of these reviews are not blind, and so your impression is inevitably affected by how the microphone looks, what its price is, opinions you’ve heard about the mic etc.

So here’s a blind test of the three microphones that I currently own. Feel free to tell me both your subjective (which one you liked more) and objective (what type/price range/model you think these are) thoughts, and I’ll tell you what these microphones are.

Mic 1 Mic 2 Mic 3

Text credit: Daniel T. Baldassarre.

Here are some other blind microphone tests you may enjoy:

Masked mics series by Allen Williams
8-Mic Shootout With David Hooper by Chris Curran

A curious associativity of the <$> operator

Mon, 22 Jul 2019 20:00:00 +0000

The <$> operator in Haskell is an infix synonym for fmap and has the type

(<$>) :: Functor f => (a -> b) -> f a -> f b

For example:

> negate <$> Just 5
Just (-5)

Like all other operators, <$> has a fixity, which can be infixl, infixr, or infix. The fixity defines how an expression like

abs <$> negate <$> Just 5

is parsed.

If <$> were defined as infix, then the above expression wouldn’t parse at all, giving an error like

Precedence parsing error
    cannot mix ‘<$>’ [infix 4] and ‘<$>’ [infix 4] in the same infix expression

If <$> instead were defined as infixr, then the above expression would parse as

abs <$> (negate <$> Just 5)

meaning first apply negate to 5 inside Just, and then go inside Just once more and apply abs.

However, <$> is defined as infixl, so that the applicative chains like

(+) <$> Just 5 <*> Just 6

would parse correctly. This means that

abs <$> negate <$> Just 5

is parsed as

(abs <$> negate) <$> Just 5

which is probably not what you meant. Or is it?

It turns out that if you paste that expression (either parenthesized or not) into ghci, you’ll get back Just 5 in all cases.

This happens because of the following instance defined in the base library:

instance Functor ((->) r) where
  fmap = (.)

(Note that (->) r is essentially the Reader monad, so it’s not surprising that it’s a functor.)

So the expression

(abs <$> negate) <$> Just 5

does make sense and is equivalent to

(abs . negate) <$> Just 5

The operators * for which a * (b * c) = (a * b) * c are called associative. Usually associative operators have the type a -> a -> a (either polymorphic of for a specific a), so that both ways to put parentheses typecheck.

The <$> operator is curious because it doesn’t have such a type and yet is associative:

f <$> (g <$> h) =
{- the definition of <$> -}
fmap f (fmap g h) =
{- the definition of (.) -}
(fmap f . fmap g) h =
{- the functor law -}
fmap (f . g) h =
{- the definition of <$> -}
(f . g) <$> h =
{- the Functor instance for (->) r -}
{- g has to be a function for the top expression to typecheck -}
(f <$> g) <$> h

Decompose ContT

Fri, 19 Jul 2019 20:00:00 +0000

Can we decompose a ContT action into separate acquire and release functions?

Motivation

Consider some resource, such as a file handle, socket, database connection etc. The two actions common to all resources are acquiring and releasing.

There are two ways to represent these actions: as a pair of functions

acquireResource :: IO Resource
releaseResource :: Resource -> IO ()

or as a single function

withResource :: (Resource -> IO a) -> IO a

As I explained in Why would you use ContT?, the latter can be nicely wrapped into a continuation monad:

getResource :: ContT r IO Resource
getResource = ContT withResource

We can go from an (acquireResource, releaseResource) pair to the ContT version by using bracket from Control.Exception:

getResource :: ContT r IO Resource
getResource = ContT $ bracket acquireResource releaseResource

But can we go in the opposite direction, i.e. decompose a value of type ContT r IO Resource into separate acquire and release functions?

To give an example of why we may want to do that, say a library only provides the withResource or getResource form, but we want to cache the allocated resource instances, deciding dynamically which ones to free. Another example would be combining resources and coroutines, see Understanding ResourceT by Michael Snoyman.

Note that not every ContT action follows the pattern “acquire; call the continuation; release”. Instead, the continuation may be called multiple times like so:

getInt :: ContT r IO Int
getInt = ContT $ \k -> last <$> mapM k [1..5]

(The parametricity ensures, however, that an action of type forall r . ContT r m a will call its continuation at least once or throw an exception.)

So what we are really asking is, assuming the ContT action has a certain form such as bracket acquire release, can we recover the acquire and release¹ actions?

A beautiful but useless solution

If our ContT action has the form

withResource :: MonadIO m => ContT r m Resource
withResource = ContT $ \k -> do
  x <- acquire
  r <- k x
  release x
  return r

then we can recover the acquire and release x actions by… adding another ContT layer! This may sound bizarre, but the intuition behind this is simple: the release x is itself a continuation of k x—it is what happens when k x returns—and the ContT monad allows us to capture that continuation. Here’s the code:

{-# LANGUAGE RankNTypes, ScopedTypeVariables #-}
import Control.Monad.Cont
import Control.Monad.IO.Class (liftIO)

runC :: Monad m => ContT a m a -> m a
runC ca = runContT ca return

shift :: Monad m => ((a -> m w) -> ContT w m w) -> ContT w m a
shift f = ContT $ runC . f

decomposeContT
  :: forall a w .
     (forall r m . MonadIO m => ContT r m a)
  -> IO (a, IO ()) -- the resource and its release action
decomposeContT ca = runC $ runContT ca $ \a ->
  shift $ \k -> do
    let
      r :: (a, IO ())
      r = (a, void $ k r)
    return r

Here, shift captures the current continuation delimited by runC; see Oleg’s delimited continuations tutorial.

Here is an example of how we can use decomposeContT. We generate two random numbers and then “release” them in their increasing order:

import System.Random
import Text.Printf (printf)

withRandomInt :: MonadIO m => ContT r m Int
withRandomInt = ContT $ \k -> do
  n :: Int <- liftIO $ randomRIO (1,100)
  liftIO $ printf "Acquired number %d\n" n
  r <- k n
  liftIO $ printf "Released number %d\n" n
  return r

main = do
  (num1, release1) <- decomposeContT withRandomInt
  (num2, release2) <- decomposeContT withRandomInt
  if num1 < num2
    then release1 >> release2
    else release2 >> release1

However, this solution has a major drawback. It relies on the ability to run our ContT action in a non-standard base monad (another ContT). Therefore, it cannot be used to decompose ContT $ bracket acquire release, because bracket cannot run in the ContT monad.

A more practical solution

A less elegant solution, but one that actually solves real-world problems, uses threads. Every resource is acquired in its own thread. The thread then blocks on an MVar until it receives a signal that the resource is no longer needed, in which case it proceeds to release it.

Here is the code.

{-# LANGUAGE RankNTypes, ScopedTypeVariables #-}
import Control.Monad.Cont
import Control.Concurrent
import Control.Exception

data ContCall a
  = ContException !SomeException
    -- ^ an exception occurred before the continuation was called
  | ContCall a
    -- ^ the continuation was called with this argument
  | ContNoCall
    -- ^ the ContT action returned wihtout ever calling the continuation

-- | Retrieve a resource from a 'ContT' action and return an action to
-- release it.
decomposeContT :: forall a . (forall r . ContT r IO a) -> IO (a, IO ())
decomposeContT ca = mask $ \restore -> do
  -- mvar_a is used to pass the 'a' result (or an exception) from the ContT thread to the
  -- calling thread
  mvar_a :: MVar (ContCall a) <- newEmptyMVar
  -- mvar_r is used to signal the ContT thread that its resources can be
  -- freed
  mvar_r :: MVar () <- newEmptyMVar
  -- mvar_e is used to communicate a possible final exception to the
  -- calling thread
  mvar_e :: MVar (Maybe SomeException) <- newEmptyMVar
  let
    -- tell the ContT thread to release its resources
    freeResources :: IO ()
    freeResources = void $ tryPutMVar mvar_r ()
    -- like freeResources, but also check and propagate any exception that
    -- arose while trying to free the resource
    freeResourcesCheckException :: IO ()
    freeResourcesCheckException = do
      void $ tryPutMVar mvar_r ()
      mb_e <- readMVar mvar_e
      maybe (return ()) throwIO mb_e
  pid <- forkIO $ do
    ----------------------------------------------------------------------
    --             The code below runs in a new thread
    ----------------------------------------------------------------------
    r <- try . restore $ runContT ca $ \a -> do
      -- Try recording the argument of the continuation call in the MVar.
      -- Might fail (return False) if this is not the first continuation
      -- call, but we don't care.
      _ :: Bool <- tryPutMVar mvar_a (ContCall a)
      -- Then wait until it's ok to free the resources.
      () <- readMVar mvar_r
      return ()
    case r of
      Right () -> do
        -- The ContT action returned successfully. We don't know how many
        -- times the continuation was called. Try putting ContNoCall in
        -- case it wasn't called at all. If it was called, then tryPutMVar
        -- will return False; we don't care.
        _ :: Bool <- tryPutMVar mvar_a ContNoCall
        _ :: Bool <- tryPutMVar mvar_e Nothing
        return ()
      Left e -> do
        -- An exception was raised. We don't know whether it was before or
        -- after the continuation was called. Try putting the exception in
        -- both mvar_r and mvar_e so that it surfaces exactly once
        -- (assuming the cleanup function eventually gets called) in the
        -- calling thread.
        _ :: Bool <- tryPutMVar mvar_a $ ContException e
        _ :: Bool <- tryPutMVar mvar_e $ Just e
        return ()
    ----------------------------------------------------------------------
    --             The code above runs in a new thread
    ----------------------------------------------------------------------

  -- Now, in the calling thread, we wait until mvar_a is filled
  -- readMVar is blocking, so it may receive exceptions. If there is
  -- an exception, we also send it to the ContT thread.
  contCall <- readMVar mvar_a `catch` (\(e :: SomeException) -> do throwTo pid e; throwIO e)
  case contCall of
    ContCall a -> return (a, freeResourcesCheckException)
    ContException e -> do
      freeResources
      throwIO e
    ContNoCall -> do
      freeResources
      throwIO $ ErrorCall "decomposeContT: the continuation was never called"

Or, more precisely, release x, where x is the value returned by acquire. We cannot infer how release would act on any other value, of course.↩︎

Why would you use ContT?

Fri, 07 Jun 2019 20:00:00 +0000

The ContT transformer is one of the more exotic and less used transformers provided by mtl. The Control.Monad.Cont module now even includes the following warning:

Before using the Continuation monad, be sure that you have a firm understanding of continuation-passing style and that continuations represent the best solution to your particular design problem. Many algorithms which require continuations in other languages do not require them in Haskell, due to Haskell’s lazy semantics. Abuse of the Continuation monad can produce code that is impossible to understand and maintain.

So what is ContT, and when does it represent the best solution to a problem?

Consider the following three functions from the base library:

openFile :: FilePath -> IOMode -> IO Handle
takeMVar :: MVar a             -> IO a
newArray :: Storable a => [a]  -> IO (Ptr a)

They are all “unsafe” in the sense that they return something to you but don’t clean up after you: don’t close the file you’ve just opened, put back the MVar you’ve just taken, or free the array’s memory. If you forget to perform the cleanup, it’s on you.

Indeed, a function that closed the file right after opening it would be rather useless. Therefore, a “safer” function should know what you intend to do with the file handle and insert that action between opening and closing the file.

The base library provides the following safer versions of the above functions:

                                 {-~~~~~~~~~~~~~~~~~~~~~~~~-}
withFile :: FilePath -> IOMode  -> (Handle -> IO r) -> IO r
withMVar :: MVar a              -> (a      -> IO r) -> IO r
withArray :: Storable a => [a]  -> (Ptr a  -> IO r) -> IO r
                                 {-~~~~~~~~~~~~~~~~~~~~~~~~-}

Notice how these functions follow the same pattern and how they relate to their unsafe versions: if the unsafe function returned IO a, then the corresponding safe function takes an additional argument of the form (a -> IO r) and returns IO r. This style of writing functions is called the continuation-passing style (CPS), and the argument a -> IO r is called the continuation.

ContT gives this pattern its own type:

newtype ContT r m a = ContT { runContT :: (a -> m r) -> m r }

(In this article, the underlying monad m will always be IO.)

You can construct ContT functions from the functions written in the CPS:

\v   -> ContT (withMVar v)   :: MVar a             -> ContT r IO a
\f m -> ContT (withFile f m) :: FilePath -> IOMode -> ContT r IO Handle
\l   -> ContT (withArray l)  :: Storable a => [a]  -> ContT r IO (Ptr a)

Notice how the types of the safe functions, once wrapped in ContT, become even more similar to the types of their non-safe versions: the only difference is ContT r IO instead of IO in the return type. And because ContT turns out to be a monad transformer, we can use and combine the safe function in all the same way as we could with the unsafe functions—getting the added safety essentially for free. Once you no longer need the resources you’ve allocated inside ContT, you turn it to IO using runContT, and when you exit a ContT block, all the cleanup actions are run—in the reverse order, of course.

Let’s consider some practical examples.

llvm-hs

In the llvm-hs package’s API, the continuation-passing style is used to reliably deallocate the allocated LLVM structures and classes, which are written in C++.

Here’s an example from llvm-hs’s test suite, reformatted such that each nesting level gets its own indentation level:

testCase "eager compilation" $ do
  resolvers <- newIORef Map.empty
  withTestModule $ \mod ->
    withHostTargetMachine $ \tm ->
      withExecutionSession $ \es ->
        withObjectLinkingLayer es (\k -> fmap (\rs -> rs Map.! k) (readIORef resolvers)) $ \linkingLayer ->
          withIRCompileLayer linkingLayer tm $ \compileLayer -> do
            testFunc <- mangleSymbol compileLayer "testFunc"
            withModuleKey es $ \k ->
              withSymbolResolver es (SymbolResolver (resolver testFunc compileLayer)) $ \resolver -> do
                modifyIORef' resolvers (Map.insert k resolver)
                withModule compileLayer k mod $ do
                  mainSymbol <- mangleSymbol compileLayer "main"
                  Right (JITSymbol mainFn _) <- CL.findSymbol compileLayer mainSymbol True
                  result <- mkMain (castPtrToFunPtr (wordPtrToPtr mainFn))
                  result @?= 42
                  Right (JITSymbol mainFn _) <- CL.findSymbolIn compileLayer k mainSymbol True
                  result <- mkMain (castPtrToFunPtr (wordPtrToPtr mainFn))
                  result @?= 42
                  unknownSymbol <- mangleSymbol compileLayer "unknownSymbol"
                  unknownSymbolRes <- CL.findSymbol compileLayer unknownSymbol True
                  unknownSymbolRes @?= Left (JITSymbolError mempty),

And here’s the same test rewritten with ContT:

testCase "eager compilation" $
  flip runContT return $ do
    resolvers <- liftIO $ newIORef Map.empty
    mod <- ContT withTestModule
    tm <- ContT withHostTargetMachine
    es <- ContT withExecutionSession
    linkingLayer <- ContT $ withObjectLinkingLayer es (\k -> fmap (\rs -> rs Map.! k) (readIORef resolvers))
    compileLayer <- ContT $ withIRCompileLayer linkingLayer tm
    testFunc <- liftIO $ mangleSymbol compileLayer "testFunc"
    k <- ContT $ withModuleKey es
    resolver <- ContT $ withSymbolResolver es (SymbolResolver (resolver testFunc compileLayer))
    liftIO $ modifyIORef' resolvers (Map.insert k resolver)
    ContT $ \c -> withModule compileLayer k mod (c ())
    -- All functions below are non-CPS, so we combine them into a single IO block
    -- instead of applying liftIO on every line.
    liftIO $ do
      mainSymbol <- mangleSymbol compileLayer "main"
      Right (JITSymbol mainFn _) <- CL.findSymbol compileLayer mainSymbol True
      result <- mkMain (castPtrToFunPtr (wordPtrToPtr mainFn))
      result @?= 42
      Right (JITSymbol mainFn _) <- CL.findSymbolIn compileLayer k mainSymbol True
      result <- mkMain (castPtrToFunPtr (wordPtrToPtr mainFn))
      result @?= 42
      unknownSymbol <- mangleSymbol compileLayer "unknownSymbol"
      unknownSymbolRes <- CL.findSymbol compileLayer unknownSymbol True
      unknownSymbolRes @?= Left (JITSymbolError mempty),

Most of the nesting and indentation is gone. The code is completely linear now, just as if we were using the unsafe, non-CPS functions. Applying ContT on every line is a bit awkward, but that would be gone if the library’s API was designed so that the CPS functions were already in the ContT monad.

Reading from a list of files

In the above example, we had merely a high nesting level. But what if the nesting level is unknown in advance?

Consider a program that takes a list of files and prints them interleaved—a line from the first file, then a line from the second, and so on, printing the line m+1 of the first file after the line m of the last file.

To implement such a program, we need to keep all the n files open all the time. If we want to use the safe withFile function, we need to nest it n times, where n is not known until the program is run. How is that even possible?

Well, it is possible with ContT.

import Control.Monad.Cont
import System.IO
import System.Environment

main = flip runContT return $ do
  args <- liftIO getArgs
  handles <- forM args $ \arg ->
    ContT $ withFile arg ReadMode
  liftIO $ print_interleaved_from_handles handles

print_interleaved_from_handles :: [Handle] -> IO ()
print_interleaved_from_handles = ...

Allocating a linked list

This example is similar to the previous one, except it actually occurred in my practice. See Way 2 in 6 ways to manage allocated memory in Haskell.

The withX-style safe functions are far from the only possible use for CPS and ContT, but it’s the one that I encounter most often. So if you were wondering why you’d use a continuation monad, hopefully this gives you some ideas.

How (not) to convert CDouble to Double

Tue, 14 May 2019 20:00:00 +0000

What’s wrong with the following code?

module Acos (acos) where

import Prelude hiding (acos)
import Foreign.C.Types (CDouble(..))

foreign import ccall "math.h acos" c_acos :: CDouble -> CDouble

acos :: Double -> Double
acos = realToFrac . c_acos . realToFrac

If you use QuickCheck to test the equivalence of Acos.acos and Prelude.acos, you’ll quickly find a counterexample:

> Prelude.acos 1.1
NaN
> Acos.acos 1.1
Infinity

You might think this is a difference in the semantics of Haskell acos vs. C acos, but the acos(3) manpage disproves that:

If x is outside the range [-1, 1], a domain error occurs, and a NaN is returned.

Moreover, you’ll notice the discrepancy only when compiling the Haskell program with -O0. If you compile with -O1 or higher, both versions will result in NaN. So what’s going on here?

What turns the NaN turned into the Infinity is realToFrac. It is defined as follows:

realToFrac :: (Real a, Fractional b) => a -> b
realToFrac = fromRational . toRational

Unlike Double, Rational, which is defined as a ratio of two Integers, has no way to represent special values such as NaN. Instead, toRational (acos 1.1) results in a fraction with some ridiculously large numerator, which turns into Infinity when converted back to Double.

When you compile with -O1 or higher, the following rewrite rules fire and avoid the round trip through Rational:

"realToFrac/a->CDouble"     realToFrac = \x -> CDouble  (realToFrac x)
"realToFrac/CDouble->a"     realToFrac = \(CDouble  x) -> realToFrac x
"realToFrac/Double->Double" realToFrac = id :: Double -> Double

Unfortunately, the Haskell 2010 Report doesn’t give you any reliable way to convert between Double and CDouble. According to the Report, CDouble is an abstract newtype, about which all you know is the list of instances, including Real and Fractional. So if you want to stay portable, realToFrac seems to be the only solution available.

However, if you only care about GHC and its base library (which pretty much everyone is using nowadays), then you can take advantage of the fact that the constructor of the CDouble newtype is exported. You can use coerce from Data.Coerce or apply the data constructor CDouble directly.

So here’s a reliable, but not portable, version of the Acos module above:

module Acos (acos) where

import Prelude hiding (acos)
import Foreign.C.Types (CDouble(..))
import Data.Coerce (coerce)

foreign import ccall "math.h acos" c_acos :: CDouble -> CDouble

acos :: Double -> Double
acos = coerce c_acos

Fix a torrent file encoding

Tue, 07 May 2019 20:00:00 +0000

Sometimes, when you open a torrent file, the non-Latin letters in the file names will appear garbled. Apparently, some programs write the utf-8 name in the field path.utf-8 and write the field path in some other encoding.

I wrote a small Perl script that fixes such torrent files by copying the path.utf-8 field into the path field. To fix the torrent file:

Download the script.
Make sure you have Perl installed.
Install the Bencode Perl module.

E.g. on Fedora, you can run dnf install perl-Bencode.noarch.

Run

perl fix-torrent-encoding.pl my_broken_file.torrent new_fixed_file.torrent

and new_fixed_file.torrent will be created.

Lazy validation

Sat, 02 Mar 2019 20:00:00 +0000

Why lazy validation?

The Validation applicative functor is a convenient tool for checking anything for errors. It’s similar to the Either type, but its Applicative instance works differently. Whereas an Either computation stops at the very first Left value, a Validation computation continues, collecting all other errors so they all can be presented to the user at once. I gave a more detailed introduction to the Validation applicative in the article Smarter validation.

However, most of the time you don’t want all of the errors; you only want some reasonable number of them. Once you reach, say, 100 errors, the computation can be aborted. In particular, if we only request a single error, then the Validation applicative should be equivalent to the Either applicative.

This is precisely the issue that I tackle in Smarter validation. In that article, I describe an implementation that maintains a bounded list of errors internally. The maximum number of errors (“capacity”) is specified on the type level so that it is accessible to the Applicative instance.

While that approach works, it has a few drawbacks:

The implementation is somewhat complicated.
Keeping track of the list capacity on the type level is inconvenient.
The capacity has to be specified in advance, before running the computation. For instance, it is not possible to get the first 10 errors, then request another 10.

There is an alternative that is more lightweight and in the spirit of Haskell: just make the Validation applicative produce the errors incrementally (lazily) as the computation progresses. Then you can take (as in Data.List.take) any number of errors you need, and the computation will only be run until it either completes or produces the requested number of errors.

An implementation

Here’s one way to implement a lazy Valdation applicative:

newtype Validation a b = Validation { runValidation :: Either a b }
  deriving Functor

instance Monoid a => Applicative (Validation a) where
  pure = Validation . Right
  af <*> ax =
    case runValidation af of
      Right f -> fmap f ax
      Left e1 ->
        let e2 =
              case runValidation ax of
                Right _ -> mempty
                Left e -> e
        in Validation (Left (e1 <> e2))

Notice that when the first argument of <*>, namely af, is Left, we return the result Validation (Left (e1 <> e2)) without analyzing the second argument, namely ax, although the value e2 does depend on it. But without knowing anything about e2, we can already tell that the result is a Left, and the inner value (the list of errors) starts with e1.

The laziness of this Validation applicative means that, for instance, the code

case runValidation a of
  Right r -> r
  Left errs -> take 10 errs

can return the first 10 errors as soon as they are available and stop the further evaluation of a. It can even run on an infinite computation, such as

Validation (Left ["error"]) *>
  fix (Validation (Right ()) *>))

Comparing different implementations

I tested the laziness of the different implementations, including the ones I could find on hackage. The test suite is available here, and its results are shown below:

	`fix (f *>)`	`f > f > fix (s *>)`	`f > (f > fix (s *>))`	`f > fix (s >)`
transformers-0.5.5.0	✗	✗	✗	✗
transformers-0.5.5.0 (patched)	✓	✓	✓	✓
either-5.0.1	✓	✗	✓	✗
validation-1	✓	✗	✓	✗
This article	✓	✓	✓	✓
Smarter (cap = 1)	✓	✓	✓	✓
Smarter (cap = 2)	✓	✓	✓	✗

Each test attempts to extract the first error from an infinite Validation computation. These computations are:

fix (f *>), an infinite stream of failures.

fix (f *>) simply means f *> (f *> (f *> ... )).
f *> f *> fix (s *>), two failures followed by an infinite stream of successes. Since *> is defined as infixl, this really means (f *> f) *> fix (s *>).
f *> (f *> fix (s *>)), again two failures followed by an infinite stream of successes, but this time the grouping is different. The Applicative laws say that this shouldn’t be any different from the previous test, but the table shows that they do differ in practice.
f *> fix (s *>), a single failure followed by an infinite stream of successes.

The implementations tested are:

The Errors type from transformers-0.5.5.0, based on the Lift applicative.
The same Errors type, but with a patched Applicative Lift instance (see below).
The Validation type from either-5.0.1.
The Validation type from validation-1, which, from what I can tell, is more or less the same code as in either-5.0.1.
The Validation type defined above in this article.
The SmartValidation type defined in the earlier article, configured to retain no more than 1 error.
The same SmartValidation type, but configured to retain no more than 2 errors.

So what conclusions can we draw from this experiment?

The Errors type from transformers is usually my go-to implementation, since it is available in the de-facto standard package and therefore doesn’t incur any extra dependencies. As the table shows, it doesn’t pass any tests, so it’s not lazy at all. This is because the current Applicative Lift instance assumes that both sides of <*> are evaluated:
```
instance (Applicative f) => Applicative (Lift f) where
    pure = Pure
    Pure f <*> Pure x = Pure (f x)
    Pure f <*> Other y = Other (f <$> y)
    Other f <*> Pure x = Other (($ x) <$> f)
    Other f <*> Other y = Other (f <*> y)
```
However, this instance can be easily made lazy:
```
instance (Applicative f) => Applicative (Lift f) where
    pure = Pure
    Pure f <*> ax = f <$> ax
    Other f <*> ax =
      Other $ f <*>
        (case ax of
          Pure x -> pure x
          Other y -> y
        )
```
This is the “transformers-0.5.5.0 (patched)” entry in the table. I also opened an issue to put the lazy version into transformers, and Ross Paterson promptly made the change, so starting from the next version, transformers will provide a lazy Validation applicative.
The either-5.0.1 and validation-1 versions of Validation appear to be identical, and they pass the same set of tests. For practical purposes they can be considered lazy, although they do fail on some edge cases. These failures, however, are not accidental: they are a consequence of relying only on the Semigroup instance of errors, not Monoid. On the one hand, this is a good property, because it makes it less likely that you’ll end up with Failure mempty.

On the other hand, because there’s no mempty, the implementation of <*> has a “latency” of 1 error: in order to emit an error, it needs to see not only this error but also the next one (or reach the end of the computation to find out that it was the last one).

Let’s see why this is the case. Say you see an error e. If the error type is a Monoid, you can emit e <> thunk, where thunk will evaluate either to e2 <> thunk2 or mempty.

Now suppose the error type is a Semigroup but not a Monoid. Say you emitted e <> thunk when you first saw it, but then the computation ended without any additional errors. What should the thunk evaluate to?

To work around that issue, wait until you see either the next error, e2, or the end of the computation. In the first case, emit e <> thunk, knowing that the thunk can at least evaluate to e2. In the second case, simply return e.

To summarize, either-5.0.1 and validation-1 are not perfectly lazy, but they come pretty close and have other advantages.

The “smart validation” applicative, when restricted to produce a single error, does pass all the tests. But it’s not really lazy—we just cheated when we told it in advance how many errors we were going to request. When we told it to hold no more than 2 errors and then gave it an infinite stream with only one error, it never returned that error.

That said, notice that “Smarter (cap = 2)” passed the f *> f *> fix (s *>) test, which either-5.0.1 and validation-1 did not pass. Because the smart validation functor is continuation-based, it doesn’t care how the original computation was grouped.

This has also practical implications regarding performance. The “smart validation” approach has linear complexity no matter how the computation is oragnized, whereas the more straightforward approaches have quadratic worst-case complexity.

In the following test, we requested not the first, but the last error from a long left-nested stream of errors. The test passes only if it finishes in under 100ms.

	`iterate (*> f) f !! 10000`
transformers-0.5.5.0	✗
transformers-0.5.5.0 (patched)	✗
either-5.0.1	✗
validation-1	✗
This article	✗
Smarter (cap = 1)	✓
Smarter (cap = 2)	✓

Surprises of the Haskell module system (part 2)

Sat, 26 Jan 2019 20:00:00 +0000

It’s been 6 years since I published Surprises of the Haskell module system (part 1), so I thought it was time to follow through on the implicit promise and publish part 2.

Exporting constructors without data type

Is it possible to export a data constructor without exporting its parent data type? To be clear, there’s probably no good reason to want to do that, but it’s a fun puzzle, and we’ll learn something while trying to solve it.

Surprisingly, the answer depends on how our constructor and type are named. First, consider the case when they are named differently, as in

data T = C

The naive attempt

module M (C) where
data T = C

does not work. The Haskell 2010 standard says this about data types in the export list:

An algebraic datatype T declared by a data or newtype declaration may be named in one of three ways:

The form T names the type but not the constructors or field names. The ability to export a type without its constructors allows the construction of abstract datatypes (see Section 5.8).

The form T(c₁,…,c_n), names the type and some or all of its constructors and field names.

The abbreviated form T(..) names the type and all its constructors and field names that are currently in scope (whether qualified or not).

Therefore, there seems to be no way to export a constructor without exporting its parent type at the same time. But here’s the trick. First, let’s move the data type definition from M to a different module, Internal:

module Internal where
data T = C

Then, in our module M, import Internal like this:

module M where
import Internal hiding (T)

This will import C but not T. Here’s Haskell 2010 on the semantics of hiding lists:

Entities can be excluded by using the form hiding(import₁ , … , import_n ), which specifies that all entities exported by the named module should be imported except for those named in the list. Data constructors may be named directly in hiding lists without being prefixed by the associated type. Thus, in

import M hiding (C)

any constructor, class, or type named C is excluded. In contrast, using C in an import list names only a class or type.

So now, within module M, we have only C in scope but not T. But M does not export any of them yet, and it seems like we need T in scope in order to add C to the export list.

Here we use a second trick: instead of naming C specifically, we export our Internal module as a whole:

module M (module Internal) where
import Internal hiding (T)

And this achieves the desired result: M now exports the data constructor C but not the data type T. See Part 1 for more discussion of module exports.

This all works because the data type and its constructor are named differently. If we have

data T = T

this won’t work, because

import Internal hiding (T)

will hide both the data type and the data constructor. I actually don’t know a way to export the constructor T without the type T; if you do, please let me know.

Exporting fields without constructor

A similar question is that of exporting a field name without exporting its parent constructor, as in

module M (T, field) where
data T = C { field :: Bool }

The above code works. But what does it mean? Field names can be used as functions (e.g. field :: T -> Bool), but they can also be used in the record construction (C { field = True }) and record update (x { field = True }) syntaxes.

Since we don’t export the data constructor, clearly record construction won’t work. But what about record update? Does field lose its magic and become a simple function when exported this way—either because it wasn’t exported as T(field), or because the data constructor is not in scope? The answer is no, it is still a valid field name, and x { field = True } still works. Here’s Haskell 2010:

It makes no difference to an importing module how an entity was exported. For example, a field name f from data type T may be exported individually (f, item (1) above); or as an explicitly-named member of its data type (T(f), item (2)); or as an implicitly-named member (T(..), item(2)); or by exporting an entire module (module M, item (5)).

And there’s no requirement that the constructor be in scope in order to use the record update syntax—even though its desugaring would require all of the constructors to be in scope.

Some authors use this to their advantage: for instance, Michael Snoyman suggests exporting fields but not constructors for settings types.

But more often than not, we want the opposite: to make the type abstract and export a getter for it. If you think the module M above exports an abstract type T and a function field :: T -> Bool, then you may believe you are free to change the implementation of T without your users noticing. So in the next version you may change T to

module M (T, field) where
data T = C { items :: [Int] }
field :: T -> Bool
field = not . null . items

This new module still exports a type T and a function field :: T -> Bool, but that function is no longer a record field. If someone was using field in a record update, x { field = True }, their code is going to break.

There’s no way to export a field as a plain function or otherwise strip it of its magic. You can either define a getter yourself, as in

data T = C Bool
field :: T -> Bool
field = \case
  C a -> a

or create a copy of a field, as in

data T = C { _field :: Bool }
field :: T -> Bool
field = _field

Hidden types and type synonyms

Consider this example:

module M (x) where

type T = Int

x :: T  
x = 1

What type does x have in a module that imports M? On the one hand, T is Int, so x :: Int. On the other hand, since T is not in scope, there’s no way to know that T = Int, so it might seem that x is now an opaque value of some abstract type T that we know nothing about. In fact, the former view is correct and the latter isn’t.

The type of an exported entity is unaffected by non-exported type synonyms. For example, in

  module M(x) where
    type T = Int
    x :: T
    x = 1

the type of x is both T and Int; these are interchangeable even when T is not in scope. That is, the definition of T is available to any module that encounters it whether or not the name T is in scope. The only reason to export T is to allow other modules to refer it by name; the type checker finds the definition of T if needed whether or not it is exported.

Similarly, you may wonder what does it mean to export a function but hide some of the types that are part of its signature. Here’s what Haskell 2010 has to say about that:

[E]ntities that the compiler requires for type checking or other compile time analysis need not be imported if they are not mentioned by name. The Haskell compilation system is responsible for finding any information needed for compilation without the help of the programmer. That is, the import of a variable x does not require that the datatypes and classes in the signature of x be brought into the module along with x unless these entities are referenced by name in the user program. The Haskell system silently imports any information that must accompany an entity for type checking or any other purposes. Such entities need not even be explicitly exported: the following program is valid even though T does not escape M1:

  module M1(x) where
    data T = T
    x = T

  module M2 where
    import M1(x)
    y = x

In this example, there is no way to supply an explicit type signature for y since T is not in scope. Whether or not T is explicitly exported, module M2 knows enough about T to correctly type check the program.

Self-exporting module

By default—when no export list is present in a module—all values, types and classes defined in the module are exported, but not those that are imported. With an explicit export list, we can export a mix of entities from the module itself and the modules it imports. But now you have to explicitly list everything defined in the current module and remember to update the list when you define a new function or type. Or do you?

There’s a nice trick to have an explicit export list and automatically export everything from the current module at the same time: just let the module export itself.

-- exports x, y, and z
module M (module M, Foo.x, Bar.y) where

import Foo
import Bar

z = undefined

Thanks to the clear semantics of module exports (which we discussed in Part 1), self-exporting is well-defined and doesn’t involve any recursion.

A better noise gate

Sat, 12 Jan 2019 20:00:00 +0000

A noise gate is a type of sound effect. It detects the parts of the audio signal with very low volume, where nothing is happening apart from the background noise, and either reduces the volume further or silences it completely. It doesn’t remove the noise from under the speech—that’s called noise reduction—but it removes the noise between the phrases.

Back when audio processing was analog and real-time, the noise gate had to be simple. It would notice when audio went either above or below the threshold, and “open” or “close” the gate correspondingly. When the gate is closed, the output is suppressed to remove the noise; when the gate is open, the input goes to the output unchanged. To soften the transition, the gate doesn’t open/close immediately but goes through the attack and decay phases, spread across a short amount of time, like 50ms.

The de-facto standard noise gate plugin for audacity, noisegate.ny, does a pretty good job emulating an analog noise gate but also doesn’t do much beyond that. There are two major drawbacks of that plugin, which I can summarize as:

Sometimes it doesn’t reduce enough
Sometimes it reduces too much

So I implemented my own noise gate plugin that addresses these issues—and it is also much faster. But before I get into that, let me explain in more detail the ways in which the usual noise gate plugins fall short.

Not enough reduction

A typical noise gate (including noisegate.ny) closes when the loudness (or, more precisely, the amplitude) of the signal falls below a specified threshold. Once the amplitude exceeds the threshold, the gate opens. Therefore, a single loud click, even a very short one, can open the gate.

Therefore, a single run of the noise gate is often not enough to get rid of all the noise. You would need then to go and manually remove the remaining patches of noise and clicks, or to run a second (and sometimes third) noise gate with different parameters.

Too much reduction

So why not just raise the threshold to cover the peaks as well? That’s the second issue with the usual noise gates. Near the beginning or the end of a phrase, the sound is usually softer and may not be enough to open the gate. This results in a characteristic loss of syllables. For example, here’s a short clip from The Effort Report podcast.

And here’s the same clip after applying a noise gate with threshold=-47dB, level reduction=-50dB, and attack/decay=50ms.

In the original clip, the “f” sound in “off” can barely be heard, but the brain still registers it. After the noise gate, it’s lost completely, and the phrase sounds unnatural: make sure you’ve got your ringer turned o’.

There are a few strategies to fight this annoying effect:

lower the threshold, e.g. -55dB instead of -47dB.
increase the attack/decay, e.g. 500ms instead of 50ms.
lower the reduction level, e.g. -10dB instead of -50dB.

In the past, I used a combination of these tricks and several runs of the noise gate to get rid of all the noise while trying not to damage the speech. It worked but took a lot of time.

To be fair, there are noise gates implementation that, in addition to attack and decay, also have a sustain phase—but noisegate.ny is not one of them.

A better noise gate

In addition to the traditional Threshold and Attack/decay parameters, my noise gate has two new parameters: window size and the minimum amount of non-silent audio per window.

At each time point, the noise gate makes the decision whether to silence it or not. To make the decision, we consider a symmetric window around the current time point, and whose width, in milliseconds, is given by the user (the second parameters on the screenshot above). Then we calculate the amount of non-silent audio, in milliseconds, present within this window. The non-silent audio is defined as audio whose amplitude exceeds the threshold (the first parameter on the screenshot). If the amount of non-silent audio exceeds the critical duration (the third parameter), then the current audio sample is not silenced. Otherwise, i.e. if most audio in the vicinity of the current time point falls below the threshold, the current sample is silenced.

So why is this algorithm a better noise gate? First, consider the case when the traditional noise gate doesn’t reduce enough. That happens when there’s a high-amplitude spike in the audio. These spikes or clicks tend to be very short, and if they are surrounded by low-volume noise, then in their window there won’t be enough non-silence to open the gate.

Next, consider the case when the traditional noise gate reduces too much. This typically happens near at the boundaries between silence and non-silence. Therefore, even if the instanteneous volume is low, in the window around it there will be enough high-volume signal to keep the gate open.

Choosing the parameters

Here’s how I suggest to pick the right parameters for the noise gate.

Set the threshold to the general noise floor level, ignoring the occasional spikes and clicks.
Find one of the widest (i.e. longest) spikes/clicks that rise above the threshold. Measure the duration of the part that’s above the threshold and set the “Non-silent audio per window” slightly above that value. If there are several wide clicks next to each other, add their widths together.
Given the threshold you chose on step 1, think about how long the beginning or ending of a phrase can be that falls below the threshold. Let’s say it’s 600ms. To get the window size, add that duration to the value of parameter #3 (non-silent audio per window) and double it. For instance, let’s say you chose 150ms on step 2. Then your window size will be 2×(600ms+150ms)=1500ms.
The attack/decay choice is subjective. I usually pick 50ms. It shouldn’t matter much because by the time the gate closes (assuming you picked the good values for the other parameters), any speech should be over, and we are only dealing with noise at this point. This is in contrast to the standard noise gates, where attack/decay may cover quieter parts of the speech.

Also, if you use a noise gate (any noise gate!), make sure to run noise reduction first. When there’s a uniform noise behind the speech that suddenly disappears in between the phrases, it sounds very unprofessional.

LADSPA

My noise gate uses the plugin interface called LADSPA, which stands for Linux Audio Developer’s Simple Plugin API. This is different from most Audacity plugins, which use the Nyquist interface. The disadvantage of LADSPA is that you’ll have to compile the plugin for your platform. But there are quite a few advantages:

It is very fast. I ran both my noise gate and Steve’s noisegate.ny on the same track, 1h19m41s in length. noigate.ny took 1 minute 18 seconds to finish; my noise gate took only 9 seconds.
The progress bar in Audacity is accurate for LADSPA plugins. When I ran noisegate.ny, the progress bar reached 100% and stayed there at 30 seconds—by which time the plugin wasn’t even halfway done.
It processes the track incrementally, so it consumes much less memory.
In Audacity, there is a live preview for LADSPA plugins but not Nyquist plugins.
It can work under a variety of audio editors and DAWs that support the LADSPA interface (Audacity, Ardour, GNU Sound etc.), unlike Nyquist, which, to my knowledge, is only integrated into Audacity (although there exists a standalone Nyquist interpreter). There is even a command-line program distributed with LADSPA, applyplugin, that can apply a plugin to an audio file in batch mode, without any GUI.

Give it a try. The source and installation instructions are on GitHub: https://github.com/UnkindPartition/noise-gate.

Homology, similarity, and alignment

Sat, 05 Jan 2019 20:00:00 +0000

Although homology, similarity, and alignment are all relatively well defined notions in biology and computer science, scientists sometimes get confused about their meaning or the relationships between them. The results range from annoyed colleagues to very annoyed colleagues.

So let’s sort these things out.

Definitions

To keep things simple and concrete, I’ll be mostly talking about DNA sequences, but everything here applies equally to RNA or amino acid sequences.

Homology

Homology is a concept from evolutionary biology. It means that two DNA sequences have a common evolutionary origin (Reeck et al., 1987). For example, the mouse gene Hba-a1 is homologous with the human gene HBA1 because both species inherited it from their latest common ancestor. Homologous sequences can be divided into paralogous and orthologous sequences, although that distinction won’t be important here.

Sequence similarity

Sequence similarity is a concept from computational biology and computer science. Sequence similarity is a number that shows how much two sequences are similar. Sequence similarity is sometimes, but not always, defined via sequence distance: the smaller the distance, the more similar the sequences¹. For instance, the following two DNA sequences are highly similar (the differences are highlighted in red):

GTCCTCATAACTCTCTCTAG
GTCGTCATAAC CTCTCTAG

Their similarity is witnessed by their low Levenshtein distance of 2—it only takes two edit operations (one substitution and one deletion) to transform the first sequence into the second one². But there are other distance metrics on sequences, and there are also similarity metrics not expressible as distances. Therefore, while sequence similarity is always a number determined based on two sequences, the specifics of how that number is calculated may vary.

Sometimes the similarity score is expressed as a percentage, namely “percent similarity” or “percent identity”. Percent identity usually refers to the ratio of the number of matching residues to the total length of the alignment (see below), e.g. 18/20=90% in the example above. See also Li, 2018. Percent similarity counts “similar” residues (usually amino acids) in addition to the identical ones. The similarity between amino acids can be defined either by their chemical properties or based on a PAM matrix.

Alignment

Alignment is another concept from computational biology. It is simply any way to align two sequences one below another, possibly with gaps³ (Gusfield, 1997). I already showed an example alignment above. Here it is again, this time with the gaps (deletions) indicated by a hyphen (-), as is customary for sequence alignments:

GTCCTCATAACTCTCTCTAG
GTCGTCATAAC-CTCTCTAG

You would think that this is the alignment for these two strings, but according to the definition, the following is also a valid alignment:

GTCCTCAT-AACTCTCTCTAG
GTCGTCATAA-C-CTCTCTAG

And this one, too:

GTCCTCATAACTCTCTCTAG-------------------
--------------------GTCGTCATAACCTCTCTAG

An alignment algorithm usually has a scoring function, which assigns every alignment a numeric score indicating how good an alignment it is, and tries to find the best alignment according to its scoring function.

Relationships

To understand these terms better, it helps to see how they are related to one another.

Homology vs. similarity

Neither homology nor high similarity imply the other one. Two sequences can be similar or even identical either by chance (if they are short) or by design (if synthesized). And two homologous loci can quickly diverge in their DNA sequence if they are not constrained by natural selection.

Whereas similarity is a property of the sequences (strings) themselves and does not depend on where they came from, homology is the opposite—a property of how two things (e.g. genes) came to be, regardless of their nucleotide content.

Thus, it does not even make sense to ask whether these two (similar!) sequences are homologous without putting them in the context of the genomes and organisms in which they evolved:

GTCCTCATAACTCTCTCTAG
GTCGTCATAAC CTCTCTAG

In fact, I generated these sequences randomly using R, so I can confidently say that they are not homologous—they simply aren’t objects of evolution.

That said, there are two valid connections between homology and similarity, and they are the source of confusion between these notions:

Similarity can be used to infer homology. Specifically, if the two similar genomic sequences are long and complex enough to have unlikely arisen independently even under similar selective pressures, then this is a strong evidence for their homology.
Most of the homologous loci that we observe have high similarity. I emphasized “that we observe” because there is an obvious circularity and bias here: we only know that two regions are homologous when they are similar. If we could somehow sample homologous regions in different organisms in an unbiased way, perhaps we would find much less similarity than we are accustomed to.

Similarity vs. alignment

An alignment could be converted to a similarity score using a scoring function. The simplest scoring function could be “add 1 for every matching characters in the alignment, 0 for every mismatch or space (gap)”. But there are many different scoring functions, and so there isn’t one canonical similarity corresponding to an alignment.

Usually when people talk about an alignment, they mean the optimal (best) alignment relative to a particular scoring function, but we need to remember that an alignment does not have to be optimal.

Similarity could be defined through an alignment—specifically, as the score of the best (highest-scoring) alignment between the two sequences.

Similarity could also be defined in many other ways—e.g. through a locality-sensitive hash (Ondov et al., 2016).

Alignment vs. homology

If two regions are homologous, they could be aligned to underline their evolutionary history. Let’s call an alignment in which aligned positions have a common origin an evolutionary alignment.

The evolutionary alignment does not have to be the optimal alignment with respect to any given scoring function. And vice versa, the optimal alignment (such as one calculated by an alignment program) does not necessarily reflect the evolutionary relationship between the sequences.

Let’s consider an example to illustrate this. In the ancestral genome, there is a sequence ATATATG. In one of the lineages, the second and third bases got deleted, resulting in ATATG. The evolutionary pairwise alignment of the ancestor and the lineage is


ATATATG
A--TATG

Note that, although this is an optimal alignment, it is not a unique optimal alignment. Assuming a simple scoring function that assigns 0 to matches and −1 to mismatches, insertions, and deletions, the alignments


ATATATG
AT--ATG

and


ATATATG
--ATATG

have the same score of −2, even though they do not represent the evolutionary relationship between the sequences.

Also note that this alignment does not necessarily tell us about the exact path by which we arrived from ATATATG to ATATG. What if there were two mutational events: first, AT got duplicated into ATAT, and then that ATAT got deleted? It would still result in the same alignment.

Now let’s consider another lineage, where the fifth base mutated to G, resulting in ATATGTG. The optimal alignment between the two lineages is


ATATG--
ATATGTG

with the score of −2. But this alignment does not reflect the evolutionary history, because the two aligned Gs do not have the same origin. Instead, the evolutionarily correct alignment is


A--TATG
ATATGTG

having a suboptimal score of −3. This shows that the optimal alignment and the evolutionary alignment do not have to coincide.

To summarize, an alignment by itself does not imply any evolutionary relationship between the sequences. The two aligned sequences do not have to be homologous, and even if they are, the alignment does not necessarily show which bases have a common origin, although an alignment can be used for this purpose. An alignment is just a way to visualize the differences and similarities between two sequences; if there is any extra meaning in a particular alignment, it has to be given explicitly.

Acknowledgments

Thanks to Nika Gurianova for the feedback on an earlier draft of this article.

References

Reeck, GR, de Haën, C, Teller, DC, Doolittle, RF, Fitch, WM, Dickerson, RE, Chambon, P, McLachlan, AD, Margoliash, E, Jukes, TH (1987). “Homology” in proteins and nucleic acids: a terminology muddle and a way out of it. Cell, 50, 5:667.

Gusfield, D. (1997). Algorithms on Strings, Trees, and Sequences: Computer Science and Computational Biology. Cambridge: Cambridge University Press.

Ondov, BD, Treangen, TJ, Melsted, P, Mallonee, AB, Bergman, NH, Koren, S, Phillippy, AM (2016). Mash: fast genome and metagenome distance estimation using MinHash. Genome Biol., 17, 1:132.

Heng Li. On the definition of sequence identity.

For something to be called a distance, it also needs to satisfy the distance axioms. That said, there are no agreed upon axioms for similarity metrics, so a proper similarity metric could be derived from an improper distance metric if desired.↩︎
Strictly speaking, the term “similarity” implies that more similar sequences get higher numbers, also known as similarity scores. To transform a distance metric into a similarity score we can simply negate it. So, if we wanted to be precise, we could say that in our example the sequences have the distance of 2 and similarity of −2.↩︎
This is more precisely called a global pairwise alignment, as opposed to a multiple sequence alignment or a local alignment, which I won’t cover here.↩︎

Bioinformatics events in Europe in 2019

Thu, 06 Dec 2018 20:00:00 +0000

Continuing the tradition from last year, here are some conferences and summer schools that will take place in Europe in 2019 and may be of interest to fellow bioinformaticians. I’ll be doing more curation this time, so the list will be shorter but with higher signal to noise ratio.

Event	Dates	Location	Early registration price	Early registration deadline
Revolutionizing Next-Generation Sequencing	March 25–26	Antwerp, Belgium	€500	February 11
Population genomics: Background, tools and programming	March 30–April 6	Procida, Italy	€1000	January 31
Advanced Lecture Course on Computational Systems Biology	March 31–April 6	Aussois, France	€800	February 28
Joint Summer School “Epigenetics in Single Cells”	April 3–10	London, UK; Berlin, Germany	Free	February 15
The 2019 whole-cell modeling summer school	April 7–13	Barcelona, Spain	€1000	December 1, 2018
European Human Genetics Conference (ESHG)	June 15–18	Gothenburg, Sweden	€500	April 4
European Conference on Computational Biology	July 21–25	Basel, Switzerland	CHF 942.38	June 20
CPANG: Computational PANGenomics	September 9–13	Oeiras, Portugal	€450	September 2
[BC]²: Basel Computational Biology Conference	September 10–11	Basel, Switzerland	CHF 300	June 12
NGSchool	October 24–31	Białobrzegi, Poland		May 1
European Bioconductor Meeting	December 9–10	Brussels, Belgium	€125

Tasty 1.2 supports dependencies between tests

Sat, 01 Dec 2018 20:00:00 +0000

Tasty (GitHub, Hackage) is a versatile and extensible testing framework for Haskell. Using tasty, you can quickly create a test suite consisting of unit tests, property tests and golden tests. Users of tasty benefit from automatic parallelism, a rich filtering language to choose which tests to run, sharing resources between tests and more.

Tasty 1.2 adds another long-awaited feature: dependencies between tests. If the tests only ran sequentially, dependencies wouldn’t be an issue: you could always arrange your tests in the desired order.

However, tasty runs all tests in parallel by default (using a user-specified or automatically determined number of worker threads). While it’s always been possible to force sequential execution by setting the number of threads to 1, sometimes you want to take advantage of the parallelism while also making sure that some tests run after some other tests due to the tests’ side effects.

This is now possible using the after combinator:

testGroup "Tests accessing the same resource"
  [ testCase "Test A" $ ...
  , after AllFinish "Test A" $
      testCase "Test B" $ ...
  ]

In this example, Test B will only start running after Test A has finished—successfully or not. Alternatively, you can tell Test B to run only when Test A succeeds (notice AllSucceed instead of AllFinish):

testGroup "Tests creating and using a resource"
  [ testCase "Test A" $ ...
  , after AllSucceed "Test A" $
      testCase "Test B" $ ...
  ]

The "Test A" argument of after in these examples is not simply a test name—it is a pattern in tasty’s AWK-like pattern language, so a test can depend simultaneously on a combination of different tests from different subtrees.

To learn more about dependencies in tasty, please refer to the README and the haddock documentation.

How do I record a podcast remotely? SquadCast vs. Zencastr vs. Cast vs. other options

Sun, 28 Oct 2018 20:00:00 +0000

One of the most frequent questions beginner (and many experienced) podcasters have is “how do I record a podcast with my co-hosts/guests being far away?”

If you want a simple recommendation, then use SquadCast. This is my own service of choice since July 2020, and so far I’ve been pretty happy with it.

If you want to explore other options, first let me explain which factors to look at when choosing a service. Then I’ll share my own experience of using different services.

What you should pay attention to

Are you getting individual tracks or a mix?

Once you finish recording, will you get a single file with all voices in it, or a file (“track”) per participant containing only that participant’s voice?

You should always prefer recording individual tracks. It may seem like more work to edit and post-process multiple tracks than a single one, but if you care at all about your audio quality, getting that quality out of a mixed track is very hard. Here are some things that a multi-track recording process lets you do:

Apply a noise gate to each track, so that when someone is not talking, their noise does not get into the final mix.
If two participants talk at the same time, you can mute one of them to let the other one be heard, or even let both of them be heard by rearranging their utterances in time so that they don’t overlap.
Adjust the audio levels of each participant so that their loudness is roughly the same in the final mix.
Use different effect settings (EQ, compression, de-noise, de-reverb) for different participants.

I find multi-track recording such an important feature for podcasters that I won’t even consider here any options that don’t provide it.

Are tracks recorded locally or remotely?

Most remote recording services that do multi-track recording do them locally: each participant’s track is recorded on that participant’s device, and then gets uploaded to the server once the recording is finished (or progressively while it lasts). A notable exception is Cleanfeed, which records tracks on their servers in real-time.

The advantages of recording locally (also known as “a double-ender”) are:

Since there is no need to compress audio for real-time transmission over the network, it is generally recorded in higher quality locally than remotely. However, this depends on the codecs used and the available bandwidth.
Locally recorded tracks are immune to network glitches. If your guest’s voice turned robotic or simply vanished, their own computer probably captured their voice in its original form.

But there are also disadvantages:

The tracks recorded on different computers tend to get out of sync with time — the effect known as “audio drift”. Some services attempt to combat this with various degrees of success. However, in practice I tend to readjust pieces of audio anyway to achieve the right duration of pauses between speakers, so unless the drift is enormous, it doesn’t really matter.
When the audio resides on your guest’s computer, there’s somewhat higher risk of it being lost or corrupted. When evaluating a service, find out whether they provide any options to restore the audio if something goes wrong, but also always have a backup recording on your side.

To elaborate on the last point: regardless of what service you use, always record two tracks locally: one with the audio from your own microphone, and another with the audio that goes into your headphones. (For Linux, see my audio recording guide; for other OSes, you can often use your DAW or a third-party app to do that.)

The first of these two tracks — the one with your own audio — probably will be of higher quality than the same track recorded by the service (even when the service records locally), so if the audio drift is not too bad, you can routinely use it in the podcast. On the other hand, the second track — the one with your guests — will probably have lower quality than the one recorded by the service (and will be a mix-down when you have multipe guests), so you wouldn’t normally use it — but if something goes wrong with the service, you’ll be glad you have it.

To summarize, I generally prefer services that record locally, but in some cases Cleanfeed can be a good alternative.

Is there video?

Here I am not talking about recording video — I am talking about being able to see other participants while recording an audio-only podcast with them.

While video is not required for recording podcasts, it’s very nice to have. In addition to being generally helpful for establishing rapport, here are some specific situations in which I’ve found it useful:

When there are issues with guest’s audio (like bad mic positioning), they are much easier to diagnose and address when I can see what’s going on on their side.
When the guest speaks, I can acknowledge their words with simple nods, instead of polluting the audio with regular uh-huh’s. When they or I need a bit of time to think on what to say, we don’t need to use filler sounds to let other person know we haven’t dropped off.
My co-host and I can use small gestures to coordinate who’s asking the next question.

Can the guest record on a mobile device?

When the guest doesn’t have an external microphone or a headset, a smartphone’s microphone can potentially give much better results than a laptop’s built-in microphone, though I haven’t tried this in practice yet.

Specific services

Here’s a list of remote-recording services you can try:

Out of these:

Cleanfeed records audio on their servers; all the other ones record locally. Welder offers simultaneous local and server-side recording (the latter considered a backup).
SquadCast, Zencastr, Riverside, Remotely, and Welder have video during recording.
I’ve only used Zencastr, Cast, and SquadCast for a significant amount of time. I left Zencastr and Cast because of audio issues — subtle ones in the case of Cast, and no so subtle ones in the case of Zencastr. I’m happy with the quality of SquadCast recordings so far.
Also, with both Cast and Zencastr I had (very rare) cases when the service would not work for a guest. I am yet to encounter such problem with SquadCast.
Ennuicastr is the only one that is open source. It is currently in beta and lacks video, but other than that it looks very promising.

The only service I would definitely advise against is RINGR:

This was the only service I couldn’t make work at all.
Despite advertising a free trial, they charged me and never refunded, even after my explicit request.
Their Android app requires a lot of permissions, including your identity, contacts, files, Wi-Fi connection information, and device ID and call information.

How to pass a logical matrix to RStan

Fri, 24 Aug 2018 20:00:00 +0000

I ran into this problem with RStan today, and I’m sure I’ve had it at least once in the past, and it took me a ridiculously long time to figure out, so I’m documenting it here.

Suppose you need to pass a multidimensional logical array to RStan (events in the example below). The details of the model don’t matter; let’s say it’s a simple beta-binomial model:

model <- stan_model(auto_write = T, model_code = "
data {
  int<lower=1> m;
  int<lower=1> n;
  int<lower=0,upper=1> events[m,n];
}
parameters {
  real<lower=0> a;
  real<lower=0> b;
  vector<lower=0,upper=1>[m] theta;
}
model {
  theta ~ beta(a,b);
  for (i in 1:m) {
    sum(events[i]) ~ binomial(n, theta[i]);
  }
}
")

The corresponding R code might look like this:

library(rstan)

model <- stan_model("model.stan", auto_write = T)
a <- 3
b <- 5
n <- 20
m <- 30
events <- t(sapply(1:m,
  function(i) {
    p <- rbeta(1, a, b)
    as.logical(rbinom(n, 1, p))
  }))

sampling(model, data = list(m = m, n = n, events = events))

But this results in the following error:

Error in new_CppObject_xp(fields$.module, fields$.pointer, ...) : 
  Exception: mismatch in number dimensions declared and found in context; processing stage=data initialization; variable name=events; dims declared=(30,20); dims found=(600)  (in 'model3e955026c5b4_model' at line 4)
## failed to create the sampler; sampling not done

The important bits are declared=(30,20); dims found=(600). It looks like the events matrix got flattened… but why? We can double check that events is still a matrix of the correct dimensions:

class(events)

## [1] "matrix"

dim(events)

## [1] 30 20

Here’s what’s going on. Because events in R is a logical (i.e. boolean) matrix, and events in Stan is an integer two-dimensional array, RStan tries to convert the former to the latter by calling as.integer. And indeed, we can check that as.integer flattens the matrix.

So how do you convert a logical matrix to an integer one? There is certainly more than one way to do it, but a quick and dirty solution, which I found in the archives of r-help, is to add a 0 to it. So:

sampling(model, data = list(m = m, n = n, events = events + 0))

Estimating the fraction of recurring events

Wed, 22 Aug 2018 20:00:00 +0000

Sometimes you have a collection of events and the number of times each of them occurred. Each event is either unique, i.e. occurring only once, or recurring. You don’t know which events are unique or recurring, but you want to estimate the fraction of recurring events.

I’ll give two examples, one from everyday life and one from bioinformatics.

Everyday life example. Let’s say you’re a small business (a barbershop, a community theater, an electrical contractor etc.) People have to call you in order to reserve seats or make appointments, so you know the names of all your past customers and how many times they ordered. Naturally, some clients enjoyed the experience and will become your recurring customers, while others didn’t like it and will never return. You’d like to know how many recurring customers you have.

Why not do the obvious thing and calculate the fraction of people who ordered twice or more? Because some fraction of the customers who have not reordered are also recurring customers—they just haven’t had a chance or need to reorder yet. Of course, based on our data, we cannot possibly tell which of those customers will reorder; but we can nevertheless estimate their fraction.

Bioinformatics example. (Feel free to skip this paragraph if you’re not into bioinformatics!) You are looking at ChIP-seq peaks in $n$ replicates. For each peak, you can calculate how many replicates the peak was found in. You want to estimate the number of replicable peaks. Again, while you could just count the peaks that were found in 2 or more replicates, some of the replicable peaks are expected to have only occurred once, and it would be a shame to miss them.

The key to this problem is the following observation. The set of unique events (customers who ordered once or peaks present in only one replicate) is “contaminated”: it consists of a mixture of recurring and non-recurring events. But the set of events that occured two or more times must only consist of recurring events. We can use these definitely recurring events to estimate the rate with which events recur, and from there find the expected number of recurring events that only occurred once.

To be more specific, let’s denote the total number of recurring events as $m$, and the number of recurring events that occurred $k$ times as $m_k$ $(k=1,2,\ldots)$. Then we don’t know $m_1$ (because we don’t know how many truly unique events there are), but we do know $m_2, m_3, \ldots$ What we would like to estimate is $m = m_1 + m_2 + \ldots$ (Notice that I didn’t include $m_0$ here; see below on why.)

The natural models for $m_k$ are the binomial and Poisson distributions. In the ChIP-seq example, where there is a clear upper bound on $m_k$—the number of replicates, $n$—the binomial model is more appropriate. The binomial model could also be applied in the case of a community theater that has only given a small number of performances to date. In other cases, when the upper bound is absent or large, the Poisson model is a better choice. There are other models one could apply here (including a negative binomial model or even nonparametric models), but I won’t cover them here; see Bunge et al. (2014) for inspiration.

Regardless of the model, we expect $m_k$ for high $k$ to be less informative because their chances of being non-zero become very low. So it is natural to use $m_2$ and $m_3$ for estimation. (Because the negative binomial distribution has an extra parameter, one would also need an extra data point, $m_4$. This is why the negative binomial model should only be used when the total number of observations, $\sum k\cdot m_k$, is large.)

The binomial model

In the binomial model with parameters $n$ and $\theta$, the probabilities of $k=2$ and $k=3$ repetitions for a particular event are:

\[\begin{align*} p\{k=2\} = {n\choose 2} \theta^2(1-\theta)^{n-2}; \\ p\{k=3\} = {n\choose 3} \theta^3(1-\theta)^{n-3}. \end{align*}\]

Dividing the second equation by the first gives

\[ \frac{p\{k=3\}}{p\{k=2\}}=\frac{n-2}3\frac{\theta}{1-\theta}, \]

from which we get

\[ \theta = \frac{1}{1+\frac{n-2}3\frac{p\{k=2\}}{p\{k=3\}}}. \]

Next, we observe that $p\{k=2\}/p\{k=3\}$ can be estimated as $m_2/m_3$, so

\[ \theta \approx \frac{1}{1+\frac{n-2}3\frac{m_2}{m_3}}. \]

Then our quantity of interest is

\[ \begin{split} m & = m_1+m_2+\ldots+m_n \\ & = \frac{m_2}{p\{k=2\}}\cdot p\{k\geq 1\}\\ & = \frac{2m_2(1-\theta^n)}{n(n-1)\theta^2(1-\theta)^{n-2}}. \end{split} \]

Let’s test this using a simulation.

set.seed(2018)
# number of unique events
u <- 40000
# number of recurring events (to be re-calculated later)
m <- 30000
# n, the maximum occurrences of a single event
n <- 15
# rate of occurrence of an event
theta <- 0.23
# occurrence data
occs <- rbinom(m, n, theta)
# remove zero occurrences, as they are unobservable
occs <- occs[occs != 0]
(m <- length(occs))

## [1] 29406

# mix in the non-recurring events
occs <- sample(c(occs, rep(1, u)))
# m_k, number of events that occurred k times
(m_k <- table(occs))

## occs
##     1     2     3     4     5     6     7     8     9    10    11 
## 42689  5581  7310  6367  4199  2139   800   235    66    17     3

# now, apply our estimation algorithm:
(theta_est <- 1/(1+(n-2)/3 * m_k[["2"]]/m_k[["3"]]))

## [1] 0.2321052

(m_est <- (2*m_k[["2"]]*(1-theta_est**n)) /
        (n*(n-1)*theta_est**2*(1-theta_est)**(n-2)))

## [1] 30565.44

So we estimated the occurrence rate $\theta$ as 0.232 (true value 0.23) and the number of observed recurring events as 30565 (true value 29406). Pretty good. For smaller sample sizes, the estimates will be not as accurate but hopefully still useful.

The Poisson model

The derivation is analogous for the Poisson model.

\[\begin{align*} p\{k=2\} = \frac{\lambda^2e^{-\lambda}}2; \\ p\{k=3\} = \frac{\lambda^3e^{-\lambda}}6; \end{align*}\] \[ \frac{p\{k=3\}}{p\{k=2\}}=\frac{\lambda}3; \] \[ \begin{split} \lambda & = 3\cdot \frac{p\{k=3\}}{p\{k=2\}} \\ & \approx 3\cdot\frac{m_3}{m_2}; \end{split} \] \[ \begin{split} m & = m_1+m_2+\ldots+m_n \\ & = \frac{m_2}{p\{k=2\}}\cdot p\{k\geq 1\}\\ & = \frac{2m_2(e^{\lambda}-1)}{\lambda^2}. \end{split} \]

Simulation:

set.seed(2018)
# number of unique events
u <- 40000
# number of recurring events (to be re-calculated later)
m <- 30000
# rate of occurrence of an event
lambda <- 1.6
# occurrence data
occs <- rpois(m, lambda)
# remove zero occurrences, as they are unobservable
occs <- occs[occs != 0]
(m <- length(occs))

## [1] 23882

# mix in the non-recurring events
occs <- sample(c(occs, rep(1, u)))
# m_k, number of events that occurred k times
(m_k <- table(occs))

## occs
##     1     2     3     4     5     6     7     8     9 
## 49744  7669  4090  1667   521   145    31    12     3

# now, apply our estimation algorithm:
(lambda_est <- 3 * m_k[["3"]]/m_k[["2"]])

## [1] 1.599948

(m_est <- (2*m_k[["2"]]*(exp(lambda_est)-1)) / lambda**2)

## [1] 23682.68

Again, extremely accurate for large sample sizes: we estimated the occurrence rate as 1.599948 (true value 1.6) and the number of observed recurring events as 23682.68 (true value 23882).

What about $m_0$?

In the above calculations, I did not include $m_0$ (the numer of unobserved but recurring events) in the definition of $m$. While it is trivial to include $m_0$, there are good reasons not to.

The first reason is practical: $m$ as defined can be used to estimate the fraction of recurring events (vs. unique ones); just divide $m$ by the total number of observed events.

The other reason is theoretical: it is not even clear what $m_0$ means. I like how Bunge et al. (2014) put it:

Consider the following metaphor or allegory (Böhning & Schön 2005). We go to a freeway interchange and record the numbers of occupants, including the drivers, of the passing cars, for a fixed period of time, e.g., during the daylight hours of a given day. In principle, we could then extract frequency count data: $f_1$ = the number of cars with one occupant, $f_2$ = the number with two, and so forth. From this data, we could in principle compute a statistical estimate of $f_0$ , the number of cars with zero occupants, i.e., cars not presently being driven, and hence the total number of cars in existence, but (apart from many other potential objections to this procedure) cars in existence where? That is, what is the target population? Is it all cars that might pass by our interchange, which could be all cars in North America, or just some local subset thereof? The answer is unclear. The reader may object that no one would carry out such a study, but it is not too different from the ocean-sampling example discussed above. In that setting, what is the microbial population under study? All microbial organisms that may enter the 1-liter collection bottle, or some more restricted subpopulation?

References

A very similar problem arises in microbial ecology; an excellent review on this is:

John Bunge, Amy Willis, Fiona Walsh. (2014) Estimating the Number of Species in Microbial Diversity Studies.

logit() and logistic() functions in R

Sat, 11 Aug 2018 20:00:00 +0000

In statistics, a pair of standard functions logit() and logistic() are defined as follows:

\[ \begin{align*} \operatorname{logit}(p) &= \log\frac{p}{1-p}; \\ \operatorname{logistic}(x) &= \frac{1}{1+\exp(-x)}. \end{align*} \]

Given the ubiquity of these functions, it may be puzzling and frustrating for an R user that there are no pre-defined functions logit() and logistic() in R.

> logit(0.6)
Error in logit(0.6) : could not find function "logit"

Some CRAN packages define this function, and some users even import these packages for the sole reason to have access to logit() and/or logistic(). Others define them directly via log() and exp().

All this is unnecessary: the standard stats package actually defines these functions, just under different names. logit() and logistic() are the quantile and cumulative distribution functions for the logistic distribution, so in line with R’s conventions for probability distributions, they are called qlogis() and plogis(), respectively.

> qlogis(0.6) # logit
[1] 0.4054651
> plogis(0.4054651) # logistic
[1] 0.6

If you still prefer to use the familiar names logit() and logistic() in your code, simply include in your script

logit <- qlogis
logistic <- plogis

One advantage of using these standard functions over defining your own or importing some random CRAN package is that they can work on the log scale, by setting log.p = TRUE.

> qlogis(log(0.6), log.p = TRUE)
[1] 0.4054651

Working with log-probabilities is often necessary to avoid underflow, and implementing logit() and logistic() that work accurately on the log scale is not trivial; see Accurately Computing $\log(1-\exp(-|a|))$ for the details. You can see how these functions are implemented in R in src/nmath/qlogis.c and src/nmath/plogis.c.

And while we’re at it, here’s another trick. Let’s say you are working with probabilities very close to 1. Then the appropriate representation is not $\log p$ but $\log (1-p)$. To convert a probability from a log-complement scale to the logit scale, use lower.tail = FALSE:

> qlogis(-3, log.p = TRUE, lower.tail = FALSE)
[1] 2.948931

Can you cheat the Brier score?

Thu, 09 Aug 2018 20:00:00 +0000

The Brier score is a common way of judging probabilistic forecasts. If you have several people or teams each giving probabilities to various events, you can judge how well they are doing by comparing their Brier scores: lower scores correspond to more accurate predictions.

However, there is a critical assumption behind the Brier score: that all forecasters compete on the same set of events. Some questions are easier to predict than others. And I’m not talking about the domain-specific difficulty—the sort of difficulty that arises when one wants to predict whether there will be a revolution in Turkey without knowing much about the Turkish politics.

It turns out that the difficulty of achieving a certain Brier score is directly related to the actual probability of the event. Intuitively we already understand that: it is easier to “predict” whether a plane will crash (it probably won’t) than to predict whether it will arrive late (who knows?).

Except here we are dealing with probabilistic predictions (not a rigid yes or no) and a specific scoring rule (the Brier score), so our intuitions do not necessarily transfer. And yet the basic result is the same: it is easier to achieve a low (i.e. good) Brier score betting on highly probable and highly improbable events than betting on uncertain events.

What made me look into this was this episode of the Rationally Speaking podcast, where Julia Galef speaks with Anthony Aguirre about his prediction platform Metaculus.

At one point the dialogue goes as follows:

Julia:	Oh, how do you measure how well it works?
Anthony:	[…] So looking over sort of the last half year or so, since December 1st, for example… If you ask for how many predictions was Metaculus on the right side of 50% — above 50% if it happened or below 50% if it didn’t happen — that happens 77 out of 81 times the question resolved, so that’s quite good. And some of the aficionados will know about Brier scores. That’s sort of the fairly easy to understand way to do it, which is that you assign a zero if something doesn’t happen, and a one if something does happen. Then you take the difference between the predicted probability and that number. So if you predict at 20% and it didn’t happen, you’d take that as a .2, or if it’s 80% and it does happen and that’s also a .2, because it’s a difference between the 80% and a one, and then you square that number. So Brier scores can run from basically zero to one, where low numbers are good. And if you calculate that for that same set of 80 questions, it’s .072, which is a pretty good score.
Julia:	Yeah, I mean I don’t really have a frame of reference — like, I don’t know what, an average college educated person guessing about these questions would be expected to guess, or to get for their Brier score. But it certainly seems low.
Anthony:	So that’s something that you can sort of … So, some references are a .25 is what you would get if you guessed 50% for everything. So that’s sort of totally uncertain. A .33 is what you would get if you just randomly assigned a probability between zero and one for everything. It’s slightly different things.

If you pay attention, there’s already a hint of the problem in this short exchange. You can get a Brier score of .25 quite easily by always predicting 50%. But if the true probability is indeed 50%, then you have to predict 50%—any other prediction will get you an even worse score.

Say I set up a platform analogous to Metaculus—I’ll call it Metaculus Prime—and want to beat Metaculus’s Brier score. All I’d need to do is to formulate slightly different versions of Metaculus questions. Currently, the #1 question on Metaculus is this: “Will SpaceX land people on Mars prior to 2030?” On Metaculus Prime, I’ll also allow betting on SpaceX landing on Mars, except I’ll ask “Will SpaceX land people on Mars prior to 2020?”. Because the answer to this modified question is more certain, a person of the same predictive skill would be expected to score better on it than on the original, less certain question.

When predicting the more certain events, one still has an incentive to get the exact probability right (to the extent such a thing exists). But for a given difference between the true and predicted probabilities, there is an extra penalty for probabilities close to 0.5, as I demonstrate below.

To be clear, this is not to criticize Anthony—it was really interesting for me to find out Metaculus’s consensus Brier score, and it doesn’t look at all like he’s manipulating the questions to get a low Brier score. But we should also recognize that the Brier score is not a valid metric to compare forecasters who compete on different questions or different platforms.

It is also interesting to think about how we should design the prediction questions. In a question like “Will the number of electric cars in the world be greater than $X$ by the end of 2020”, how should we pick $X$? If $X$ is very high or very low, the outcome will be obvious, and there will also be an incentive to make more extreme predictions (closer to 0 or 1 than warranted) since you’re unlikely to be proven wrong in the short term.

On the other hand, pick $X$ “right in the middle”, and the best Brier score anyone can achieve on average becomes the unimpressive 0.25.

Mathematical details

Consider an event $X$ that may happen $(X=1)$ with probability p and not happen $(X=0)$ with probability $1-p$. Say you predict the probability of $X$ happening being $q$. Then the Brier score is calculated as $(X-q)^2$. For multiple events, take the average of the individual Brier scores.

The expectation of $(X-q)^2$ is

\[\begin{equation} \operatorname{E}[(X-q)^2] = (p-q)^2+\operatorname{Var}(X). \label{eq1} \end{equation}\]

For a given event $X$, $\operatorname{Var}(X)$ is a constant, and to minimize the Brier score you should pick $q=p$.

But if you have a choice of which events $X$ to bet on, it also makes sense to try and minimize the second term, $\operatorname{Var}(X)$. The variance of a binary (“Bernoulli”) variable $X$ is given by $\operatorname{Var}(X)=p(1-p)$. It is maximal when $p=0.5$ and falls as $p$ moves towards 0 or 1.

Additional considerations

There are two other considerations that exacerbate this effect. First is that a given absolute error in probabilities—say, 5 percentage points—is actually more severe for probabilities closer to 0 or 1, so if anything, the errors should be penalized more for highly certain events, not less. This could be seen, for instance, on the log-odds scale: the difference between 0.6 and 0.65 is only 0.21, while the difference between 0.9 and 0.95 is 0.75.

The other consideration is that if I tend to assign the probability of 1 for 99% events—a serious error, if you ask me—it will take quite a few predictions to prove me wrong. In the short term, I will actually be rewarded for making more “certain” forecasts for rare events.

Most of this article assumes that there exists some inherent uncertainty in the predicted event. The alternative view is that most of events (barring quantum experiments and chaotic systems) can be predicted exactly by a skilled enough team and given enough resources. While in this case it doesn’t make sense to talk about the true probability of an event, the fact that different events may be of different difficulty becomes even more obvious.

Updates from 2021

Probability of a regex occurrence

Wed, 01 Aug 2018 20:00:00 +0000

Given a regular expression $R$ and a random string $L$ of length $n$, what is the probability that $R$ matches somewhere in $L$?

This seems like an intimidating problem at first, but I was surprised how easy it becomes after a couple of insights.

The first insight is to turn a regex into a DFA. This moves us from “how can I possibly know whether a regex matches?” to “feed the string to the automaton and see if it ends up in an accepting state”.

Still, the DFA can be anything at all — how do we know the probability of ending up in an accepting state? The second insight is to ascribe probabilities to the edges (transitions) of the DFA and observe that this turns it into a Markov chain¹.

Why is that helpful? Because a Markov chain can be represented by its transition matrix $P$, and the probability of getting from any state $a$ to any state $b$ can be computed in logarithmic time by computing $P^n_{ab}$, the $a,b$-th element of the $n$-th power of the matrix $P$.

So how do we assign the probabilities to DFA transitions? If all characters are equally likely, $P_{ab}$ is simply the number of characters that move the DFA from state $a$ to state $b$ divided by the number of all possible characters. More generally, if every letter $l$ has a fixed probability $p_l$, we compute \[ P_{ab}=\sum_{a\overset{l}{\rightarrow} b}p_l, \] where the summation is over all characters $l$ that lead from $a$ to $b$.

And to get the probability of the regex $R$ matching anywhere inside the string as opposed to matching the whole string, replace $R$ with ${.}{*}R{.}{*}$ before compiling it to a DFA. Then calculate

\[ \sum_{f\in F} P^n_{sf}, \] where $s$ is the start state of the DFA, and $F$ is the set of accepting states.

Here’s a simple implementation of this algorithm. (The library code supports full regular expressions over arbitrary finite alphabets, but the command line tool only accepts simple IUPAC patterns over the DNA alphabet.)

References

There is nothing new or original here — specialists know how to solve this and even more involved problems. For these more involved problems, generating functions are typically used. It’s nice, however, that for this specific case simple matrix exponentiation suffices.

For example, Nicodème, Salvy and Flajolet (2002) show how to use generating functions to calculate the probability distribution of the number of occurrences of a regex, both overlapping and non-overlapping, as well as the mean and variance of that number. They also derive the asymptotics and handle (first-order) Markov dependencies between the consequtive positions in $L$. But for our task — the probability of a regex occurring at least once — their method essentially reduces to the one described here.

Régnier (2000) also uses generating functions but explicitly models possible overlaps between patterns, which allows her to analyze regular expressions more efficiently than Nicodème et al. (see page 20).

Filion 2017 is a very accessible introduction to the paradigm of transfer graphs, which are a bridge between DFAs, transition matrices, and generating functions.

Here I assume that different positions in $L$ are independent.↩︎

Combine ChIP-seq peaks from multiple replicates via consensus voting

Wed, 11 Jul 2018 20:00:00 +0000

As in most of biology, having biological replicates in ChIP-seq experiments is important to ensure external validity.

Or, in simpler words, your low-q-value peak from a single sample only means that the peak was definitely there in this one sample. Without replicates, you don’t know if this peak is reliably there across samples or was specific to that one sample at that one time.

So, in order to know that your results are reliable and can be expected to replicate in other similar experiments, you should replicate them yourself on different biological samples. The more diverse your replicates, the more externally valid your findings. For instance, if all your replicates come from the same inbred strain, you can conclude that your findings probably generalize to most individuals in that strain but may be caused by the strain’s genotype and not generalize to the whole population.

But say you do have biological replicates — how do you analyze them? While tools for differential analysis of peaks typically know how to deal with replicates, tools for peak calling (such as MACS) deal only with a single replicate at a time. This is a problem if you are only interested in finding transcription factor binding sites.

Often people seem to resort to either intersecting or taking the union of all peaks just because it’s very easy to do using bedtools. However, as Yang et al. argue, doing so will result in high false negative and false positive rates, respectively. Instead, they propose a compromise: call a consensus peak if it’s supported by at least a certain number of replicates — say, by at least half of them. But unlike simple union or intersection, there do not appear to be any ready-to-use tools to perform this kind of analysis.

So here I’ll explain how to use R/Bioconductor to find regions that are covered by at least a given number of replicate peaks. Note that my approach does not correspond directly to any of the 4 approaches described by Yang et al. Their method results in a smaller number of wider peaks, but parts of those consensus peaks may have smaller coverage as shown below.

This could be a problem when peaks spanning hundreds or thousands of bp overlap only a little. A better approach would be to first to identify regions supported by the given number of replicates and then optionally merge those that are close to each other, which I’ll also show how to do.

Of course this is very subjective and arbitrary, and ideally we would use a robust statistical model instead of a hand-picked threshold. And there are some tools that attempt this:

I discussed GenoGAM with its authors last year, but I am yet to study and compare the other ones.

Calling consensus peaks in Bioconductor

First, load the Bioconductor libraries to work with genomic ranges:

library(rtracklayer)
library(GenomicRanges)

Get the list of .narrowPeak files in the current directory. We’ll treat them all as replicates. The same code should work with .bed or .gff files.

peak_files <- list.files(pattern = ".*\\.narrowPeak")
peak_files

## [1] "NRF2SFN-Chorley-rep1_bw300_narrow_peaks.narrowPeak"
## [2] "NRF2SFN-Chorley-rep2_bw300_narrow_peaks.narrowPeak"
## [3] "NRF2SFN-Chorley-rep3_bw300_narrow_peaks.narrowPeak"

Read these files using rtracklayer’s import function:

peak_granges <- lapply(peak_files, import)
peak_granges

## [[1]]
## GRanges object with 1415 ranges and 6 metadata columns:
##                seqnames          ranges strand |                                        name
##                   <Rle>       <IRanges>  <Rle> |                                 <character>
##      [1]   NC_000001.11     10041-10186      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_1
##      [2]   NC_000001.11   629597-629707      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_2
##      [3]   NC_000001.11   630278-630444      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_3
##      [4]   NC_000001.11   631154-631304      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_4
##      [5]   NC_000001.11   631961-632210      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_5
##      ...            ...             ...    ... .                                         ...
##   [1411] NW_017852933.1 1174567-1174677      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1411
##   [1412] NW_017852933.1 1593093-1593226      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1412
##   [1413] NW_018654708.1   221820-221930      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1413
##   [1414] NW_018654713.1     29933-30043      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1414
##   [1415] NW_019805500.1     69753-69895      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1415
##              score signalValue    pValue    qValue      peak
##          <numeric>   <numeric> <numeric> <numeric> <integer>
##      [1]        27     5.17033   6.91041   2.79078        38
##      [2]        66     3.80497  11.22005   6.63982        82
##      [3]        93     4.31874   14.1121   9.31245        98
##      [4]        50     3.50501   9.46955   5.06445        99
##      [5]        77     4.25812  12.40796   7.76729       174
##      ...       ...         ...       ...       ...       ...
##   [1411]        31     4.58205   7.31183   3.15002        63
##   [1412]       158    10.99692  21.19611  15.89375        50
##   [1413]        60     5.71026  10.64406   6.09574        23
##   [1414]        53     6.15752   9.80128   5.37605        17
##   [1415]        47     5.49846   9.13087    4.7435        78
##   -------
##   seqinfo: 81 sequences from an unspecified genome; no seqlengths
## 
## [[2]]
## GRanges object with 10073 ranges and 6 metadata columns:
##                 seqnames        ranges strand |                                         name
##                    <Rle>     <IRanges>  <Rle> |                                  <character>
##       [1]   NC_000001.11    9913-10101      * |     NRF2SFN-Chorley-rep2_bw300_narrow_peak_1
##       [2]   NC_000001.11 118531-118688      * |     NRF2SFN-Chorley-rep2_bw300_narrow_peak_2
##       [3]   NC_000001.11 175750-175907      * |     NRF2SFN-Chorley-rep2_bw300_narrow_peak_3
##       [4]   NC_000001.11 180786-181117      * |     NRF2SFN-Chorley-rep2_bw300_narrow_peak_4
##       [5]   NC_000001.11 295097-295254      * |     NRF2SFN-Chorley-rep2_bw300_narrow_peak_5
##       ...            ...           ...    ... .                                          ...
##   [10069] NW_019805499.1   41689-41864      * | NRF2SFN-Chorley-rep2_bw300_narrow_peak_10069
##   [10070] NW_019805499.1   42535-42692      * | NRF2SFN-Chorley-rep2_bw300_narrow_peak_10070
##   [10071] NW_019805500.1   15499-15656      * | NRF2SFN-Chorley-rep2_bw300_narrow_peak_10071
##   [10072] NW_019805501.1   42374-42531      * | NRF2SFN-Chorley-rep2_bw300_narrow_peak_10072
##   [10073] NW_019805503.1   32183-32340      * | NRF2SFN-Chorley-rep2_bw300_narrow_peak_10073
##               score signalValue    pValue    qValue      peak
##           <numeric>   <numeric> <numeric> <numeric> <integer>
##       [1]        56     6.82969  10.16471   5.62669       138
##       [2]        30     3.84819   7.00974   3.01917        96
##       [3]        15      2.8959    5.1229   1.53902        65
##       [4]        64     7.20184  11.08078   6.47703        95
##       [5]        25     3.75744   6.16293   2.50373       104
##       ...       ...         ...       ...       ...       ...
##   [10069]        67     5.79181  11.53681   6.76819        34
##   [10070]        30      3.8263   6.76788   3.00398        61
##   [10071]        24     3.74499   6.07139   2.41671       109
##   [10072]        30      3.8263   6.76788   3.00398       115
##   [10073]        30     3.86121   7.17009   3.01917        23
##   -------
##   seqinfo: 250 sequences from an unspecified genome; no seqlengths
## 
## [[3]]
## GRanges object with 5834 ranges and 6 metadata columns:
##                seqnames        ranges strand |                                        name
##                   <Rle>     <IRanges>  <Rle> |                                 <character>
##      [1]   NC_000001.11   75561-75825      * |    NRF2SFN-Chorley-rep3_bw300_narrow_peak_1
##      [2]   NC_000001.11 118452-118776      * |    NRF2SFN-Chorley-rep3_bw300_narrow_peak_2
##      [3]   NC_000001.11 295059-295342      * |    NRF2SFN-Chorley-rep3_bw300_narrow_peak_3
##      [4]   NC_000001.11 376867-377338      * |    NRF2SFN-Chorley-rep3_bw300_narrow_peak_4
##      [5]   NC_000001.11 474500-474783      * |    NRF2SFN-Chorley-rep3_bw300_narrow_peak_5
##      ...            ...           ...    ... .                                         ...
##   [5830] NW_019805495.1 242418-242709      * | NRF2SFN-Chorley-rep3_bw300_narrow_peak_5830
##   [5831] NW_019805495.1 258219-258668      * | NRF2SFN-Chorley-rep3_bw300_narrow_peak_5831
##   [5832] NW_019805496.1 127568-127832      * | NRF2SFN-Chorley-rep3_bw300_narrow_peak_5832
##   [5833] NW_019805499.1   30032-30296      * | NRF2SFN-Chorley-rep3_bw300_narrow_peak_5833
##   [5834] NW_019805500.1     7260-7524      * | NRF2SFN-Chorley-rep3_bw300_narrow_peak_5834
##              score signalValue    pValue    qValue      peak
##          <numeric>   <numeric> <numeric> <numeric> <integer>
##      [1]        30     4.46341   6.72273   3.00157        88
##      [2]        76     7.14145  12.01206   7.62227        66
##      [3]        57     6.52364   9.85515   5.70239        27
##      [4]       225     14.2829  28.09008  22.54761       253
##      [5]        55      6.4522   9.65794    5.5344       181
##      ...       ...         ...       ...       ...       ...
##   [5830]        21     4.07727   5.38735     2.108       182
##   [5831]        42     5.28269   8.11069   4.24385       249
##   [5832]        43     5.35609   8.42216   4.39757        73
##   [5833]        42     5.28269   8.11069   4.24385       104
##   [5834]        30     4.46341   6.72273   3.00157       180
##   -------
##   seqinfo: 170 sequences from an unspecified genome; no seqlengths

peak_ranges is a simple R list with 3 elements, which individually are GRanges objects. Now let’s turn this list into a GRangesList, which in this case is more convenient to work with.

peak_grangeslist <- GRangesList(peak_granges)
peak_grangeslist

## GRangesList object of length 3:
## [[1]] 
## GRanges object with 1415 ranges and 6 metadata columns:
##                seqnames          ranges strand |                                        name
##                   <Rle>       <IRanges>  <Rle> |                                 <character>
##      [1]   NC_000001.11     10041-10186      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_1
##      [2]   NC_000001.11   629597-629707      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_2
##      [3]   NC_000001.11   630278-630444      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_3
##      [4]   NC_000001.11   631154-631304      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_4
##      [5]   NC_000001.11   631961-632210      * |    NRF2SFN-Chorley-rep1_bw300_narrow_peak_5
##      ...            ...             ...    ... .                                         ...
##   [1411] NW_017852933.1 1174567-1174677      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1411
##   [1412] NW_017852933.1 1593093-1593226      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1412
##   [1413] NW_018654708.1   221820-221930      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1413
##   [1414] NW_018654713.1     29933-30043      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1414
##   [1415] NW_019805500.1     69753-69895      * | NRF2SFN-Chorley-rep1_bw300_narrow_peak_1415
##              score signalValue    pValue    qValue      peak
##          <numeric>   <numeric> <numeric> <numeric> <integer>
##      [1]        27     5.17033   6.91041   2.79078        38
##      [2]        66     3.80497  11.22005   6.63982        82
##      [3]        93     4.31874   14.1121   9.31245        98
##      [4]        50     3.50501   9.46955   5.06445        99
##      [5]        77     4.25812  12.40796   7.76729       174
##      ...       ...         ...       ...       ...       ...
##   [1411]        31     4.58205   7.31183   3.15002        63
##   [1412]       158    10.99692  21.19611  15.89375        50
##   [1413]        60     5.71026  10.64406   6.09574        23
##   [1414]        53     6.15752   9.80128   5.37605        17
##   [1415]        47     5.49846   9.13087    4.7435        78
## 
## ...
## <2 more elements>
## -------
## seqinfo: 299 sequences from an unspecified genome; no seqlengths

Now, find the genome regions which are covered by at least 2 of the 3 sets of peaks:

peak_coverage <- coverage(peak_grangeslist)
peak_coverage

## RleList of length 299
## $NC_000001.11
## integer-Rle of length 248946484 with 3023 runs
##   Lengths:   9912    128     61     85  65374    265 ... 220661    275    137     85    197     84
##   Values :      0      1      2      1      0      1 ...      0      1      2      1      2      1
## 
## $NC_000002.12
## integer-Rle of length 242183602 with 1773 runs
##   Lengths:  214478     265  588943     158     260 ...     288   50217     103     159      28
##   Values :       0       1       0       1       0 ...       1       0       1       2       1
## 
## $NC_000003.12
## integer-Rle of length 198174065 with 1492 runs
##   Lengths:    9884     259      73     209     217 ...     158   98693     221    4926     294
##   Values :       0       1       2       3       2 ...       1       0       1       0       1
## 
## $NC_000004.12
## integer-Rle of length 190204610 with 1998 runs
##   Lengths:   9848     94     53    243     44     62 ...   1005     11    162    106  23142    170
##   Values :      0      1      2      3      2      1 ...      0      1      2      1      0      1
## 
## $NC_000005.10
## integer-Rle of length 181450116 with 1874 runs
##   Lengths:    9901       7      97    1388      54 ...     112     158     202    6807    1286
##   Values :       0       1       2       3       2 ...       1       2       1       0       1
## 
## ...
## <294 more elements>

peak_coverage is a list of numbers for every single genome position, encoded into an efficient structure Rle. Now, using the slice function, let’s get the regions in the genome where coverage is at least 2:

covered_ranges <- slice(peak_coverage, lower=2, rangesOnly=T)
covered_ranges

## IRangesList of length 299
## $NC_000001.11
## IRanges object with 154 ranges and 0 metadata columns:
##             start       end     width
##         <integer> <integer> <integer>
##     [1]     10041     10101        61
##     [2]    118531    118688       158
##     [3]    295097    295254       158
##     [4]    377069    377226       158
##     [5]    709516    709673       158
##     ...       ...       ...       ...
##   [150] 228636089 228636238       150
##   [151] 243037096 243037253       158
##   [152] 244523624 244523802       179
##   [153] 248945982 248946118       137
##   [154] 248946204 248946400       197
## 
## ...
## <298 more elements>

This is a simple IRangesList object; let’s covert it to a GRanges object:

covered_granges <- GRanges(covered_ranges)
covered_granges

## GRanges object with 1401 ranges and 0 metadata columns:
##                seqnames        ranges strand
##                   <Rle>     <IRanges>  <Rle>
##      [1]   NC_000001.11   10041-10101      *
##      [2]   NC_000001.11 118531-118688      *
##      [3]   NC_000001.11 295097-295254      *
##      [4]   NC_000001.11 377069-377226      *
##      [5]   NC_000001.11 709516-709673      *
##      ...            ...           ...    ...
##   [1397]    NT_187617.1   59572-59732      *
##   [1398] NW_003315930.1   14229-14368      *
##   [1399] NW_003315952.3   57671-57785      *
##   [1400] NW_003315959.1 134807-134820      *
##   [1401] NW_003571050.1 380055-380202      *
##   -------
##   seqinfo: 299 sequences from an unspecified genome; no seqlengths

And we are done! We can also save these regions as a .bed file using rtracklayer’s export:

export(covered_granges, "consensus.bed")

Merging nearby peaks

As I mentioned at the beginning, you might want to merge the peaks that are close to each other to avoid having many small fragments. In Bioconductor, you can do that using the min.gapwidth parameters to reduce. For example, this will merge the peaks that are separated by no more than 30 bp:

reduce(covered_granges, min.gapwidth=31)

## GRanges object with 1379 ranges and 0 metadata columns:
##                seqnames        ranges strand
##                   <Rle>     <IRanges>  <Rle>
##      [1]   NC_000001.11   10041-10101      *
##      [2]   NC_000001.11 118531-118688      *
##      [3]   NC_000001.11 295097-295254      *
##      [4]   NC_000001.11 377069-377226      *
##      [5]   NC_000001.11 709516-709673      *
##      ...            ...           ...    ...
##   [1375]    NT_187617.1   59572-59732      *
##   [1376] NW_003315930.1   14229-14368      *
##   [1377] NW_003315952.3   57671-57785      *
##   [1378] NW_003315959.1 134807-134820      *
##   [1379] NW_003571050.1 380055-380202      *
##   -------
##   seqinfo: 299 sequences from an unspecified genome; no seqlengths

As you can see, this reduced the number of ranges from 1401 to 1379.

Sometimes all you need is to cut adapters

Tue, 19 Jun 2018 20:00:00 +0000

When you see a FastQC box like this, you know that something is wrong.

But what exactly?

Overrepresented sequences may indicate adapters (although, ironically, the Adapter Content is one of the few green check marks here), and duplicated sequences could be removed, but what do you do about the GC levels, per base sequence content, and per tile quality bias?

Here’s the table of overrepresented sequences identified by FastQC:

Sequence	Count	Percentage	Possible Source
GATCGGAAGAGCACACGTCTGAACTCCAGTCACTCTCGCGCATCTCGTAT	17953666	21.724791264966143	TruSeq Adapter, Index 8 (97% over 36bp)
ATCGGAAGAGCACACGTCTGAACTCCAGTCACTCTCGCGCATCTCGTATG	3032064	3.6689419031198587	TruSeq Adapter, Index 8 (97% over 35bp)
GTTCGGAAGAGCACACGTCTGAACTCCAGTCACTCTCGCGCATCTCGTAT	161627	0.19557637074136738	TruSeq Adapter, Index 3 (97% over 34bp)
AATCGGAAGAGCACACGTCTGAACTCCAGTCACTCTCGCGCATCTCGTAT	117432	0.14209831506431633	TruSeq Adapter, Index 3 (97% over 34bp)
ATCGGAAGAGCACACGTCTGAACTCCCGTCACTCTCGCGCATCTCGTATG	98036	0.11862823093914193	TruSeq Adapter, Index 3 (96% over 33bp)
GATAGGAAGAGCACACGTCTGAACTCCAGTCACTCTCGCGCATCTCGTAT	82961	0.10038676268862615	TruSeq Adapter, Index 3 (97% over 34bp)

Examining the reads further clarifies the picture: it’s not just that 25% of reads contain adapter sequences; most of them are adapter sequences right from base 1.

This can already explain a lot of what we’re seeing. If a quarter of the reads are variations of the same fixed string, it will definitely skew the GC content and the per base sequence content distributions. This also explains the tile bias — there was some issue with the library that was localized to a couple of lanes.

But the most miraculous thing is how trimming/removal of these adapter sequences affects the quality score distribution. Even if part of the reads were sequenced from adapter DNA instead of endogenous DNA, you wouldn’t necessarily expect those reads to be of lower quality, would you?

Here’s the per-base sequence quality in the raw data:

The analysis that I’m doing (ChIP-seq peak calling) is not very sensitive to the read length, so I was thinking about trimming the reads to 75bp or so just to avoid those questionable bases at the end.

But look what happens if I simply remove the adapters, without doing any other trimming or quality filtering:

I don’t quite understand why this happens, but a couple of explanations come to mind:

There’s a stretch of As in the second half of the adapter sequence; e.g.

GATCGGAAGAGCACACGTCTGAACTCCAGTCACTCTCGCGCATCTCGTATGCCGTCTTCTGCTTGAAAAAAAAAAAAAAACACGAACATGCCGGTTGAAGCAAGTCTCCTGTATCTGGCCTCCACC
                                                                 ^^^^^^^^^^^^^^^

So it could be this poly(A) segment that leads to errors.

There could also be a common issue that caused both adapter-only fragments and low quality scores, although I don’t have any ideas about what it could be.

In any case, simply trimming the adapter sequence and removing all reads shorter than 40bp fixed pretty much everything:

The only remaining warning simply tells us that not all sequences are of the same length, as we would expect after trimming adapters:

This module will raise a warning if all sequences are not the same length.

[…]

For some sequencing platforms it is entirely normal to have different read lengths so warnings here can be ignored.

For the record, the command I used was

cutadapt --cores={threads} -a GATCGGAAGAGCACACGTCTGAACTCCAGTCAC -o {output} {input} --overlap 4 --minimum-length 40

You can also browse and compare the full FastQC reports before and after adapter trimming.

Encrypted reminders with Nudgemail

Fri, 15 Jun 2018 20:00:00 +0000

I use Trello as my main GTD tool, and it works great for the short-term stuff. Sometimes, however, I have a task that I won’t be able to do until 4 months later. While I could create a Trello card and set its due date to October 20th, I will keep seeing it every time I review my trello board. Each time I’ll be reminded of this relatively unpleasant task, adding up to a significant mental cost.

The perfect solution to this problem is provided by Nudgemail. Here’s how it works: you send an email to nudge@nudgemail.com, put the desired time and/or date in the subject field (e.g. “October 20”), and on that date it sends you the email back.

That has a few problems of its own, however:

There is something creepy about sending an email containing personal information to a random email address. Will they sell it to advertizers?

To be fair, their privacy policy looks very noble:

Nudgemail does not create profiles for our users, public or private. We will not share your email address or any other information transmitted through the System with anyone outside of your personally designated recipients. […] No member of the Nudgemail team or third party, aside from your designated recipients, will have direct access to emails or attachments sent to the Service.

Still feels creepy though.
When I get my email back, I might not even remember the fact that I wrote it. How do I know it was really me, and not someone giving me instructions by pretending to be me from the past?

Easy: sign and encrypt the email with GPG. Here are a few tips:

Send an empty email to plain@nudgemail.com; this will instruct Nudgemail to send you plain-text email instead of HTML ones, which will make it easier to extract the PGP text.
Nudgemail does not support attachments; so use inline PGP instead of PGP/MIME.

If you use Enigmail, it allows you to declare per-recipient rules, where you can instruct it to use your PGP key, force signing and encryption, and disable PGP/MIME as shown below.
For PGP/MIME to work better, make sure your outgoing emails are plain text. (Thunderbird itself has per-recipient rules where you can configure this.)

Here’s how the result looks:

Unfortunately Nudgemail messes with the email’s contents a bit, and it can no longer be automatically decoded, but you can easily copy-paste the PGP part and decrypt it:

QuickCheck vs. SmallCheck

Fri, 25 May 2018 20:00:00 +0000

QuickCheck and SmallCheck are the two classical libraries for property testing in Haskell. There are newer developments, like SmartCheck and Hedgehog, but I haven’t tried them much yet.

So here is my take on the comparative strengths and weaknesses of the current versions of QuickCheck and SmallCheck (2.11.3 and 1.1.4, respectively). Being the maintainer of SmallCheck, I am obviously somewhat biased towards it. But quite often I would prefer QuickCheck to SmallCheck for a specific task. It also sometimes happens that I think QuickCheck would work better in principle for a given task, but in practice SmallCheck is more convenient for that task — usually it has to do with generic instances or failure reporting. So here are the considerations I take into account when choosing between the two.

If you do not know much about QuickCheck and/or SmallCheck, the main difference between them is that QuickCheck generates random test cases, whereas SmallCheck enumerates test cases exhaustively up to some depth. Some (though not all) of the pros and cons come from this fundamental difference in paradigms.

Advantages of QuickCheck

Control over the number of tests/execution time

With SmallCheck, the number of tests is usually exponential in their depth. Say, when enumerating trees (from Data.Tree), depth 5 gives you 189 tests in less then 0.01s, while depth 6 gives 6479 tests in 4s. If you only have the patience to run 1000 tests, you have to settle for 189. QuickCheck, on the other hand, allows you to set the exact number of tests independently from their depth (which is called size in QuickCheck).

Update (2018-07-05). In smallcheck-1.1.5, I added a function

limit :: Monad m => Int -> Series m a -> Series m a

which acts similarly to take for lists. It is still not as convenient as QuickCheck, because you have to apply it to each “big” generator as opposed to setting a single global limit on the number of tests, but you can use it to fight the exponential blowup.

Floating-point numbers

SmallCheck has instances for floating-point numbers, e.g.

> listSeries 5 :: [Positive Float]
[1.0,2.0,0.5,4.0,0.25,8.0,0.125,16.0,3.0,6.25e-2,6.0,1.5,12.0,0.75,24.0,0.375,48.0,0.1875]

These are probably enough if you need to fill some fractional field in a data type, but the logic of your functions doesn’t depend on the number in a complex way.

But if you are testing any truly numerical code (say, matrix operations or physics simulation), it is much more useful to have QuickCheck generate random floating-point numbers.

Advantages of SmallCheck

Generic deriving

If you have a couple of types that rarely change, writing the instances by hand may not a big deal.

Often, especially when writing tests for commercial code, there is a big number of nested algebraic data types. And even if you have the patience to write the instances, they will become incomplete the next time a constructor is added to one of the types (and the compiler won’t warn you).

So it is essential to be able to derive most of the instances. SmallCheck has had generic instances since 2011, thanks to the contribution from Bas van Dijk. QuickCheck still does not have them (github issue).

There is a separate package, generic-arbitrary, but it doesn’t seem to support recursive data types. The recursive types also seem to be the reason why QuickCheck itself doesn’t have the generic implementation.

The solution to the recursive type problem is straightforward: reduce the depth (size) every time you descend into a nested type. As an example, here’s an instance for trees from Data.Tree:

instance Arbitrary a => Arbitrary (Tree a) where
  arbitrary = scale (\n -> max (n-1) 0) $
    Node <$> arbitrary <*> arbitrary

This approach is not ideal (your size may get down to 0 too quickly), but IMO it is better than no generic instances at all.

Update (2018-07-05). There are some third-party packages that implement generic deriving for QuickCheck and support recursive types, in particular testing-feat and generic-random.

testing-feat looks very cool and powerful from a theoretical perspective, and I’d like to study it in depth some day. In practice it involves definitions like

deriveEnumerable ''MyType
instance Arbitrary MyType
  where
    arbitrary = sized uniform
    shrink = genericShrink

Sometimes you need to define an Enumerable instance for an abstract or primitive type, which I ended up doing like this (for Text):

instance Enumerable T.Text where
  enumerate = share (c1 T.pack)

generic-random is a bit easier to use, since there’s no additional Enumerable class. The declaration looks like this:

instance Arbitrary MyType
  where
    arbitrary = genericArbitraryU'
    shrink = genericShrink

In both cases, genericShrink comes from the QuickCheck package itself, but it’s not the default implementation for shrink; the default is shrink _ = [].

Failure reporting

Say we have a property

prop x = reverse (reverse x) == x

By default, both SmallCheck and QuickCheck only tell you the generated values that led to the failure (i.e. x). But I would also like to know why the test failed — in this case, what reverse (reverse x) was.

There is a StackOverflow question about this with two answers, neither of which is perfect.

Edward Z. Yang says:

Because QuickCheck gives you the inputs to the function, and because the code under test is pure (it is, right?), you can just feed those inputs to the function and get the result. This is more flexible, because with those inputs you can also repeatedly test with tweaks to the original function until it’s correct.

There are many reasons why this may not work:

The code may not be pure by design — both SmallCheck and QuickCheck are capable of testing IO code.
The code may be intended to be pure, but the very bug you are trying to fix may lead to it being impure/non-deterministic.
Copy-pasting values may not be very convenient: think large matrices.
Also, strings produced by Show do not always map back to the original values effortlessly. E.g. Maps, Sets, and Vectors all result in fromList [...] when shown. If you have several of them in the same structure, you’ll have to disambiguate those fromLists.

The other answer, by Paul Johnson, gives a working solution (using MkResult), but it is rather unwieldy, and you have to include the parameter values manually if you need them.

Update (2018-07-05). I’ve since found and posted a much better way to report a failure in QuickCheck:

import Test.QuickCheck.Property (succeeded, failed, reason)

prop a b =
  if res /= []
    then succeeded
    else failed { reason = reason }
   where
      (res, reason) = checkCode a b

In SmallCheck, you can return Either String String instead of Bool, where the Strings give explanation for failure and success, respectively. For example, this property

prop_reverse :: [Int] -> Bool
prop_reverse x = reverse x == x

results in the message

there exists [0,1] such that
  condition is false

But we can change the property to explain itself

prop_reverse :: [Int] -> Either String String
prop_reverse x =
  if reverse x == x
    then Right "Match"
    else Left $ "reverse x == " ++ show (reverse x)

which results in

there exists [0,1] such that
  reverse x = [1,0]

Better than shrinking

QuickCheck tends to produce complex counterexamples and then tries to shrink them to find simpler ones. While shrinking does lead to somewhat simpler examples, they are usually not minimal. Also, it’s on you to implement the shrinking function for your own types (although there is genericShrink).

Because SmallCheck enumerates the values in order, from smaller to bigger, it automatically produces the minimal example.

Determinism and exhaustiveness

Because QuickCheck generates its examples randomly, the set of test cases your property is tested on is rather arbitrary. (It also changes from run to run by default, though this can be fixed by suppplying a fixed seed.)

Sometimes that’s good: if you have a high- or infinite-dimensional space of possibilities, all you can hope for is to test on a few diverse samples from that space. But other times, that space is narrow, and it’s better to test deterministically and exhaustively.

Disable Sony What's New notifications

Mon, 14 May 2018 20:00:00 +0000

If you have a Sony smartphone, you noticed the annoying notifications from Sony’s built-in WhatsNew app. They look like this:

Sony also made it impossible to disable the app or block the notifications through the usual means. Where normally you’d see an option to block all of the app’s notifications or disable the app itself, there are no such options for WhatsNew.

So for a long time I assumed that it was impossible to get rid of these notifications, short of rooting the phone.

Then one day I watched The Shawshank Redemption and was inspired by the scene where Andy was writing letters to senators. Maybe if I write a lot of letters to Sony’s support, they will do something about it?

To my surprise, I got a response right away.

Dear Roman,

Thank you for contacting the Sony Xperia support centre.

In regards of your query on how to disable the WhatsNew notifications on your Sony Xperia Device.

Kindly follow the below steps.

You can avoid getting notifications from the What’s New app by turning off Show notifications in the app. To disable notifications in What’s New

From your Home screen, tap the Application screen icon.

Find and tap What’s New.

Drag the left edge of the screen to the right, then find and tap Settings.

Drag the slider beside Show notifications to the left.

Note! The notifications of the application updates are exempted from the above setting. These updates may contain critical security and functionality patches that make your device more secure since it protects all the personal data on your device, and also improves the performance.

We hope this helps.

Kind Regards,
Mike [redacted]
Sony Xperia UK Customer Services

And indeed, the setting was there:

Thanks, Mike!

The only caveat (and the reason I hadn’t looked in the settings before) is that in order to get to the settings, you need to accept the app’s terms and conditions.

What is this weird math letter?

Fri, 11 May 2018 20:00:00 +0000

Occasionally I would read a mathematics paper or book and see a strange calligraphic symbol that I have no idea how to pronounce.

Like here:

Or here:

This is a problem because I, as most other people, struggle to read what I can’t pronounce even when not reading aloud.

When you encounter a symbol like this (which belongs to a family of typefaces called formal scripts), just look it up in the table below, and you’ll know what letter it stands for.

The table of formal script (“mathscr”) letters.
A	$\mathscr{A}$	N	$\mathscr{N}$
B	$\mathscr{B}$	O	$\mathscr{O}$
C	$\mathscr{C}$	P	$\mathscr{P}$
D	$\mathscr{D}$	Q	$\mathscr{Q}$
E	$\mathscr{E}$	R	$\mathscr{R}$
F	$\mathscr{F}$	S	$\mathscr{S}$
G	$\mathscr{G}$	T	$\mathscr{T}$
H	$\mathscr{H}$	U	$\mathscr{U}$
I	$\mathscr{I}$	V	$\mathscr{V}$
J	$\mathscr{J}$	W	$\mathscr{W}$
K	$\mathscr{K}$	X	$\mathscr{X}$
L	$\mathscr{L}$	Y	$\mathscr{Y}$
M	$\mathscr{M}$	Z	$\mathscr{Z}$

Alternatively, use the excellent Detexify tool:

The formal script typeface is available in LaTeX using the \mathscr command from the mathrsfs package:

\usepackage{mathrsfs}

$\mathscr{Q}$

GHC-style note snippet for Vim

Wed, 02 May 2018 20:00:00 +0000

I am a big fan of GHC-style notes. Notes are a convention for long, in-depth comments that do not interrupt the flow of the code.

A note is a comment that is formatted to be easily searchable and recognizable:

{- Note [Extra args in rule matching]
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   If we find a matching rule, we return (Just (rule, rhs)),
   but the rule firing has only consumed as many of the input args
   as the ruleArity says.  It's up to the caller to keep track
   of any left-over args.  E.g. if you call
       lookupRule ... f [e1, e2, e3]
   and it returns Just (r, rhs), where r has ruleArity 2
   then the real rewrite is
       f e1 e2 e3 ==> rhs e3
-}

A note can be referenced in the code like this:

lookupRule :: (Activation -> Bool) -> InScopeSet
        -> Id -> [CoreExpr]
        -> [CoreRule] -> Maybe (CoreRule, CoreExpr)
-- See Note [Extra args in rule matching]
lookupRule is_active in_scope fn args rules
  = case go [] rules of
    []     -> Nothing
    (m:ms) -> Just (findBest (fn,args) m ms)

This convention for notes was originally developed for GHC, but it can be used in absolutely any project and in any programming language. You can read more about notes in the GHC commentary and in The Architecture of Open Source Applications.

The only issue I used to have with notes was that I could never remember how to format them properly, so every time I wanted to insert one, I would find and copy-paste one either from my own code or from the GHC source.

This motivated me to create a SnipMate snippet for Vim that automates the process.

Here are the steps:

Install SnipMate
Make sure it uses the new snippet parser. Put
```
let g:snipMate = { 'snippet_version': 1 }
```
in your ~/.vimrc.

Run

base64 -d >> ~/.vim/snippets/haskell.snippets <<EOF
c25pcHBldCBOb3RlCgktLSBTZWUgTm90ZSBbJHsxOnRpdGxlfV0KCXstIE5vdGUgWyQxXQoJICAg
fn5+fn5+fiR7MS8uL34vZ30KCSAgICR7Mjp0ZXh0fQoJLX0K
EOF

(The reason for base64 is to preserve the relevant tab characters.)

This will install the following snippet:

snippet Note
  -- See Note [${1:title}]
  {- Note [$1]
     ~~~~~~~${1/./~/g}
     ${2:text}
  -}

Now you are ready to insert a note. Open a Haskell file in Vim and type

Note<Tab>

This will expand into a template

-- See Note [title]
{- Note [title]
   ~~~~~~~~~~~~
   text
-}

Just start typing the title of the note, and it will replace both occurrences of title at the same time. Then press <Tab>, and start typing in the text of the note.

The snippet consists of two part: the note itself, and the reference to the note. You leave one in place and cut-and-paste the other one to where you want it to be.

Is differentiability arbitrary?

Tue, 01 May 2018 20:00:00 +0000

A function $f$ is called differentiable at point $x_0$ if it can be approximated by a linear function near $x_0$.

More formally, $f$ is differentiable at $x_0$ iff for some number $A$ and for all $x$

\[ f(x)=f(x_0)+A\cdot(x-x_0) + o(x-x_0). \]

Here $f(x_0)+A\cdot(x-x_0)$ is the linear function of $x$ that approximates $f(x)$ near $x_0$, and $o(x-x_0)$ is the approximation error — a function of $x$ such that $\frac{o(x-x_0)}{x-x_0}$ tends to 0 when $x-x_0$ tends to 0.

There is something arbitrary about this definition. Why are we approximating $f$ by a linear function, and not by the square function $x^2$, or the square root function $\sqrt{x}$, or the sine function $\sin x$?

In my experience, this is almost never explained by high school teachers or university professors who introduce differentiability to students. At best they may say that linear functions play a very important role in mathematics which, while true, is just begging the question.

Approximating by the square function

If a function can be approximated by the square function, i.e.

\[ f(x)=f(x_0)+B\cdot(x-x_0)^2 + o((x-x_0)^2), \]

then it can also be appoximated by a linear function — or, more specifically, a constant function $f(x_0)$. This is because $(x-x_0)^2=o(x-x_0)$, so the whole term $B\cdot(x-x_0)^2 + o((x-x_0)^2)$ can put into the approximation error $o(x-x_0)$ in the usual definition, and $A$ can be set to zero.

Conversely, if a function is differentiable and its derivative is zero, then it can be approximated by the quadratic function.

Approximating by the square root function

Consider a function that can be approximated by the square root function:

\[ f(x)=f(x_0)+B\cdot\sqrt{x-x_0} + o(\sqrt{x-x_0}). \]

An example would be the square function itself, $f(x)=\sqrt{x}$, near $x_0=0$.

But is this property “sustainable”? It can hold at a single point $x_0$, but can it hold for all points in an interval in the same way as a function can be differentiable at all points of an interval?

Suppose that

\[ f(x)=f(x_0)+B(x_0)\cdot\sqrt{x-x_0} + o(\sqrt{x-x_0}) \]

for all $x_0$ in $[a,b]$. Pick an integer $M>0$ and divide $[a,b]$ into $M$ equal sub-intervals \[\left[a+(b-a)\cdot \frac{m}{M}, a+(b-a)\cdot \frac{m+1}M\right],\] $0\leq m\leq M-1$.

Then

\[ \begin{split} f(b)-f(a) & =\sum_{m=0}^{M-1} f\left(a+(b-a)\cdot \frac{m+1}M\right)-f\left(a+(b-a)\cdot \frac{m}{M}\mathstrut\right) \\ & =\sum_{m=0}^{M-1} B\left(a+(b-a)\cdot \frac{m}M\right)\sqrt{\frac{b-a}M} + o(1/M). \end{split} \]

You can already notice that something is strange. Suppose that $B(x)$ does not depend on $x$, $B(x)\equiv B$. Then the right-hand side above reduces to $B\sqrt{M(b-a)}+o(1/M)$ and goes to infinity as $M$ increases; it cannot possibly remain equal to $f(b)-f(a)$.

More generally (but for the same reason), there can’t be any point on $[a,b]$ where $B$ is continuous and not equal to 0, or any sub-interval of $[a,b]$ where $B(x)>\epsilon>0$.

So this approximation is rather weird unless $B(x)\equiv 0$.

Approximating by the sine function

What happens if we try to approximate by, say, $\sin x$?

\[ f(x)=f(x_0)+B\cdot\sin(x-x_0) + o(\sin(x-x_0)). \]

To be careful, we need to restrict this approximation to $x$ in some neighborhood of $x_0$. We didn’t have to do this before because when $x$ is not close to $x_0$, $o((x-x_0)^{\alpha})$ can be anything at all. Here, however, $\sin(x-x_0)$ can be small even if $x-x_0$ is not small.

We know that $\sin x$ is itself differentiable in the usual sense, its derivative at 0 being 1:

\[ \sin (x-x_0)=(x-x_0) + o(x-x_0). \]

Therefore,

\[ f(x)=f(x_0)+B\cdot(x-x_0) + o(x-x_0), \]

i.e. $f$ is differentiable in the usual sense as well, and vice versa.

Conclusion

Generalizing from these examples, let’s say that we approximate a function $f$ with a function $g$ in the neighborhood of $x_0$:

\[ f(x)=f(x_0)+B\cdot g(x-x_0) + o(g(x-x_0)), \]

where $g(0)=0$ so that $o(g(x-x_0))$ has the usual meaning.

Then:

If $g$ is itself differentiable at 0 and $g'(0)\neq 0$ (as in $g(x)=\sin x$), we get the usual class of differentiable functions.
If $g$ is differentiable at 0 and $g'(0)=0$ (as in $g(x)=x^2$), then we get a subclass of differentiable functions for which the derivative at $x_0$ is 0.
If $g$ itself is not differentiable at 0 (as in $g(x)=\sqrt x$), then this property will hold either at an isolated point or describe some strange (non-smooth) functions.

This doesn’t quite tell us why we pick $x$ instead of $\sin x$ — the answer would be some hand-wavy simplicity arguments — but at least it reassures us that we are not missing much by focusing on linear approximations.

Rational approach to conferences

Sun, 22 Apr 2018 20:00:00 +0000

When I went to any conference in the past, I would try to extract the maximum out of it: stay for the whole duration of the main event and attend all satellites. Now I think I might change my conference strategy.

So what changed?

Previously, I would stay in hostels, usually €20–25 a night. Once I travel somewhere for a conference, there is little cost to stay longer there. In comparison, fixed costs were high. In addition to the cost of the flight, I needed to apply for a visa to travel anywhere, which usually cost me €100–200, a trip to an embassy in Kiev, and quite a bit of time and effort to gather all the necessary documents.

Now I stay in hotels, usually around €100 per night, which makes the marginal cost of staying an additional day non-negligible. On the other hand, I no longer need a visa for short trips to continental Europe, so that reduces the fixed costs of going to a conference.

(The argument about the fixed costs sounds like a sunk cost fallacy: because I paid for the visa and flight, I now need to stay for the whole event. Yes, I am susceptible to sunk cost fallacies.)

Then there is the time itself. When I was younger, I did not value my time too much and probably didn’t have better alternatives than to stay in a different country and attend another day of the conference. Nowadays, I’d rather do some work, spend time with my family, or study.

Finally, thanks to my past experiences, I can now better predict what I get the most value from, and when the diminishing returns kick in. The best time at a conference for me is the first 2–3 days. This is when I am fresh and excited. After these first few days I get tired, going to the talks becomes a chore, and I’d rather sleep in than attend a keynote.

Another observation is that when an event has satellites, the satellites are usually more interesting to me than the main event. I don’t know how universal this is, but I found it true about ICFP and RECOMB. That the satellites are usually cheaper to attend than the main event is a nice bonus.

So for future conferences, I am going to only attend the satellites or, if there are no satellites, limit my attendance to the 2–3 most interesting days of the conference. This may allow me to go to a larger number of conferences. That said, I should also conduct due diligence on the content/programme of the conferences I plan to attend. Some of the events I went to this year were so disappointing that they mostly weren’t worth attending even for a day.

Seneca on social networks

Sat, 14 Apr 2018 20:00:00 +0000

The recent scandal around Facebook brought up many discussions on how apparently free things do have a price. But it’s nothing new, really; Seneca warned us about it two thousand years ago.

Our stupidity may be clearly proved by the fact that we hold that “buying” refers only to the objects for which we pay cash, and we regard as free gifts the things for which we spend our very selves. These we should refuse to buy, if we were compelled to give in payment for them our houses or some attractive and profitable estate; but we are eager to attain them at the cost of anxiety, of danger, and of lost honour, personal freedom, and time; so true it is that each man regards nothing as cheaper than himself.

Let us therefore act, in all our plans and conduct, just as we are accustomed to act whenever we approach a huckster who has certain wares for sale; let us see how much we must pay for that which we crave. Very often the things that cost nothing cost us the most heavily; I can show you many objects the quest and acquisition of which have wrested freedom from our hands. We should belong to ourselves, if only these things did not belong to us.

— Seneca. Moral letters to Lucilius. Letter 42: On values.

The correct way to mark formulas in LaTeX

Thu, 12 Apr 2018 20:00:00 +0000

The first thing I learned when I was introduced to LaTeX in 2003 was that you put inline formulas in $...$ and display formulas in $$...$$. Over the years I became aware of the alternative forms, $...$ and \[...\], but I found them ugly (because they are) and assumed they were relics from the early days of TeX, sort of like trigraphs in C.

As it turns out, it’s all exactly the other way around. $...$ and $$...$$ are the old TeX syntax; $...$ and \[...\] are the new LaTeX syntax.

Couldn’t Leslie Lamport think of a better syntax? I bet he could; he just couldn’t implement it. These days if we don’t like a language, we create a new one, with its own parser. But Lamport’s LaTeX is not a new language. It is a set of macros that work on top of Knuth’s TeX. What LaTeX has achieved is a testament to TeX’s flexibility and Lamport’s ingenuity. The \[...\] “syntax” is just a pair of TeX macros, \[ and \], and like all other control sequences they have to start with a backslash.

So why should we prefer Lamport’s ugly $...$ and \[...\] to Knuth’s beautiful $...$ and $$...$$?

You can find the exact details in the answers to the following TeX.SE questions:

Another good reference is An essential guide to LaTeX2ε usage: Obsolete commands and packages.

It boils down to the fact that $...$ and \[...\] can be (and are) redefined by packages, whereas $...$ and $$...$$ can’t.

The main example you’ll find in the above TeX.SE discussion is that $$...$$, when used between paragraphs, inserts unwanted white space; so \[...\] is (re)defined to suppress it.

The specific problem I had with $$...$$ is that it breaks the lineno package, which is used to add line numbers to the output document. The lines preceding a display formula don’t get numbered:

This is a problem with all the equation environments, as described in the lineno manual, section 7.1. lineno redefines the \[ and \] macros and displaymath, equation, eqnarray, and eqnarray* environments to work around this issue. But lineno can’t redefine the $$...$$ syntax; hence if you use it, it’ll break the numbering. (Oddly enough, lineno also doesn’t redefine the equation* environment, which is another problem I’ve run into.)

Here’s a cool thing: pandoc will automatically emit the new LaTeX formula syntax regardless of what syntax you use in the input file. This means you can continue to use $ and $$ in markdown and get the correct result:

% echo '$$a^2+b^2=c^2$$' | pandoc -f markdown -t latex
\[a^2+b^2=c^2\]

If you feel adventurous, you could even filter your LaTeX source through pandoc:

% echo '$$a^2+b^2=c^2$$' | pandoc -f latex -t latex
\[a^2+b^2=c^2\]

Install all R packages used in the code

Mon, 02 Apr 2018 20:00:00 +0000

Here is a one-liner I use to install all R dependencies of a script. It is especially useful if you need to run your code on a server or cluster.

perl -lne '/library\((.*)\)/ && print $1' *.R *.Rmd | xargs Rscript -e 'install.packages(commandArgs(T), repos = "https://cran.rstudio.com")'

An alternative is to use p_load from the pacman package in your R scripts instead of library.

See also the discussion in Elegant way to check for missing packages and install them?.

Fighting referral spam

Sun, 25 Mar 2018 20:00:00 +0000

Occasionally I like to look through this site’s http logs to get an idea of how many people read my articles and find out when someone links to one of them.

At some point I noticed that the visitor numbers inflated and most referrers became fake. But this wasn’t the referral spam that I was used to, when the referrer points to the spammer’s website.

Quite the opposite, referrals often pointed to other Haskell-related blogs — byorgey.wordpress.com, blog.ezyang.com, blog.plover.com etc. I suspect these were harvested from Planet Haskell.

Yet it was clear that the referrers were fake. The referring and referred articles were not related in any meaningful way, and upon inspection the referring articles indeed did not refer to this site. For example, the logs indicated that Mark Dominus linked to my article about stable pointers from his article about a Turkish John Doe, or that FPComplete found my intro to golden tests integral to measuring the success of devops.

But that left me puzzled — if this was spam, where’s the message? Who would profit from sending me to these seemingly relevant but unrelated articles on legit blogs?

My searches led me to this article by now defunct project Referrerspamblocker.com. Turns out the message is nowadays placed in the http headers such as Accept-Language, which are not traditionally logged by the web servers but shown by the spammers’ main target, Google Analytics.

Finding spammers’ IP addresses

Most of the spam IP addresses I observed came from two hosting providers: OVH and Online SAS. Both companies are French, and both have been noticed by webmasters before (OVH, Online SAS). A small fraction of spam requests also came from Tiscali France B2B, apparently a French division of an Italian telecom company.

So I needed to collect the IP addresses that belong to these hosting providers. Since they are not ISPs, I was not worried about blocking legitimate users.

Once I had the IP addresses, I could

clean up the existing http logs, and
block the future spam traffic to avoid wasting resources.

I used the method described here to extract all RIPE addresses whose maintainers matched (?:OVH|ONLINESAS)-MNT|MNT-TISCALIFR-B2B. All of these are French companies, so most of their IP addresses are in the RIPE database.

However, OVH also has some Canadian IPs, which are covered by ARIN. Sadly, ARIN does not provide its IP address database for bulk download like RIPE does. For now I manually added the block 158.69.0.0/16, which I recovered via WHOIS from a couple of IP addresses in my logs.

You can download the final list of IPv4 ranges here.

Cleaning up the existing logs

I used the R packages iptools, purrr, and dplyr to filter the referral spam:

refspam_ranges <- scan("refspam.txt", what=character())
is_refspam_ip <- function(ips) {
  map(refspam_ranges, function(range) ip_in_range(ips, range)) %>%
    reduce(`|`)
}
remove_refspam <- function(d) filter(d, !is_refspam_ip(address))

Blocking the spam using firewalld

I expect something like this to work:

xargs -n 1 -I{} firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='{}' drop" < refspam.txt

But I haven’t blocked them just yet. Instead, I’ve added $http_accept_language to the nginx’s log_format and am now anxious to finally learn what the spammers have been trying to tell me for so long.

Also, see the next section.

Legitimate bots

(Added on 2018-03-26.)

Mark Dominus wrote to me to say that he also had this problem and also added:

Is your idea that because the sources are hosting providers rather than ISPs, you will only be blocking bots running on hosted services? But isn’t there still a concern that you might also block legitimate bots hosted at the same providers?

Fair point. I ran a query to see if there were any self-identified bots coming from those addresses, and I was very surprised to see so many (n is the number of requests made in the last 90 days):

agent	n
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)	2142
Barkrowler/0.7 (+http://www.exensa.com/crawl)	1387
CommaFeed/2.5.0-SNAPSHOT (https://github.com/Athou/commafeed)	398
Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +http://ahrefs.com/robot/)	296
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)	148
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)	106
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)	64
FeedHQ/2017.07.18.1500363079 (https://github.com/feedhq/feedhq; ping; https://github.com/feedhq/feedhq/wiki/fetcher; like FeedFetcher-Google)	32
Mozilla/5.0 (compatible; HackerfallBot; +http://hackerfall.com/help/bot)	30
python-requests/2.18.4	22
LivelapBot/0.2 (http://site.livelap.com/crawler)	17
python-requests/2.11.1	10
SocialRankIOBot; http://socialrank.io/about	5
Tiny Tiny RSS/17.12 (e35a467) (http://tt-rss.org/)	3
FeedHQ/2018.03.21.1521618317 (https://github.com/feedhq/feedhq; ping; https://github.com/feedhq/feedhq/wiki/fetcher; like FeedFetcher-Google)	2
Mozilla/5.0 (compatible; ImageFetcher/7.0; +http://images.weserv.nl/)	2
Mozilla/5.0 (compatible; yiphysearchbot; +http://37.187.121.74/bot.php)	2
node-superagent/0.18.2	2
Python-urllib/3.5	2
curl/7.57.0	1
GuzzleHttp/6.2.1 curl/7.47.0 PHP/7.1.11-1+ubuntu16.04.1+deb.sury.org+1	1
Mozilla/5.0 (compatible; Grobbot/2.2; +https://grob.it)	1
Tiny Tiny RSS/17.12 (http://tt-rss.org/)	1
Wget/1.14 (linux-gnu)	1

Self-identified bots accounted for 14% of all http requests coming from OVH et al (the rest must be spammers). Most of these bots look legit. Moreover, several are RSS readers, which I most certainly don’t want to block. So I won’t be adding any firewall rules after all.

Are these bots self-aware?

(Added on 2018-04-08.)

Since I published this post, the spammy referrers magically stopped.

The total number of requests from the identified IP ranges dropped sharply, and the remaining ones had either no referrer or a sane referrer like ro-che.info or planet.haskell.org.

These are some hypotheses as to what could happen, although none of them strikes me as very likely.

These bots are self-aware; they found out that I exposed them and altered their behavior.
The bots’ owners found my article and altered their bots’ behavior.
Around the same time as I wrote this article, I moved the site from a server in Germany to one in the UK. Was there something preventing the bots from updating their DNS records for two weeks?
Or maybe they are not interested in spamming British servers?

Notes on Jekyll

Sat, 24 Mar 2018 20:00:00 +0000

Here I describe a couple of Jekyll issues that I had to debug recently.

Broken symlinks

I have many relative symlinks in my jekyll directory that point outside of it. For instance, docs/2010-10-22-posix.pdf is a symlink to ../../kpi/POSIX/2010-10-22-posix.pdf.

Recently I noticed that these symlinks are broken in jekyll’s generated directory, _site. It appears that jekyll copies the link as is, that is as a relative link to ../../kpi/POSIX/2010-10-22-posix.pdf, but because the location of the link changed from $JEKYLL_ROOT to $JEKYLL_ROOT/_site, it no longer pointed into the right place.

So I dug into the source code and found this in lib/jekyll/static_file.rb:

def copy_file(dest_path)
  if @site.safe || Jekyll.env == "production"
    FileUtils.cp(path, dest_path)
  else
    FileUtils.copy_entry(path, dest_path)
  end

  unless File.symlink?(dest_path)
    File.utime(self.class.mtimes[path], self.class.mtimes[path], dest_path)
  end
end

As described in the docs for FileUtils, cp copies the file content, similarly to cp -L, while copy_entry copies a link as a link, like cp -d.

As the code implies, the behavior we want (FileUtils.cp) can be triggered by setting the environment to production:

JEKYLL_ENV=production

I don’t see why this shouldn’t be the default behavior, so I opened an issue.

Interestingly, I’m pretty sure it used to work properly before. When I discovered the issue, I rushed to check whether my site has been broken all this time, but the files were in the right places on the server. I thought that maybe this is a recent regression, but git blame says that this piece of code hasn’t changed in years. I still have not solved this mystery.

A case of a missing file

I needed to publish the files of a research project I’m working on. The main code for the project is in an R markdown notebook (“Rmd”). I was going to publish the raw code.Rmd file, so that others could run it, as well as its rendered version, code.html.

When I looked into the _site directory, code.html was there, but code.Rmd was mysteriously missing.

It took me a while to find the problem, but it is a simple one: Rmd is a markdown file with a YAML front matter — just like the markdown files jekyll has to process. So the code.html I was seeing was not the original code.html (which contains the output of the R code) but the Rmd file rendered by jekyll. And like all the other jekyll’s input files, code.Rmd was not copied into the output directory.

This problem was discussed before (here and here), and the advice was to make jekyll ignore the problematic files/directories and then use something else (like Gulp) to copy them into the _site directory.

This seemed too complicated to me, so I made a small patch to support “verbatim” directories, which are copied as is even if some of the files contain the YAML front matter. To enable this, simply add

verbatim:
  - files

to your _config.yml, where files is the name of the directory to copy verbatim.

I submitted this feature as a pull request, which I hope will be accepted, but for the time being you can get it from my fork of the jekyll repo.

Export transactions from Santander

Sun, 04 Mar 2018 20:00:00 +0000

Getting your transactions out of Santander’s online banking in a machine readable format like csv is far from trivial.

At first this seems like a lot of options, but most of them are useless:

“Midata (csv)” gives a semicolon-separated(!) file with only three transactions (out of 50+). It seems that these correspond to the two deposits and one cash withdrawal I’ve had, but all the cashless payments are absent. Additionally, the two deposits have their description masked with asterisks.

“Microsoft Excel (XLS)” indeed gives a file with an xls extension, but that extension has nothing to do with the file’s contents.

% file 2018-03-03.xls
2018-03-03.xls: HTML document, ISO-8859 text, with very long lines

It’s an html file consisting of a giant badly-formatted table, littered with FONT tags.

I thought that I could extract the data with htmltab, but it only recognizes the date column, not sure why.

Someone who goes by the name Infernus published a Ruby script to extract the data from the html file. The script, however, does not distinguish between the “money in” and “money out” columns in the html table, and so a ₤100 deposit and a ₤100 payment/withdrawal look exactly the same in the output file.

“Microsoft Money (QIF)” and “Intuit Quicken (QIF)” give exactly the same file, and that file cannot be parsed with Finance::QIF and this script, resulting in an “Unknown header format” error.

The only robust options to export the transactions seems to be through their txt file format, which looks like this:

From: 01/01/2018 to 03/03/2018
                            
Account: XXXX XXXX XXXX XXXX
                        
Date: 03/03/2018
Description: CARD PAYMENT TO STAGECOACH BUS,4.40 GBP, RATE 1.00/GBP ON 01-03-2018
Amount: -4.40   
Balance: XXXX

...

There are some peculiarities in the format, of course, like the Latin1 encoding and the use of non-breaking spaces as a separator, but all in all it can be easily parsed. Here’s a Perl script to convert Santander’s text format to the tab-separated value format (tsv).

#!/usr/bin/perl

use warnings;
use strict;

# The input file is encoded in iso-8859-1.
open my $f, "<:encoding(iso-8859-1)", $ARGV[0]
    or die qq{Could not open $ARGV[0]: $!};
binmode(STDOUT, ":utf8");

# Field and record separators.
$, = "\t";
$\ = "\n";

# Field names, for both the input and the output file.
# The order is arbitrary and determines the order of the
# columns in the output file.
my @names = qw(Date Description Amount Balance);
# Mapping from column names to column numbers.
my %pos;
@pos{@names} = (0..$#names);

# Print the tsv header.
print @names;

my @this;
while (<$f>) {
  if (my ($name,$value) = /^(\S+):\s*(.*?)(\s*|\x{A0}.*)$/g) {
    if ($name eq 'Date' and @this) {
      print @this;
      @this = ();
    }
    if (defined $pos{$name}) {
      $this[$pos{$name}] = $value;
    }
  }
}

Update (2018-06-05)

The script is now updated to deal with an inconsistency in Santander’s txt format. Within the same file, I see some records that include the currency

Amount:<A0>-3.00<A0>

and some that don’t

Amount:<A0>-41.28<A0>GBP

It’s scary to imagine how they generate this file.

Undefined behavior with StablePtr in Haskell

Sat, 03 Feb 2018 20:00:00 +0000

What will the following Haskell code snippet do?

  ptr1 <- newStablePtr 1
  ptr2 <- newStablePtr 2

  print =<< deRefStablePtr ptr1

Of course it will print 1… most of the time. But it’s also easy to make it print 2.

Here is the full program:

import Foreign.StablePtr

main = do
  ptr <- newStablePtr ()
  freeStablePtr ptr
  freeStablePtr ptr

  ptr1 <- newStablePtr 1
  ptr2 <- newStablePtr 2

  print =<< deRefStablePtr ptr1

If I compile it with ghc 8.0.2 on x86-64 Linux, it prints 2 and exits cleanly. (Under ghci or runghc, it also prints 2 but then gets a SIGBUS.) You can imagine how much fun it was to debug this issue in a complex library with a non-obvious double free.

The docs for freeStablePtr say:

Dissolve the association between the stable pointer and the Haskell value. Afterwards, if the stable pointer is passed to deRefStablePtr or freeStablePtr, the behaviour is undefined.

As far as undefined behaviors go, this one is fairly benign — at least it didn’t delete my code or install a backdoor.

Let’s see what’s going on here. The relevant definitions are in includes/stg/Types.h, includes/rts/Stable.h, rts/Stable.h, and rts/Stable.c. The excerpts below are simplified compared to the actual ghc source.

A stable pointer is just an integer index into an array, stable_ptr_table, although it is represented as a void*.

/*
 * Stable Pointers: A stable pointer is represented as an index into
 * the stable pointer table.
 *
 * StgStablePtr used to be a synonym for StgWord, but stable pointers
 * are guaranteed to be void* on the C-side, so we have to do some
 * occasional casting. Size is not a matter, because StgWord is always
 * the same size as a void*.
 */

typedef void* StgStablePtr;

typedef struct {
    StgPtr addr;
} spEntry;

spEntry *stable_ptr_table;

This is how the table works:

If an index i is allocated to a valid StablePtr, then stable_ptr_table[i].addr points to whatever heap object the stable pointer is supposed to point to.
```
StgPtr deRefStablePtr(StgStablePtr sp)
{
    return stable_ptr_table[(StgWord)sp].addr;
}
```

If the index i is free (not allocated to any valid StablePtr), then the corresponding entry in the table acts as a node in a linked list that contains all free entries. The variable stable_ptr_free contains a pointer to the start of this linked list.

static spEntry *stable_ptr_free;

/* Free a StgStablePtr */
void freeSpEntry(spEntry *sp)
{
    sp->addr = (P_)stable_ptr_free;
    stable_ptr_free = sp;
}

/* Allocate a fresh StgStablePtr */
StgStablePtr getStablePtr(StgPtr p)
{
  StgWord sp;

  sp = stable_ptr_free - stable_ptr_table;
  stable_ptr_free  = (spEntry*)(stable_ptr_free->addr);
  stable_ptr_table[sp].addr = p;
  return (StgStablePtr)(sp);
}

Suppose that the first two free entries in stable_ptr_table are 18 and 19, which is what I actually observe at the program startup. (The RTS creates a few stable pointers of its own for the purpose of running the program, which explains why these don’t start at 0.) Here’s the double-free code annotated with what happens on each step.

  ptr <- newStablePtr ()
  -- ptr == 18
  -- stable_ptr_free == &stable_ptr_table[19]
  -- stable_ptr_table[18].addr is a pointer to a Haskell value ()
  freeStablePtr ptr
  -- stable_ptr_free == &stable_ptr_table[18]
  -- stable_ptr_table[18].addr == &stable_ptr_table[19]
  freeStablePtr ptr
  -- stable_ptr_free == &stable_ptr_table[18]
  -- stable_ptr_table[18].addr == &stable_ptr_table[18]

  ptr1 <- newStablePtr 1
  -- ptr1 == 18
  -- stable_ptr_free == &stable_ptr_table[18]
  -- stable_ptr_table[18].addr is a pointer to a Haskell value 1
  ptr2 <- newStablePtr 2
  -- ptr2 == 18
  -- stable_ptr_free is a pointer to a Haskell value 1
  -- stable_ptr_table[18].addr is a pointer to a Haskell value 2

Because stable_ptr_free now points into the Haskell heap and outside of stable_ptr_table, further allocations of stable pointers will corrupt Haskell memory and eventually result in SIGBUS or SIGSEGV.

RPM packager's cheat sheet

Thu, 25 Jan 2018 20:00:00 +0000

These are some random notes on creating RPM packages that I wrote some time ago for my own use.

I don’t have the time or motivation to organize them in a proper article, but I thought I’d publish them anyway in case they are useful to someone.

The RPM build tree

On Fedora, the default rpm build tree is ~/rpmbuild.

ls ~/rpmbuild 
BUILD/
BUILDROOT/
RPMS/
SOURCES/
SPECS/
SRPMS/

You can create this directory structure with rpmdev-setuptree from the rpmdevtools package.

The purpose of each directory is:

SPECS: spec files live here. They contain package metainformation and build instructions. This is what you’ll be writing when creating packages.
SOURCES: download the source tarballs into this directory.
SRPMS: .src.rpm bundle together a spec file and the sources for that spec. You can create one with rpmbuild -bs, or download with dnf download --source, and then build a binary package from it with rpmbuild -rb.
BUILD: sources are unpacked into this directory and get compiled here.
BUILDROOT: during the install phase, files are copied here.
RPMS: the final rpms are created here.

To change the location of the build tree, put this into ~/.rpmmacros:

%_topdir %{getenv:HOME}/myrpmbuild

SRPMs

To customize a package that already exists in Fedora (for instance, to update to a newer upstream version), it is convenient to download and install an srpm (also known as src.rpm).

First, download an srpm from dl.fedoraproject.org (substitute your release number) to ~/rpmbuild/SRPMS. Alternatively, use the program yumdownloader from the dnf-utils package:

yumdownloader --destdir ~/rpmbuild/SRPMS --source packagename

Next, install it into your build tree:

rpm -i ~/rpmbuild/SRPMS/packagename.src.rpm

Now the spec file should appear under ~/rpmbuild/SPECS and sources under ~/rpmbuild/SOURCES.

RPM macros

Evaluate an RPM macro expression:

rpm -E '%_topdir'

Show all macro definitions:

rpm --showrc

It will also show where the definitions are located: look for something like Macro path: /usr/lib/rpm/macros:/usr/lib/rpm/macros.d/macros.*:/usr/lib/rpm/platform/%{_target}/macros:/usr/lib/rpm/fileattrs/*.attr:/usr/lib/rpm/redhat/macros:/etc/rpm/macros.*:/etc/rpm/macros:/etc/rpm/%{_target}/macros:~/.rpmmacros.

Unfortunately, it doesn’t seem to show where each particular macro comes from.

Defining your own

It’s often useful to define variables in the SPEC file.

This can be done with either %define or %global, and %global is preferred:

%define commit a0a746442daf8dfca01859f1efcc8663b47b3705
%global commit a0a746442daf8dfca01859f1efcc8663b47b3705

Minimal RPM spec file

I usually start with the minimal spec file:

Name:       minimal
Version:    1
Release:    1%{?dist}
Summary:    Minimal package
License:    MIT

%description
...

%prep

%build

%install

%files
/*

RPM has many defaults and convenient macros suited for traditional ./configure && make && make install tarballs, but nowadays those are actually rare.

Sources

If you have any source files, they should be declared (usually above %description) like this:

URL:        ftp://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/edirect.zip
Source0:    edirect.zip

You can then download the sources automatically using spectool:

spectool -C ~/rpmbuild/SOURCES -g package.spec

Sometimes I don’t declare any sources and instead use a tool (such as pypi or gem) to download and install the software. This is not reproducible, and therefore unacceptable when creating public packages, but is usually fine for personal use.

github sources

Here’s an example of using a specific commit from a github repo.

See also the pages on SourceURL for git and versioning snapshots.

%global   commit a0a746442daf8dfca01859f1efcc8663b47b3705
%global   shortcommit %(c=%{commit}; echo ${c:0:7})
Release:  1.%{shortcommit}
Source0:  https://github.com/lwindolf/%{name}/archive/%{commit}/%{name}-%{commit}.tar.gz

%prep
%setup -n %{name}-%{commit}

Build stages

%prep

At this stage, you unpack the source tarballs and apply patches. If you have any sources to unpack, try to find a %setup invocation that does the trick. The flags to %setup are described elsewhere.

Auto-provides

http://stackoverflow.com/a/39819960/110081

Disable debuginfo

Put this in the spec file:

%global debug_package %{nil}

Extract dependencies from a spec file and install them

dnf builddep package.spec

Asymmetry of costs in (t-)SNE

Tue, 23 Jan 2018 20:00:00 +0000

Both the SNE and t-SNE papers contain the following claim:

Because the Kullback-Leibler divergence is not symmetric, different types of error in the pairwise distances in the low-dimensional map are not weighted equally. In particular, there is a large cost for using widely separated map points to represent nearby datapoints (i.e., for using a small $q_{j|i}$ to model a large $p_{j|i}$), but there is only a small cost for using nearby map points to represent widely separated datapoints.

(t-SNE)

Notice that making $q_{ij}$ large when $p_{ij}$ is small wastes some of the probability mass in the $q$ distribution so there is a cost for modeling a big distance in the high-dimensional space with a small distance in the low-dimensional space, though it is less than the cost of modeling a small distance with a big one.

(SNE)

Here $p_{ij}$ and $q_{ij}$ refer to the probabilities¹ in the Kullback–Leibler cost function

\[ KL(P||Q)=\sum_{i,j} p_{ij} \log\frac{p_{ij}}{q_{ij}}, \]

which are also measures of similarity: $p_{ij}$ is high when the points $i$ and $j$ are close to each other in the original, high-dimensional space, and $q_{ij}$ is high when the points are close in the low-dimensional map space.

This got me thinking:

How do we interpret this claim?
Does it hold universally as a mathematical theorem, or is it an empirical observation that holds most of the time?

There is a simple but weak interpretation of the claim that obviously holds. If $q_{ij}$ tends to 0 (so the points in the low-dimensional space become arbitrarily far apart), the cost function tends to infinity, whereas if $p_{ij}$ tends to 0, the cost function remains bounded. This interpretation is weak because it only applies asymptotically, whereas in real (t-)SNE we are concerned with distributions of points within small rectangular plots.

Interpreting the claim is not trivial because we need two different datasets:

A dataset where two close points are mapped to two points that are farther away (high $p$, low $q$), and
a dataset where two distant points are mapped to two points that are close (low $p$, high $q$).

Then the claim is that the value of the cost function in (1) is higher than in (2). Clearly, the claim will not hold in general without requiring some sort of relationship between the datasets (1) and (2).

This is a good time to introduce some simplifying assumptions:

The conditional probability distributions will be Gaussian, as in the original SNE.
We shall ignore the perplexity-based adaptive computation of the variance in the data space and assume $\sigma^2=1/2$ instead, just like in the map space. This makes $p_{ij}$ symmetric and easy to compute.
Our original (“high-dimensional”) and map (“low-dimensional”) space will both have dimension 2. This allows us to draw them easily and makes it trivial to find the optimal embedding: take $y_i=x_i$.
Finally, our dataset will contain 3 points. We cannot work with only 2 points, because then $p_{12}$ and $q_{12}$ will always be 1. But 3 points are enough to make SNE interesting and test the claim. We will arrange these points to form isosceles triangles, where we fix the base and move the apex towards or away from the base. The length of the base will be 1, and the length of the legs will be denoted by $d$.

Given these constraints, is it true that when we move $x_3$ closer to $x_1$ and $x_2$, we suffer more than if we move it away? Again, this has no hope to hold in general (for every combination of left and right displacements): because the cost function is continuous, for every given displacement to the left there is a displacement to the right with a lower cost.

What if we displace $x_3$ to the left and to the right by the same amount? Then the result would depend on the specifics of the conditional distributions (normal, Cauchy) and their parameters ($\sigma$), which we are not modelling faithfully here, and would not be a result of the asymmetry of the KL divergence as implied by the statements quoted above.

Motivated by these thoughts, I came up with an interpretation that focuses solely on the asymmetry of the KL divergence, although it still uses the Gaussian distribution to calculate $p_{ij}$ and $q_{ij}$.

In this interpretation, we consider two isosceles triangles with the same base but different legs, $d_1$ and $d_2$. These triangles define probability distributions $P(d_1)$ and $P(d_2)$. The asymmetry of the KL divergence means that $KL(P(d_1)||P(d_2))$ and $KL(P(d_2)||P(d_1))$ are, in general, different. These divergences arise when mapping the longer triangle onto the shorter one and vice versa.

So the question becomes: is it true that $d_1>d_2$ implies $KL(P(d_2)||P(d_1)) > KL(P(d_1)||P(d_2))$?

This answer can be easily computed for different values of $d_1$ and $d_2$ in R:

library(dplyr)
library(ggplot2)

probs <- function(d) {
  p <- exp(-c(1,d,d)^2)
  return(p/sum(p))
}

kl <- Vectorize(function(d1, d2) {
  p <- probs(d1)
  q <- probs(d2)
  return(sum(p*log(p/q)))
})

d <- expand.grid(d1=seq(0.5,3,by=0.03), d2=seq(0.5,3,by=0.03)) %>%
  filter(d1>d2) %>%
  mutate(claim=(kl(d1,d2)<kl(d2,d1)))

ggplot(d) + geom_point(aes(d1,d2,color=claim),size=1) +
  scale_color_manual(values=c("red","darkgreen")) +
  theme_bw() + coord_fixed()

As can be seen from the graph, the claim holds when $d_1$ is high enough, consistent with the asymptotic interpretation. But it does not hold universally.

This is the formulation taken in t-SNE and the one I explore here. In the original SNE paper, the probabilities were asymmetric, and the cost function was a sum of KL divergences of pairs of conditional distributions.↩︎

New patterns in tasty

Mon, 08 Jan 2018 20:00:00 +0000

When I wrote tasty in 2013, I borrowed the pattern language and its implementation from test-framework. I wasn’t fond of that pattern language, but it did the job most of the time, and the task of coming up with a better alternative was daunting.

Over the years, however, the pattern language and implementation received more feature requests and complaints than any other aspect of tasty.

Mikhail Glushenkov wanted to run all tests in a group except a selected few, e.g. --pattern 'test-group/**' --ignore 'test-group/test-3'.
Utku Demir wanted to specify a list of tests to run, such as “A and B, but not C and D”; Philipp Hausmann and Süleyman Özarslan requested something similar in the same issue.
Rob Stewart wanted a pattern that would match Bar but not FooBar.
Daniel Mendler reported that patterns with slashes didn’t work because slashes have special meaning in the patterns.
Levent Erkök, whose package sbv has more than 30k tasty tests, was concerned about the performance of pattern matching.
Allowing regular expressions in patterns would fulfill some (though not all) of these requests. However, using a regular expression library has its downsides. Carter Schonwald repeatedly complained to me that regex-tdfa takes ages to compile. Simon Jakobi noted that regex-tdfa brings along several transitive dependencies (including parsec), which further increases compilation time, makes the package more prone to breakages, and makes it harder for the maintainers of core packages to use tasty.

Every time someone filed an issue about tasty patterns, I would say something like “oh, this will be fixed once I get around to that issue from 2013 about the new patterns”. But I still had no idea what that new pattern language should look like.

The new pattern language had to be expressive, containing at least the boolean operators. It had to allow matching against the test name, the name of any of the groups containing the test, or the full test path, like Foo/Bar/Baz for a test named Baz in the test group Bar, which itself is contained in the top-level group Foo.

Finally, there was an issue of familiarity. Whatever ad-hoc DSL I would come up with, I had to document thoroughly its syntax and semantics, and then I had to convince tasty users to learn a new language and read the docs every time they wanted to filter their tests. (Not that the old patterns were particularly intuitive.)

The insight came to me last summer while I was spending time with my family and working remotely from a cabin in Poltava oblast, Ukraine. The language I needed already existed and was relatively well-known. It’s called AWK!

In AWK, the variable¹ $0 refers to the current line of input (called a “record”), and the variables $1, $2 etc. refer to the fields resulting from splitting the record on the field separator (like a tab or a comma).

The analogy with test names in tasty is straightforward: $0 denotes the full path of the test, $1 denotes the outermost test group name, $2 for the next group name, and so on. The test’s own name is $NF.

Then you can use these variables together with string, numeric, and boolean operators. Some examples:

$2 == "Two" — select the subgroup Two
$2 == "Two" && $3 == "Three" — select the test or subgroup named Three in the subgroup named Two
$2 == "Two" || $2 == "Twenty-two" — select two subgroups
$0 !~ /skip/ or ! /skip/ — select tests whose full names (including group names) do not contain the word skip
$NF !~ /skip/ — select tests whose own names (but not group names) do not contain the word skip
$(NF-1) ~ /QuickCheck/ — select tests whose immediate parent group name contains QuickCheck

The list of all supported functions and operators can be found in the README.

As a shortcut, if the -p/--pattern argument consists of letters, digits, and characters, it is matched against the full test path, so -p foo is equivalent to -p /foo/.

The subset of AWK recognized by tasty contains only expressions (no statements like loops or function definitions), no assignment operators, and no variables except NF. Other than that, the most salient deviation is that pattern matching (as in $3 ~ /foo/) does not use regular expressions, for the reasons stated above. Instead, a pattern match means a simple substring search — an idea suggested by Levent Erkök. So /foo/ in tasty means exactly the same as in AWK, while AWK’s /foo+/ cannot be expressed.

This allowed me to drop regex-tdfa as a dependency and significantly speed up the compilation time. An installation of tasty-1.0 (the new major release featuring AWK patterns) from scratch (a fresh cabal sandbox) takes 24 seconds on my laptop², while an installation of tasty-0.12.0.1 (the previous version, which depends on regex-tdfa) takes 2 minutes 43 seconds.

The performance improved, too. I tried Levent’s example, which runs 30k dummy tests (j @?= j). When run with --quiet (so no time is wasted on output), tasty-0.12.0.1 takes 0.3 seconds to run all tests and 0.6 seconds to run a single test selected by a pattern (-p 9_2729). The new tasty-1.0 takes the same time without a pattern, and less than 0.1 seconds with a pattern (also -p 9_2729, which is equivalent to $0 ~ /9_2729/). The overhead of pattern matching, although it was pretty low already (0.3 seconds per 30k tests), became much smaller — so that it is now outweighed by the benefit of running fewer dummy tests. I haven’t done any performance optimization at all³, so I don’t even know where the speedup came from, exactly.

Earlier I said that I dropped regex-tdfa as a dependency for tasty and that regex-tdfa in turn depended on parsec; but didn’t I have to retain parsec or a similar library to parse the AWK syntax? No! We already have a perfectly fine parser combinators module in the base library, Text.ParserCombinators.ReadP. Its original purpose was to back the standard Read instances, but there is no reason it can’t be used for something more fun.

I did borrow one small module from megaparsec for parsing expressions (Text.Megaparsec.Expr), which I adapted to work with ReadP and to parse ternary operators. The expression parser originally comes from parsec, but Mark Karpov did a great job refactoring it, so I recommend you read Mark’s version instead. The expression parser is an ingenious algorithm deserving a separate blog post.

Enjoy the new tasty patterns!

Actually, unlike in Perl, in AWK $0 is an expression: the operator $ applied to the number 0. Instead of $0 you could write $(1-1). The most practically relevant implication is that you can write $NF for the last field in the record, $(NF-1) for the second to last field and so on.↩︎
Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz, 2 cores (4 virtual), SSD.↩︎
I had an idea to do partial evaluation of the expression, so that the condition $2 == "X" would prune the whole test subgroup with name $2 == "Y" without having to consider each test individually. But after doing the measurement, I saw that matching is already fast enough. and the extra complexity is not justified.↩︎

Where did all my probability go?

Fri, 29 Dec 2017 20:00:00 +0000

Let $X$ be a one-dimensional normal variable with mean 0 and standard deviation $\sigma$. What is the limit of $P(X>a|\sigma)$ when $\sigma$ tends to infinity?

We can compute the probability $P(X>a|\sigma)$ as an integral of the normal density function:

\[ P(X>a|\sigma)=\int_{a}^{+\infty}\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

As $\sigma$ goes to infinity, $\frac{1}{\sqrt{2\pi}\sigma}$ goes to 0, and $e^{-x^2/2\sigma^2}$ goes to 1. Their product, therefore, goes to 0, and so

\[ \lim_{\sigma\to+\infty}P(X>a|\sigma)= \int_{a}^{+\infty}\lim_{\sigma\to+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx =0. \]

But this does not make sense, does it? As the standard deviation increases, the normal distribution becomes more spread out, and the probability that $X$ will exceed a fixed threshold $a$ should increase, not fall down to 0.

Indeed, the same argument could be used to “show” that the probability $P(X\leq a|\sigma)$ also tends to 0 as $\sigma$ increases. So where does all the probability go?

The problem in our reasoning is swapping the integral and the limit operations, i.e. equating

\[ \lim_{\sigma\to+\infty}\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx \]

\[ \int_{a}^{+\infty}\lim_{\sigma\to+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

But why exactly this is a problem requries a little digging.

Proper integrals

We might suspect that the reason swapping the integral and the limit didn’t work is that we are integrating over an infinite interval $[a;+\infty)$; i.e. that our integral is improper.

One reason to think so is that we would be right to conclude that \[\lim_{\sigma\to+\infty} P(a\leq X\leq b|\sigma) = 0,\] where the integration happens over a finite interval $[a,b]$.

The improperness of the integral does play a role, as we shall see below; but by itself, properness is neither necessary nor sufficient to swap the integral and the limit.

A. Ya. Dorogovtsev in his mathematical analysis textbook gives the following example (section 13.2.3):

\[ f(x,y)=\frac1y\left(1-x^{1/y}\right)x^{1/y}, \]

where $x\in[1/2,1]$ and $y\in(0,1]$.

For any given $y$, $f(\cdot,y):[1/2,1]\to\mathbb{R}$ is a continuous function on a closed interval, so $\int_{1/2}^1 f(x,y)dx$ is proper. And yet, it can be shown that

\[ \forall x\in[1/2,1]\; \lim_{y\to 0+} f(x,y) = 0 \] while \[ \lim_{y\to 0+}\int_{1/2}^1 f(x,y)dx = 1/2. \]

Let’s look at the graph of $f(x,y)$ for different values of $y$ to see what’s going on.

Graph of $f(\cdot,y)$ for decreasing values of $y$.

On the one hand, at any given $x$, $f(x,y)$ tends to 0. (Look at how the function value changes for a fixed $x$, say, $x=0.8$ or $x=0.9$. It may be less obvious that the same is true for $x=0.99$; you’ll have to trust that the trend continues or verify it analytically.)

On the other hand, the function as a whole does not seem to converge to 0. This is formalized by the notion of uniform convergence.

We say that $g(x,y)$ converges uniformly to $g(x)$ when $y\to y_0$ iff the maximum vertical distance between the graphs of $g(x,y)$ and $g(x)$, i.e. $\sup_x |g(x,y)-g(x)|$, goes to 0.

Uniform convergence on $[a,b]$ is sufficient (although not necessary) to replace $\lim_{y\to y_0}\int_a^b g(x,y)dx$ with $\int_a^b \lim_{y\to y_0}g(x,y)dx$.

Clearly, our $f(x,y)$ converges to 0 non-uniformly: not only does $\sup_x |f(x,y)|$ not tend to 0, it tends to infinity. Non-uniform convergence allows a travelling wave. Such a wave can contribute to the integral while escaping the point-wise convergence analysis (the function converges at every point, but only after the wave has passed).

Improper integrals

Let’s revisit our first example, the limit \[ \lim_{\sigma\to+\infty}P(X>a|\sigma)dx = \lim_{\sigma\to+\infty}\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

A piece of the right tail of a normal distribution, $P(X|\sigma)$, for increasing values of $\sigma$.

Here again there is a wave traveling to the right (and a symmetric one traveling to the left, not shown here).

Because the wave travels with a finite speed, we may suspect non-uniform convergence: no matter how large $\sigma$ is, there are always distant enough points at which the density will temporarily increase as $\sigma$ continues to increase.

And yet, the height of the wave diminishes, and $\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}$ does converge to 0 uniformly:

\[ \sup_x \left|\frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}\right|= \frac{1}{\sqrt{2\pi}\sigma}\to 0\quad (\sigma\to+\infty). \]

The reason why this is not enough to bring the limit under the integral is that here we are dealing with an improper integral. Improper integrals are themselves limits; recall that the improper integral is defined as

\[ \int_{a}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx = \lim_{b\to+\infty}\int_{a}^{b} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx. \]

Before we get to bring $\lim_{\sigma\to+\infty}$ under $\int_a^b$, we need to sneak it past $\lim_{b\to+\infty}$ first. For that, the limit \[\lim_{b\to+\infty}\int_{a}^{b} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\] itself needs to converge uniformly as a function of $\sigma$; i.e. \[\sup_{\sigma} \left|\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\right|\to 0\quad (b\to+\infty).\]

Look, this is the same quantity that we started with: because the integral is a positive and increasing function of $\sigma$,

\[ \begin{split} \sup_{\sigma} \left|\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\right|&= \lim_{\sigma\to+\infty}\int_{b}^{+\infty} \frac{1}{\sqrt{2\pi}\sigma} e^{-x^2/2\sigma^2}dx\\&= \lim_{\sigma\to+\infty}P(X>b|\sigma). \end{split} \]

If we assume that $\forall b\:\lim_{\sigma\to+\infty}P(X>b|\sigma)=0$, then the integral converges uniformly, we can bring the limit under the integral, and prove that indeed, $\forall a\:\lim_{\sigma\to+\infty}P(X>a|\sigma)=0$.

But there is no reason to believe that the limit is zero, and that explains why our calculation yielding 0 was wrong: the integral does not converge uniformly.

The actual calculation of $\lim_{\sigma\to+\infty}P(X>a|\sigma)$ is simple: because $\forall a>0\:\lim_{\sigma\to+\infty}P(-a\leq X\leq a|\sigma)=0$ (thanks to uniform convergence!), all the probability mass goes to infinity, and because of the symmetry, each tail gets a half of it; so

\[\lim_{\sigma\to+\infty}P(X>a|\sigma)=1/2.\]

But I thought it would be instructive to investigate why the naive calculation did not work, so here we are.

Bioinformatics events in Europe in 2018

Wed, 27 Dec 2017 20:00:00 +0000

Here are some conferences and summer schools that will take place in Europe in 2018 and may be of interest to fellow bioinformaticians.

Event	Dates	Location	Early registration price	Early registration deadline
Festival of Genomics London	January 30–31	London, UK	Free
Information transmission in biological systems	March 5–9	Będlewo, Poland		February 10
Single Cell Biology	March 6–8	Hinxton, UK	₤229–429	Novermber 28 (2017)
From individual based models to structured population level description	March 12–16	Będlewo, Poland	€200
NGS	April 9–11	Barcelona, Spain	€125–395	March 12
RECOMB	April 21–24 Satellites: April 19–20	Paris, France	€135–700	February 26
Workshop on Data Structures in Bioinformatics	May 16–17	Helsinki, Finland
Quantitative Genomics	June 14	London, UK	₤40	March 31
From Cells to Process: Advances and Challenges in Single Cell Biology	June 28–29	Berlin, Germany	Free	May 15
Statistical Data Analysis for Genome Scale Biology	July 8–13	Brixen, Italy	€850–2440	May 15
Computational Immunology, Immunotherapy and Autoimmune Diseases Summer School	July 25–31	Lipari, Italy	€500	April 20
Dresden Summer School in Systems Biology	August 11–18	Dresden, Germany	€100–500	April 20
Joint Congress on Evolutionary Biology	August 18–22	Montpellier, France	€100–500
European Conference on Computational Biology	September 8–12	Athens, Greece	€420–860	July 31
NGSchool	September 16–23	Lublin, Poland		June 30
Genome Informatics	September 17–20	Hinxton, UK	₤320–520	June 26
Single-cell RNABIO & organoids	September 28–30	Kyiv, Ukraine	Free	August 15
German Conference on Bioinformatics	September 25–28	Vienna, Austria	€400	July 15
Grand BIMSB Opening Symposium	October 25–27	Berlin, Germany

Explained variance in PCA

Mon, 11 Dec 2017 20:00:00 +0000

There are quite a few explanations of the principal component analysis (PCA) on the internet, some of them quite insightful. However, one issue that is usually skipped over is the variance explained by principal components, as in “the first 5 PCs explain 86% of variance”. So this is my attempt to explain the explained variance.

TL;DR

The total variance is the sum of variances of all individual principal components.

The fraction of variance explained by a principal component is the ratio between the variance of that principal component and the total variance.

For several principal components, add up their variances and divide by the total variance.

Example & explanation

Let’s define a data set (matrix) in R that consists of 3 variables (columns) and 4 observations (rows), where the third variable is roughly the average of the first two.

set.seed(2018)
mx <- matrix(rnorm(4*2), nrow=4, ncol=2)
mx <- cbind(mx, (mx[,1]+mx[,2])/2 + rnorm(4, sd=0.4))
mx

            [,1]       [,2]       [,3]
[1,] -0.42298398  1.7352837  0.4119150
[2,] -1.54987816 -0.2647112 -0.6524724
[3,] -0.06442932  2.0994707  0.7603068
[4,]  0.27088135  0.8633512  0.1551048

And let’s compute the sample variances of each of the 3 variables (columns):

vars <- apply(mx, 2, var)
vars

[1] 0.6261715 1.1068959 0.3612204

Note that these are different from true population variances, which we know to be equal to 1, 1, and 0.66, respectively.

When talking about PCA, the sum of the sample variances of all individual variables is called the total variance. If we divide individual variances by the total variance, we’ll see how much variance each variable explains:

vars/sum(vars)

[1] 0.2989902 0.5285309 0.1724789

The highest fraction of explained variance among these variables is 53%, and the lowest one is 17%. We can also compute these fractions for subsets of variables. For instance, variables 1 and 2 together explain 83% of the total variance, and variables 1 and 3 explain 47%.

Principal component analysis computes a new set of variables (“principal components”) and expresses the data in terms of these new variables. Considered together, the new variables represent the same amount of information as the original variables, in the sense that we can restore the original data set from the transformed one.

Moreover, the total variance remains the same. However, it is redistributed among the new variables in the most “unequal” way: the first variable not only explains the most variance among the new variables, but the most variance a single variable can possibly explain.

More generally, the first $k$ principal components (where $k$ can be 1, 2, 3 etc.) explain the most variance any $k$ variables can explain, and the last $k$ variables explain the least variance any $k$ variables can explain, under some general restrictions. (The restrictions ensure, for example, that we cannot adjust a variable’s explained variance simply by scaling it.)

Let’s try this in practice. This is our data set expressed in the new variables (“principal components”) called PC1, PC2, and PC3:

pca <- prcomp(mx, retx=T)
mx_transformed <- pca$x
mx_transformed

            PC1        PC2          PC3
[1,] -0.5876362 -0.3227198  0.055468818
[2,]  1.9378229 -0.1805488 -0.012606347
[3,] -1.1907211 -0.2331953 -0.048668119
[4,] -0.1594657  0.7364640  0.005805649

These are the sample variances of the new variables.

vars_transformed <- apply(mx_transformed, 2, var)
# or: pca$sdev^2
vars_transformed

        PC1         PC2         PC3 
1.847906652 0.244501735 0.001879334

Notice that their sum, the total variance, is the same as for the original variables: 2.09.

And these are the same variances divided by the total variance, i.e. how much of the total variance each new variable explains:

vars_transformed/sum(vars_transformed)

         PC1          PC2          PC3 
0.8823556734 0.1167469648 0.0008973618

In the original data set, the highest explained variance by a single variable was 53%; here it’s 88%. And the lowest explained variance dropped from 17% to less than 0.1%. The highest explained variance by two variables also increased, from 83% to 99.9%, and so on. (Because the total variance has not changed, these observation about the fractions of the total variance are equally valid for the variances themselves.)

Caution: PCA, being derived from noisy data, is itself noisy. For instance, it would be a mistake to conclude that there exists a parameter that explains 88% of the variability in the actual quantities we have measured. The true fraction of total variance that can be captured by a single variable in this case is only around 60%, and we would get closer to it if we increased our sample size.

Mathematical justification

Hopefully the above explanation makes an intuitive sense. But, from a mathematical perspective, it raises several questions:

What does it mean to introduce new variables?
Why is the total variance preserved?
Why can’t we make the variance for individual variables as low or high as we want simply by scaling them?
Why is it desirable to maximize explained variance?

Let $X$ be an $n\times p$ matrix whose columns correspond to $p$ variables and whose rows correspond to $n$ samples or measurements.

What does it mean to introduce new variables? In a very general sense, it means coming up with a mapping (mathematical function) $f:\mathbb{R}^p\to\mathbb{R}^q$ from the old variables to the new ones, such that it has an inverse $f^{-1}:\mathbb{R}^q\to\mathbb{R}^p$, which restores the original data. The function $f$ is then applied to each row of $X$ to get the new data matrix, $Z$.

This formulation is too general, however. For example, because $\mathbb{R}^p$ is isomoprhic (as a set) to $\mathbb{R}$, we can encode everything in a single variable. But the relationship between the old variables and the new one will be very non-trivial. The new variable, even though it encodes precisely the same information, would tell us nothing meaningful about the data.

Therefore, we need to impose some restrictions on the nature of $f$. Here we require that $f$ be a linear function; so that $f(\mathrm{x})=\mathrm{x}A$, where $\mathrm{x}$ is a row of $X$ and $A$ is a $p\times q$ matrix. The new data matrix can be then computed as $Z = X A$.

Now it’s no longer possible to squeeze everything into a single variable; for the inverse mapping to exist we need $q\geq p$. Because we are trying to reduce the dimensionality of the data, not expand it, we’ll stick with $q=p$, so $A$ is an invertible $p\times p$ square matrix.

But even an invertible linear transformation is too general for PCA because it does not preserve the total variance.

From now on, we shall assume that the sample mean of each column of $X$ is 0. This can be achieved by subtracting from each column its mean value. Under this assumption, the covariance matrix of $X$ has a very simple form: $\mathrm{Cov}(X)=\frac1{n-1} X'X$. Also, if the columns of $X$ have zero mean, so do the column of $Z=XA$:

\[ \begin{split} \left(\frac1n\quad \frac1n\quad \ldots\quad \frac1n\right) Z &= \left(\frac1n\quad \frac1n\quad \ldots\quad \frac1n\right) X A\\ &= \left(0\quad 0\quad \ldots\quad 0\right) A\\ &= \left(0\quad 0\quad \ldots\quad 0\right). \end{split} \]

Then the covariance matrix of $Z$ is \[\mathrm{Cov}(Z)=\frac1{n-1} Z'Z=\frac1{n-1} A'X'XA=A'\mathrm{Cov}(X)A.\]

The total variance of $X$ is the sum of the diagonal elements in the covariance matrix $\mathrm{Cov}(X)$, i.e. its trace, $\mathrm{Tr}(\mathrm{Cov}(X))$.

In general, a transformation $\mathrm{Cov}(X)\mapsto A'\mathrm{Cov}(X)A$ does not preserve the trace of the matrix $\mathrm{Cov}(X)$. For instance, multiplying all variables by a constant number $c$ is a linear transformation with matrix $c\cdot I$. This transformation will inflate the total variance by a factor of $c^2$.

To ensure that the total variance does not change, we require that $\mathrm{Cov}(X)\mapsto A'\mathrm{Cov}(X)A$ is a similarity transformation, which always preserves the trace of the matrix it transforms. This is equivalent to saying that $A^{-1}=A'$, or that $A$ is orthogonal.

Preserving the total variance is not the only (or even an important) reason to require that the change of variables is an orthogonal transformation. As we saw earlier, the transformation $c\cdot I$ inflates the total variance by the factor of $c^2$, but it does so by uniformly inflating the variance of each variable. So when we compute the fraction of the total variance explained by the variables, that common factor cancels out.

The real problem is that we could rescale individual variables. Consider a $2\times 2$ matrix

\[ A=\pmatrix{a_{11}&0\\0&a_{22}}. \]

Unless $a_{11}$ and $a_{22}$ are $\pm 1$, $A$ is not orthogonal. So, in general, $A$ will not preserve the total variance of every matrix. However, for any given matrix $X$, there are many possible values of $a_{11}$ and $a_{22}$ that will preserve $X$’s total variance: they form an ellipse

\[a_{11}^2\mathrm{Var}(X_{\cdot 1})+a_{22}^2\mathrm{Var}(X_{\cdot 2})=\mathrm{Var}(X_{\cdot 1})+\mathrm{Var}(X_{\cdot 2}).\]

Recall that the objective of PCA is make the first variable explain the maximum fraction of the total variance. By choosing $a_{22}$ close to zero (and inferring $a_{11}$ from the above equation), we can make the fraction of variance “explained” by the first principal component arbitrarily close to 1 without transforming the data in any meaningful way.

This kind of cheating is made impossible by requiring that $A$ is orthogonal.

Now let’s talk about why it’s desirable to maximize the explained variance. The typical use of PCA is to keep only the first $k<p$ principal components. Because PCA is an orthogonal transformation, this corresponds to projecting the data from its original $p$-dimensional space to a $k$-dimensional subspace. The remaining $p-k$ components are lost in this projection; so it makes sense to minimize the variability of the data in those directions. Because the total variance is constant, minimizing the variance of the last $p-k$ variables is the same as maximizing the variance of the first $k$ variables. The choices we make in PCA are motivated precisely by this objective:

PCA itself is designed to maximize the variance of the first $k$ components, and minimize the variance of the last $p-k$ components, compared to all other orthogonal transformations.
We choose the first $k$ components, and not just some $k$ components, because they have the highest variance out of all principal components.
We try to choose $k$ big enough to make the lost information — the variance of the last $p-k$ components — sufficiently small.

Introduction to golden testing

Mon, 04 Dec 2017 20:00:00 +0000

Golden tests are like unit tests, except the expected output is stored in a separate file. I learned about them in 2010 from Max Grigorev at ZuriHac.

Let’s say you want to test Python’s json module. One way to do that would be to encode an object and compare the result to a reference string:

import json

assert(json.dumps([1,2,3]) == "[1, 2, 3]")

Alternatively, you could create a file with contents

[1, 2, 3]

and read it to know the expected output:

import json

with open("example1.json", "r") as ex1_file:
    ex1 = ex1_file.read().rstrip()
    assert(json.dumps([1,2,3]) == ex1)

The file example1.json is called a golden file.

Here are some advantages of golden tests over ordinary unit tests:

If the expected output is large in size, it may be impractical to put it inside the source code.
No need to escape quotes or binary data in the expected output.
When you add a new test, your testing framework can generate the missing golden file from the current output of the function.

It is best if you can write down the expected output without looking at the actual output, but it is not always possible. The output may be too big to type it character by character, or it may be hard to predict. For instance, in the json example, you couldn’t tell in advance whether there would be spaces between array elements or not. So often what you do is launch an interactive interpreter (if your language of choice even has one), run the function, and then copy-paste its output into the test code.

This process can be easily automated if you use golden files.
The expected output can be automatically updated.

Say you changed your json module to replace some of the spaces with newlines to make the output more aesthetically pleasing. You have 40 test cases that need updating. Can you imagine doing this by hand?

With golden tests, you can tell your test framework to update all golden files from the current outputs, then check git diff to ensure that all changes are valid, and commit them.
If some of your tests suddently started failing, you can use diff or other such tools to compare the golden file to the actual file and figure out what exactly changed. Perhaps your testing framework could even show the diff automatically on test failure?

While advantages 1-2 are automatic, 3-5 require special support from your testing framework. The rest of this article will be focused on a Haskell testing framework tasty and its add-on package for golden tests, tasty-golden.

Basic usage

To illustrate how tasty-golden works, consider this yaml-to-json conversion module:

{-# LANGUAGE TypeApplications #-}
module YamlToJson where

import qualified Data.Yaml as Y
import Data.Aeson as J
import qualified Data.ByteString.Lazy as LBS

yamlToJson :: LBS.ByteString -> LBS.ByteString
yamlToJson = J.encode . Y.decode @Value . LBS.toStrict

Because JSON contains quotes and YAML spans multiple lines, it is not very practical to store them as string literals in the source code file. Instead, you will keep them both in files.

Note that the name “golden file” only refers to the file containing the output, not the input. There is no requirement that the input is stored in a file or that there even is any “input” at all; but in practice it is often convenient to store them both in files so that there is an input file for every output file and vice versa.

import Test.Tasty (defaultMain, TestTree, testGroup)
import Test.Tasty.Golden (goldenVsString, findByExtension)
import qualified Data.ByteString.Lazy as LBS
import YamlToJson (yamlToJson)
import System.FilePath (takeBaseName, replaceExtension)

main :: IO ()
main = defaultMain =<< goldenTests

goldenTests :: IO TestTree
goldenTests = do
  yamlFiles <- findByExtension [".yaml"] "."
  return $ testGroup "YamlToJson golden tests"
    [ goldenVsString
        (takeBaseName yamlFile) -- test name
        jsonFile -- golden file path
        (yamlToJson <$> LBS.readFile yamlFile) -- action whose result is tested
    | yamlFile <- yamlFiles
    , let jsonFile = replaceExtension yamlFile ".json"
    ]

This is all the code you need to support one, two, or a thousand test cases. When run, this code will:

find all .yaml files in the current directory
for each .yaml file, construct a golden test that evaluates yamlToJson on the input read from file and compares the result to the golden file, which has the name and the .json extension
put all individual tests in a test group and pass it to defaultMain for execution

To see how this works in practice, create an input file, fruits.yaml, with the following contents:

- orange
- apple
- banana

Now run your test suite (note: in a proper cabalized project, you’d run cabal test or stack test instead):

% stack runghc test.hs
YamlToJson golden tests
  fruits: OK
    Golden file did not exist; created

All 1 tests passed (0.00s)

tasty-golden realized that this is a new test case because the golden file was absent, so it went ahead and initialized the golden file based on the function’s output. You can now examine the file to see if it makes sense:

% cat fruits.json
["orange","apple","banana"]

If you are happy with it, check in both input and output files to git. This is important so that your collaborators can run the tests, but it also helps when dealing with failing tests, as you’ll see next.

% git add fruits.yaml fruits.json && git commit -m "fruits test case"

Dealing with test failures

Occasionally, your tests will fail. A test that cannot fail is a useless test.

A golden test fails when the actual output does not match the contents of the golden file. You then need to figure out whether this is a bug or an intentional code change.

Let’s say you decide that the output of yamlToJson should end with a newline.

The new function definition is

yamlToJson = (<> "\n") . J.encode . Y.decode @Value . LBS.toStrict

Now run the test suite:

% stack runghc test.hs
YamlToJson golden tests
  fruits: FAIL
    Test output was different from './fruits.json'. It was: "[\"orange\",\"apple\",\"banana\"]\n"

1 out of 1 tests failed (0.00s)

Ok, this is not very helpful. There are two main ways to get better diagnostics. One is to use the goldenVsStringDiff function as an alternative to goldenVsString. This will include the diff right in the tasty output.

But my preferred workflow is to use git for this. First, rerun the tests and pass the --accept option. This will update the golden files with the new output:

% stack runghc -- test.hs --accept
YamlToJson golden tests
  fruits: OK
    Accepted the new version

All 1 tests passed (0.00s)

Now, because your golden file is tracked by git, you can examine the differences between the old and new golden files with git diff:

% git diff
diff --git fruits.json fruits.json
index c244c0a..ed447d4 100644
--- fruits.json
+++ fruits.json
@@ -1 +1 @@
-["orange","apple","banana"]
\ No newline at end of file
+["orange","apple","banana"]

Because this is the change you expected, you can now commit the updated file to git.

This workflow lets you use all the powerful git diff options like --color-words, or even launch a graphical diff tool like kdiff3 with git difftool.

Algebra of permutations in R

Sat, 18 Nov 2017 20:00:00 +0000

A permutation is a mathematical function that maps numbers from the set $\{1,2,\ldots,n\}$ to other numbers from the same set.

An example of a permutation for $n=5$ would be

\[ s=\begin{pmatrix} 1 & 2 & 3 & 4 & 5 \\ 2 & 5 & 4 & 3 & 1 \end{pmatrix} \]

This two-line notation denotes a function $s$ such that $s(1)=2$, $s(2)=5$ and so on.

In R, we can represent a permutation by its second row, and assume that the first row implicitly consists of the numbers from $1$ to $n$ in the ascending order:

s <- c(2, 5, 4, 3, 1)

An R vector represents a permutation if and only if it consists of n integers between 1 and n, and each integer occurs in the vector exactly once.

Here is how the mathematics of permutations maps to R:

The application $s(i)$ (where $i\in \{1,\ldots,n\}$) corresponds to s[i].
The composition of permutations $s$ and $t$, $s \circ t$, corresponds to the vector s[t].
the identity permutation, $\mathrm{id}$, corresponds to the vector 1:n, because s[1:n] == s.
The inverse of a permutation $s$, $s^{-1}$, corresponds to the vector order(s). This is because s[order(s)] == 1:s.
sample(n) gives a random permutation of size n.

The main use for permutations in R is to rearrange the data. If s is a permutation vector and d is an arbitrary vector (i.e. its elements are not necessarily numbers from 1 to n), then d[s] is some rearrangement of d. Specifically, d[s][i] == d[s[i]], so the ith element of d[s] is the s[i]th element of d.

Often we need to solve the reverse problem: we need to move the ith element of d to the position s[i] in the resulting vector. Because the inverse permutation of s is order(s), d[order(s)] is exactly what we need.

As I explain in rank vs. order in R, rank(a) and order(a) are inverse permutations of each other. But above I said that order(s) is the inverse of s itself. It follows, therefore, that when s is a permutation vector, rank(s) coincides with s itself:

s <- sample(100)
all(rank(s) == s)

[1] TRUE

Example: sort monthly data

As an example, let’s use the algebra of permutations to put some monthly data in the chronological order:

set.seed(2017-11-18)
v <- rpois(12, 50)
names(v) <- sample(month.abb)
v

Oct Mar Jul Aug Sep Nov Dec Apr Jun Jan May Feb 
 50  47  48  53  42  56  61  59  56  57  50  58

The proper chronological order is given by the month.abb vector:

 [1] "Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct" "Nov" "Dec"

Because our goal is to rearrange the vector v, the solution will have the form v[order(s)], where the permutation s specifies, for each index i, that the ith element should be moved to the s[i]th position.

Our strategy will be to build s as a composition of two permutations, x[y], where y transforms our random order into the alphabetical order of month names, and x then transforms the alphabetical order to the chronological order. (Remember that x[y][i] == x[y[i]], so y is the permutation that gets applied first.)

We need y to send the month name names(v)[i] to its correct position in the alphabetically-sorted vector. This position is given by rank(names(v))[i]:

y <- rank(names(v))

Let’s now consider x. It must send an index i in the alphabetically sorted vector to the corresponding index in the chronological vector. rank(month.abb) sends indexes from the chronological vector to the alphabetical vector, so x must be the inverse of that:

x <- order(rank(month.abb))

Let’s see if we got x and y right:

s <- x[y]
v[order(s)]

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 
 57  58  47  59  50  56  48  53  42  50  56  61

Yay!

Now, let’s simplify our code. Let’s denote rank(names(v)) as $r_1$ and rank(month.abb) as $r_2$, and recall that order of a permutation is its inverse.

Then $y$ is $r_1$, $x$ is $r_2^{-1}$, and order(s) is

\[ s^{-1} = (xy)^{-1} = (r_2^{-1} r_1)^{-1} = r_1^{-1} r_2. \]

Which, when translated back to R, corresponds to the permutation

order(rank(names(v)))[rank(month.abb)]

Can we also simplify order(rank(names(v)))? You may remember that order and rank are inverse of each other and cancel them out. That would be a mistake. For any a (without duplicate elements), order(a) and rank(a) are inverse permutations (order(a)[rank(a)] == 1:length(a)); however, order and rank are not inverse functions. It turns out that order(rank(a)) is simply order(a), because rank preserves the order of the elements.

Our final version is therefore

v[order(names(v))[rank(month.abb)]]

One might arrive at this result via an easier route, perhaps observing that v[order(names(v))] gives the alphabetically sorted vector and [rank(month.abb)] then restores the chronological order, but it may be less obvious in more complicated problems.

Also, as I show in rank vs. order, if you do this by trial and error and use small vectors to test your code, you can easily arrive at something that looks right but isn’t. Let’s say your data had only 4 months worth of data:

set.seed(2)
month.abb <- month.abb[1:4]
v <- rpois(4, 50)
names(v) <- sample(month.abb)
v

Jan Mar Apr Feb 
 43  43  42  50

Then this may look like the right solution (it’s not!):

v[rank(names(v))[rank(month.abb)]]

Jan Feb Mar Apr 
 43  50  43  42

So here’s how I recommend to approach such rearrangement problems:

Start with v[order(s)], transforming the problem into “where do I send the ith element”
Find or derive s
Optionally, simplify order(s) using the algebra of permutations

Estimate program run time

Thu, 09 Nov 2017 20:00:00 +0000

Programs used in bioinformatics vary a lot in how efficient their algorithms and implementations are. Different programs may take from minutes to years on the same data set.

As a bioinformatician, you must have some idea about how long the program will run. If the estimated time is too high, you may request more resources (more and faster computers) or even decide not to run it at all, rather than keep it running for two weeks and then kill it without getting any results.

You can estimate the run time of a program in three steps:

Collect the data
Build a model
Extrapolate

In this article, I will demonstrate this technique on a simple example: estimate how long it takes bowtie2 to index the third human chromosome.

Collect the data

First, we download and unpack the chromosome’s nucleotide sequence:

% wget ftp://ftp.ncbi.nih.gov/genomes/H_sapiens/CHR_03/hs_ref_GRCh38.p7_chr3.fa.gz
% gunzip hs_ref_GRCh38.p7_chr3.fa.gz

In order to collect some data, we need to run the program. But if we run the program on the full data set, it may take too long to complete — the very problem we are trying to avoid.

Therefore, we run the program on small subsets of the whole data set. For instance, if we are working with many short sequences of similar lengths, we may generate files consisting of the first $n$ sequences, as I explain here.

In our example, however, we have a few very long sequences, so we will use the following awk script (which we’ll put in the file head_fasta.awk) to extract the first limit nucleotides:

/^>/ {
  print;
  next;
}
{
  if ((l=length($0)) < limit) {
    print;
    limit -= l;
  } else {
    print(substr($0, 0, limit));
    exit;
  }
}

Next, we write a shell script (bench.sh) that will try several numbers of nucleotides, generate a file of that size, run bowtie2-build, and record the time it took to complete.

#!/bin/bash
# We rely on bash to get custom-formatted 'time' output.
# Otherwise, this script should be portable across shells.

# change the format of 'time' output to be the number of wall-clock seconds
TIMEFORMAT='%R'

# Print the csv header. 'n' stands for the number of nucleotides, 't' stands for
# run time in seconds.
printf "n,t\n"

# iterate over 1M, 2M, ..., 10M nucleotides
for n in `seq 1000000 1000000 10000000` # start, step, end
do
  # extract the first n nucleotides
  awk -v limit="$n" -f head_fasta.awk hs_ref_GRCh38.p7_chr3.fa > sample.fa
  # run bowtie2-build, noting the time
  # ignore the bowtie's output, and capture time's output from stderr
  t=$(time (bowtie2-build --noauto -f sample.fa sample >/dev/null 2>&1) 2>&1)
  # print the csv row
  printf "%d,%f\n" "$n" "$t"
done

There are a few things you should take into account when choosing the range of values for $n$:

Obviously, the values shouldn’t be too high, so that data gathering doesn’t take too long.
The values also shouldn’t be too low; otherwise, the measurements will be noisy and dominated by the fixed costs like program start time. I usually try to come up with a range for which the first (fastest) iteration completes in about a second.
The range of values shouldn’t be too narrow compared to the values themselves. This is so that any non-linear behavior can show itself. For instance, the width of range $101, 102, \ldots, 110$ is $110-101=9$, which is less than $10\%$ compared to the values themselves. Instead, I usually pick the range $n, 2n, \ldots, 10n$, whose width is nine times bigger than the smallest value.
The number of values shouldn’t be too low, so that a regression model can be fit. Ten is a good starting value.

The script will produce the output in the csv format; we use tee to look at it and write it to the file:

% ./bench.sh | tee time.csv

Here’s the output I got on my laptop:

n,t
1000000,0.718000
2000000,1.439000
3000000,1.827000
4000000,3.115000
5000000,3.235000
6000000,4.390000
7000000,5.248000
8000000,7.153000
9000000,7.065000
10000000,8.664000

Build the model

Let’s plot our data points. We do this to see how the run time grows with $n$. Most tools designed to work with big data sets will run in a linear or almost linear time, but it’s better to check.

To load and plot the data, we run the following R code:

library(readr)
library(ggplot2)

time <- read_csv("time.csv")
ggplot(time, aes(n,t)) + geom_point() + geom_smooth(method="lm",se=F)

The blue line is the linear regression line. The fact that the middle points are below the regression line is a sign of a faster-than-linear growth. But the non-linearity is too subtle to be quadratic; so it’s probably $O(n\log n)$.

To build a model with correlated predictors ($n$ and $n\log n$ in this case), it’s better to have more data. Let’s rerun our benchmark script, bench.sh, with a smaller step size of 100000 and tee the output to time2.csv.

Now we fit the model

\[t = a + b\cdot n + c\cdot n\log n\]

and plot it along the data points. As a rule of thumb, if you include higher-order terms (like $n\log n$ or $n^2$), include all lower-order terms (like $n$) as well. Note that lm adds the intercept term $a$ automatically.

library(dplyr)
time <- read_csv("time2.csv")
nlm <- lm(t ~ n + I(n*log(n)), data=time)
time <- mutate(time, model=predict(nlm))

ggplot(time) + geom_point(aes(n,t)) + geom_line(aes(n,model), color="blue")

The model with an $n\log n$ term gives a more realistic fit, so we’ll stick with it.

Extrapolate

Let’s find out how many nucleotides are in the whole chromosome:

% awk '/^[^>]/{n+=length($0)} END{print(n)}' hs_ref_GRCh38.p7_chr3.fa
202786747

Now we simply plug this number into our R model:

predict(nlm, newdata=list(n=202786747))

332.2153

So, our prediction is 332 seconds, or 5.5 minutes.

To check this prediction, I actually left bowtie2-build running for a while. It completed in 350 seconds — within 5% of our prediction!

On the other hand, if we went with the linear model, our prediction would have been 176 seconds — a severe underestimation by a factor of 2. The model with $n \log n$ but without the linear term would have predicted 206 seconds — only somewhat better than the linear model.

Does correlation imply causation?

Thu, 28 Sep 2017 20:00:00 +0000

In episode 55 of The Effort Report podcast, Roger Peng makes an interesting observation: correlation implies causation all the time! That’s how we know most of things in science, including that smoking causes lung cancer.

(around 00:37:30)

Roger is not wrong, but the informal language makes his statement ambiguous.

In formal logic, we almost never say “A implies B” $(A \Rightarrow B)$, except as a shorthand for something else.

Instead, we say A implies B under a set of assumptions Γ, or

\[\Gamma \vdash A \Rightarrow B.\]

Therefore, the statement “A implies B” can be valid under some Γ and invalid under others.

When we say “correlation does not imply causation”, we implicitly mean “without any additional assumptions”, that is,

\[\varnothing \nvdash \text{X correlates with Y}\Rightarrow\text{X causes Y}.\]

But under a suitable set of causal assumptions Γ, correlation may very well imply causation, which is one way to interpret Roger’s claim:

\[\Gamma \vdash \text{X correlates with Y}\Rightarrow\text{X causes Y}.\]

Elizabeth Matsui actually points to this difference when she says “correlation does not equal causation”.

To learn about causal assumptions, read the excellent overview of causal inference by Judea Pearl or one of his many other works.

How Booking.com manipulates you

Sun, 17 Sep 2017 20:00:00 +0000

Many websites and applications these days are designed to trick you into doing things that their creators want. Here are some examples from timewellspent.io:

YouTube autoplays more videos to keep us from leaving.
Instagram shows new likes one at a time, to keep us checking for more.
Facebook wants to show whatever keeps us scrolling.
Snapchat turns conversations into streaks we don’t want to lose.
Our media turns events into breaking news to keep us watching.

But one of the most manipulative websites I’ve ever come across is Booking.com, the large hotel search & booking service.

If you ever used Booking.com, you probably noticed (and hopefully resisted!) some ways it nudges you to book whatever property you are looking at:

Let’s see what’s going on here.

Prices

First, it tries to persuade you that the price is low. “Jackpot! This is the cheapest price you’ve seen in London for your dates!” Of course it is — this is literally the first price I am seeing for these days — so that statement is tautological. The first price I see will automatically be the lowest I will have seen, no matter how ridiculously high it’s going to be. The statement “This is the highest price you’ve seen in London for your dates!” would be just as valid.

Likewise, the struck-through prices are there to anchor you and make the actual price seem like a great deal. The struck-through US$175 is before applying my 10% “genius” discount — ok, that’s fair. But where does the US$189 come from? Let’s hover over the price to get an explanation:

I imagine most people will feel intimidated by the complex description, skip to the last sentence, and get the impression that “you get the same room for a lower price compared to other check-in dates”. (If this wasn’t the case, Booking.com’s marketing department would change the wording to make it so.)

But what is it actually saying? If there is only one room of this type, then 90% of time there will be an appearance of a lucky deal. What you should be reading is “If I choose this, I am not a total loser”. And if there are 3 comparable room types with differential pricing, there is even less of a reason to feel good about this — you’ve successfully avoided the worst 3% of the offerings.

Urgency

Another way Booking.com manipulates you is by conveying the sense of urgency.

“In high demand - only 3 rooms left on our site!”

“33 other people looking now, according to our Booking.com travel scientists” (what?)

“Last chance! Only 1 room left on our site!”

And just to prove they are not kidding, they will show you something that you’ve missed already:

But my favorite one is this red badge:

Although it cannot be seen on the screenshot, the badge doesn’t appear right away when you open the page. Instead, it pops up one or two seconds later, making it seem like a realtime notification — an impression reinforced by the alarm clock icon. To be clear, it is not realtime, and there is no reason to delay its display other than to trick you:

How much time has elapsed since the last booking doesn’t simply play on our irrational emotions. This is a valuable piece of information that can be used to estimate at what rate these rooms are being booked. By a Gott-like argument (see Algorithms to Live By for an excellent explanation), if the last booking was made 4 hours ago, you can estimate that a room is booked about every 8 hours. Plenty of time to relax and compare your options. If, on the other hand, the last booking happened just two seconds ago, you’d better not waste another second before entering your credit card number.

Kudos to Booking for at least providing the actual information in a tooltip window (I wonder what regulations make them do that), but not all users will hover to read it, and even then, something that you experience (a badge popping up) will probably take precedence over what you later read.

The “Someone just booked this” badge does not just make you worry that the room you are considering will soon be snatched; it also reassures you. If other people are actively booking this property, it must be good. Of course, the person who made a reservation 4 hours ago has not yet visited the hotel and so probably has little more knowledge than you. Their decision to book is probably to some extent influenced by the same red badge. This situation, where everyone relies on everyone else to have accurate information, is also well described in Algorithms to Live By.

Reviews

Instead of listening to people who have just booked the property but have not yet visited it, we should turn to those who have been there, right? That’s what reviews are for!

But Booking.com managed to game the reviews, too. There are quite a few crappy hotels out there, especially on the cheaper end of the spectrum. These are the hotels you and I would want to avoid, but if we did, Booking would not make money on them.

An extreme example is the hotel I’m currently staying at, New Union. From its Booking.com page you wouldn’t think there’s anything wrong with it, would you?

If you spend enough time perusing that page, you’ll eventually stumble upon the “fine print”:

“noise may be heard whilst the bar is open” is an understatement; the music is so loud that the floor in my room shakes very perceptibly, and the bar is open until late at night/early in the morning. And why is this warning hidden in the fine print instead of being a big red fucking badge?

To be fair, this is more of the hotel’s fault than Booking’s. Also, I should have read the fine print. Or the reviews. But wait, I’ve read the reviews:

What I didn’t realize skimming an overloaded webpage was that the reviews displayed on the main page had been cherry-picked. The full list of reviews is available from a separate page, e.g.

Ratings

Notice something interesting here: the first, moderately negative review, gives the place a rating of 7 out of 10, and the second review, probably as negative as it gets, gives it almost a 6. As a result, the overall rating of the property is high (7.6), and even the distribution of ratings does not look alarming:

Unlike IMDB or Amazon, where you simply give a movie or product a number of stars, when you rate your stay on Booking.com, you evaluate it on several factors: location, cleanliness, facilities etc.

But Booking.com doesn’t present individual ratings; it presents the average (like 7.1 or 5.8) and the average of averages (the overall rating of a property). It is unlikely that all factors will be bad simultaneously, but a problem in one of them may easily ruin your trip. A great location will not compensate for sleepless nights, but Booking.com thinks otherwise.

Baran Toppare and @tqenn both pointed out another way in which Booking.com ratings are inflated:

You missed the fact their reviews are not on a 0-10 scale, but 2.5-10. Every review score is intentionally inflated. Try reviewing & see.
— Tqenn (@tqenn) September 21, 2017

(To explain Booking’s system, each factor’s rating is on the scale {2.5, 5, 7.5, 10}, and the combined rating is the arithmetic mean of all factors.)

I agree, except ratings on the internet are almost never 0-based. The number of stars usually goes from 1 to 5. But if they really wanted a 4-point scale culminating at 10, they could have picked the scale {1, 4, 7, 10}. So yes, this is another way in which Booking.com artificially increases hotel ratings.

What to do with all of this

Frankly, I don’t think I am going to stop using Booking.com. I am not aware of any other service with a comparable number of properties and reviews.

Instead, we need to be aware of all the ways Booking is trying to screw us over and try to counter them:

Ignore urgency-provoking red text and anchoring struck-through prices.
Do not rely on the magnitude of the ratings. I think it is still fine to use them as a sorting criterion.
Do not read the cherry-picked reviews on the main page. Go to the review page (“Our guests’ experiences”) and sort the reviews from newest to oldest, to get an up-to-date and hopefully unbiased selection.

As I learned from several commenters on Hacker News, these manipulative tricks are called “dark patterns”, and they are catalogued over at darkpatterns.org.

Understanding Asymmetric Numeral Systems

Sun, 20 Aug 2017 20:00:00 +0000

Apparently, Google is trying to patent (an application of) Asymmetric Numeral Systems, so I spent some time today learning what it is.

In its essense lies a simple and beautiful idea.

ANS is a lossless compression algorithm. Its input is a list of symbols from some finite set. Its output is a positive integer. Each symbol $s$ has a fixed known probability $p_s$ of occurring in the list. The algorithm tries to assign each list a unique integer so that the more probable lists get smaller integers.

If we ignore the compression part (assigning smaller integers to more probable inputs), the encoding could be done as follows: convert each symbol to a number from $0$ to $B-1$ (where $B$ is the number of symbols), add a leading 1 to avoid ambiguities caused by leading zeros, and interpret the list as an integer written in a base-$B$ positional system.

This encoding process is an iterative/recursive algorithm:

Start with the number 1;
If the current number is $n$, and the incoming symbol correspond to a number $s$, update the number to be $s + n\cdot B$.

The decoding process is a corecursive algorithm:

Start with the number that we are decoding;
Split the current number $n$ into the quotient and remainder modulo $B$;
Emit the remainder and continue decoding the quotient;
Stop when the current number reaches 1.

(The decoding is LIFO: the first decoded element will be the last encoded element.)

This encoding scheme relies on the standard isomorphism between the sets $\{0,\ldots,B-1\}\times \mathbb{Z}_{\geq 1}$ and $\mathbb{Z}_{\geq B}$, established by the functions

\[f(s,n) = s + n\cdot B;\] \[g(n) = (n \bmod B, [n/B]).\]

(The peculiar domain and codomain of this isomorphism are chosen so that we have $\forall n,s.\;f(s,n) > n$; this ensures that the decoding process doesn’t get stuck.)

We can represent this in Haskell as

{-# LANGUAGE ScopedTypeVariables, TypeApplications,
             NamedFieldPuns, AllowAmbiguousTypes #-}

import Data.Ord
import Data.List
import Numeric.Natural

data Iso a b = Iso
  { to :: a -> b
  , from :: b -> a
  }

encode :: Iso (s, Natural) Natural -> [s] -> Natural
encode Iso{to} = foldl' (\acc s -> to (s, acc)) 1

decode :: Iso (s, Natural) Natural -> Natural -> [s]
decode Iso{from} = reverse . unfoldr
  (\n ->
    if n == 1
      then Nothing
      else Just $ from n
  )

And the standard isomorphism which we used in the simple encoding process is

std_iso :: forall s . (Bounded s, Enum s) => Iso (s, Natural) Natural
std_iso = Iso (\(s,n) -> s2n s + base @s * n) (\n -> (n2s $ n `mod` base @s, n `div` base @s))

s2n :: forall s . (Bounded s, Enum s) => s -> Natural
s2n s = fromIntegral $
  ((fromIntegral . fromEnum) s             :: Integer) -
  ((fromIntegral . fromEnum) (minBound @s) :: Integer)

n2s :: forall s . (Bounded s, Enum s) => Natural -> s
n2s n = toEnum . fromIntegral $
  (fromIntegral n + (fromIntegral . fromEnum) (minBound @s) :: Integer)

base :: forall s . (Bounded s, Enum s) => Natural
base = s2n (maxBound @s) + 1

(The functions are more complicated than they have to be to support symbol types like Int. Int does not start at 0 and is prone to overflow.)

Let’s now turn to the general form of the isomorphism

\[f \colon \{0,\ldots,B-1\}\times \mathbb{Z}_{\geq 1} \to \mathbb{Z}_{\geq \beta};\] \[g \colon \mathbb{Z}_{\geq \beta} \to \{0,\ldots,B-1\}\times \mathbb{Z}_{\geq 1}.\]

(In general, $\beta$, the smallest value of $f$, does not have to equal $B$, the number of symbols.)

If we know (or postulate) that the second component of $g$, $g_2\colon \mathbb{Z}_{\geq \beta} \to \mathbb{Z}_{\geq 1}$, is increasing, then we can recover it from the first component, $g_1\colon \mathbb{Z}_{\geq \beta} \to \{0,\ldots,B-1\}$.

Indeed, for a given $s=g_1(n)$, $g_2$ must be the unique increasing isomorphism from \[A_s = \{f(s,m)\mid m\in\mathbb{Z}_{\geq 1}\} = \{n\mid n\in\mathbb{Z}_{\geq \beta}, g_1(n) = s\}\] to $\mathbb{Z}_{\geq 1}$. To find $g_2(n)$, count the number of elements in $A_s$ that are $\leq n$.

Similarly, we can recover $f$ from $g_1$. To compute $f(s,n)$, take $n$th smallest number in $A_s$.

In Haskell:

ans_iso :: forall s . Eq s => (Natural, Natural -> s) -> Iso (s, Natural) Natural
ans_iso (b, classify) = Iso{to, from} where
  to :: (s, Natural) -> Natural
  to (s, n) = [ k | k <- [b..], classify k == s ] `genericIndex` (n-1)

  from :: Natural -> (s, Natural)
  from n =
    let s = classify n
        n' = genericLength [ () | k <- [b..n], classify k == s ]
    in (s, n')

For every function $g_1\colon \mathbb{Z}_{\geq \beta} \to \{0,\ldots,B-1\}$ (named classify in Haskell), we have a pair of encode/decode functions, provided that each of the sets $A_s$ is infinite. In particular, we can get the standard encode/decode functions (originally defined by std_iso) by setting classify to

classify_mod_base :: forall s . (Bounded s, Enum s) => (Natural, Natural -> s)
classify_mod_base = (base @s, \n -> n2s (n `mod` base @s))

By varying $g_1$ (and therefore the sets $A_s$), we can control which inputs get mapped to smaller integers.

If $A_s$ is more dense, $f(s,n)$, defined as $n$th smallest number in $A_s$, will be smaller.

If $A_s$ is more sparse, $f(s,n)$ will be larger.

The standard isomorphism makes the sets \[A_s = \{ s+n\cdot B \mid n\in \mathbb Z_{\geq 1} \} \] equally dense for all values of $s$. This makes sense when all $s$ are equally probable.

But in general, we should make $A_s$ denser for those $s$ that are more frequent. Specifically, we want

\[ \frac{|\{k\in A_s \mid k \leq x\}|}{x} \approx p_s. \]

Substituting $x=f(s,n)$ then gives $\log_2 f(s,n) \approx \log_2 n + \log_2 (1/p_s)$. This means that adding a symbol $s$ costs $\log_2 (1/p_s)$ bits, which is what we should strive for.

Here’s a simple example of a suitable $g_1$:

classify_prob :: Show s => (Bounded s, Enum s) => [Double] -> (Natural, Natural -> s)
classify_prob probs =
  let beta = 2 -- arbitrary number > 1
      t = genericLength l
      l = concatMap (\(s, t) -> replicate t s)
        . sortBy (comparing (Down . snd))
        . zip [minBound..maxBound]
        $ map (round . (/ minimum probs)) probs
      g1 n = l `genericIndex` ((n-beta) `mod` t)
  in (beta, g1)

This is a periodic function. It computes the number of times each symbol $s$ will appear within a single period as $k_s=\mathrm{round}(p_s/\min \{p_s\})$. The number $p_s/\min \{p_s\}$ is chosen for its following two properties:

it is proportional to the probability of the symbol, $p_s$;
it is $\geq 1$, so that even the least likely symbol occurs among the values of the function.

The function then works by mapping the first $k_0$ numbers to symbol $0$, the next $k_1$ numbers to symbol $1$, and so on, until it maps $k_{B-1}$ numbers to symbol $B-1$ and repeats itself. The period of the function is $\sum_s k_s\approx 1/\min \{p_s\}$.

classify_prob rearranges the symbols in the order of decreasing probability, which gives further advantage to the more probable symbols. This is probably the best strategy if we want to allocate integers in blocks; a better way would be to interleave the blocks in a fair or random way in order to keep the densities more uniform.

Another downside of this function is that its period may be too small to distinguish between similar probabilities, such as 0.4 and 0.6. The function used in rANS is better in this regard; it uses progressively larger intervals, which provide progressively better approximations.

But classify_prob is enough to demonstate the idea. Let’s encode a list of booleans where True is expected 90% of time.

> iso = ans_iso $ classify_prob [0.1,0.9]
> encode iso (replicate 4 True)
5
> encode iso (replicate 4 False)
11111

Four Trues compress much better than four Falses. Let’s also compare the number of bits in 11111 with the number of bits that the information theory predicts are needed to encode four events with probability 0.1:

> logBase 2 11111
13.439701045971955
> 4 * logBase 2 (1/0.1)
13.28771237954945

Not bad.

The implementation of ANS in this article is terribly inefficient, especially its decoding part, mostly because the isomorphism uses brute force search instead of computation. The intention is to elucidate what the encoding scheme looks like and where it comes from. An efficient implementation of ANS and its different variants is an interesting topic in itself, but I’ll leave it for another day.

The full code (including tests) is available here.

Thanks to /u/sgraf812 for pointing out a mistake in a previous version of classify_prob.

Call an R function from C

Fri, 18 Aug 2017 20:00:00 +0000

This article explains how to call an R function from a C extension.

The “Writing R Extensions” guide gives such an example, but their approach — composing an expression and then evaluating it — strikes me as unnecessarily complicated and roundabout.

Instead, you can use the R_forceAndCall function:

SEXP R_forceAndCall(SEXP e, int n, SEXP rho);

This C function is not documented or part of the official API, so use it on your own risk. We can get some idea about what it does by reading the documentation for the corresponding R function, forceAndCall:

Call a function with a specified number of leading arguments forced before the call if the function is a closure.

So the arguments to R_forceAndCall are:

int n — the number of arguments to force.
SEXP e — a call object — a LANGSXP linked list whose head is the function object and whose tail is function arguments. You construct such lists with the LCONS macro, end them with R_NilValue (which corresponds to R’s NULL), and assign argument names using the SET_TAG macro (if needed).
SEXP rho — the environment in which the evaluation will happen.

This suggests the following implementation:

#include <Rinternals.h>

SEXP R_apply_fun(SEXP f, SEXP x, SEXP rho) {
  SEXP call = PROTECT(LCONS(f, LCONS(x, R_NilValue)));
  SEXP val = R_forceAndCall(call, 1, rho);
  UNPROTECT(1);
  return val;
}

The code relies on the following comment found in do_lapply:

/* Notice that it is OK to have one arg to LCONS do memory
   allocation and not PROTECT the result (LCONS does memory
   protection of its args internally), but not both of them,
   since the computation of one may destroy the other */

The calling environment, rho, needs to be passed in from the R land. Even though R’s own functions can access the current environment through R_GlobalContext->sysparent (see do_envir), neither R_GlobalContext nor do_envir is part of the public API (i.e. they are not declared in installed headers).

So the R wrapper for R_apply_fun should be something like this:

apply_fun <- function(f, x) .Call(R_apply_fun, f, x, environment())

The full example can be downloaded as a git repo:

git clone https://ro-che.info/files/2017-08-18-call-r-function-from-c.git

To test it, install the devtools package, go to the cloned repo, launch R, and run:

devtools::load_all()
apply_fun(function(x) x+1, 3)

6 ways to manage allocated memory in Haskell

Sun, 06 Aug 2017 20:00:00 +0000

Let’s say we have a foreign function that takes a C data structure. Our task is to allocate the structure in memory, fill in the fields, call the function, and deallocate the memory.

The data structure is not flat but contains pointers to other data structures that also need to be allocated. An example of such a data structure is a linked list:

typedef struct list {
  int car;
  struct list *cdr;
} list;

And an example of a C function is one that computes the sum of all elements in the list:

int sum(list *l) {
  int s = 0;
  while (l) {
    s += l->car;
    l = l->cdr;
  }
  return s;
}

In this article, I will explore different ways to track all the allocated pointers and free them reliably.

The complete code can be downloaded as a git repo:

git clone https://ro-che.info/files/2017-08-06-manage-allocated-memory-haskell.git

The modules below use the hsc2hs preprocessor; it replaces things like #{peek ...}, #{poke ...}, and #{size ...} with Haskell code.

Way 1: traverse the structure

Since all pointers are stored somewhere in the data structure, we could traverse it to recover and free those pointers.

mkList :: [Int] -> IO (Ptr List)
mkList l =
  case l of
    [] -> return nullPtr
    x:xs -> do
      struct <- mallocBytes #{size list}
      #{poke list, car} struct x
      xs_c <- mkList xs
      #{poke list, cdr} struct xs_c
      return struct

freeList :: Ptr List -> IO ()
freeList l
  | l == nullPtr = return ()
  | otherwise = do
      cdr <- #{peek list, cdr} l
      free l
      freeList cdr

way1 :: Int -> IO Int
way1 n = bracket (mkList [1..n]) freeList csum

This is how we’d probably do it in C, but in Haskell it has several disadvantages compared to the other options we have:

The code to traverse the structure has to be written manually and is prone to errors.
Having to do the extra work of traversing the structure makes it slower than some of the alternatives.
It is not exception-safe; if something happens inside mkList, the already allocated pointers will be lost. Note that this code is async-exception-safe (bracket masks async exceptions for mkList), so the only exceptions we need to worry about must come from mkList or freeList (e.g. from mallocBytes).

Way 2: allocaBytes and ContT

The Foreign.Marshal.Alloc module provides the bracket-style allocaBytes function, which allocates the memory and then automatically releases it.

-- |@'allocaBytes' n f@ executes the computation @f@, passing as argument
-- a pointer to a temporarily allocated block of memory of @n@ bytes.
-- The block of memory is sufficiently aligned for any of the basic
-- foreign types that fits into a memory block of the allocated size.
--
-- The memory is freed when @f@ terminates (either normally or via an
-- exception), so the pointer passed to @f@ must /not/ be used after this.
allocaBytes :: Int -> (Ptr a -> IO b) -> IO b

It works great if we need to allocate a small fixed number of structures:

allocaBytes size1 $ \ptr1 ->
  allocaBytes size2 $ \ptr2 -> do
    -- do something with ptr1 and ptr2

But what if the number of allocations is large or even unknown, as in our case?

For that, we have the continuation monad!

mallocBytesC :: Int -> ContT r IO (Ptr a)
mallocBytesC n = ContT $ \k -> allocaBytes n k

mallocBytesC is a monadic function that returns a pointer to the allocated memory, so it is as convenient and flexible as the simple mallocBytes used in Way 1. But, unlike mallocBytes, all allocated memory will be safely and automatically released at the point where we run the ContT layer.

mkList :: [Int] -> ContT r IO (Ptr List)
mkList l =
  case l of
    [] -> return nullPtr
    x:xs -> do
      struct <- mallocBytesC #{size list}
      liftIO $ #{poke list, car} struct x
      xs_c <- mkList xs
      liftIO $ #{poke list, cdr} struct xs_c
      return struct

way2 :: Int -> IO Int
way2 n = runContT (liftIO . csum =<< mkList [1..n]) return

This is my favorite way: it is simple, safe, fast, and elegant.

Way 3: ResourceT

If we replace ContT r with ResourceT, we get a similar function with essentially the same semantics: the memory released when the ResourceT layer is run.

mallocBytesR :: Int -> ResourceT IO (Ptr a)
mallocBytesR n = snd <$> allocate (mallocBytes n) free

The rest of the code barely changes:

mkList :: [Int] -> ResourceT IO (Ptr List)
mkList l =
  case l of
    [] -> return nullPtr
    x:xs -> do
      struct <- mallocBytesR #{size list}
      liftIO $ #{poke list, car} struct x
      xs_c <- mkList xs
      liftIO $ #{poke list, cdr} struct xs_c
      return struct

way3 :: Int -> IO Int
way3 n = runResourceT (liftIO . csum =<< mkList [1..n])

However, as we shall see, this version is an order of magnitude slower than the ContT version.

Way 4: single-block allocation via mallocBytes

Even though our structure contains lots of pointers, we don’t have to allocate each one of them separately. Instead, we could allocate a single chunk of memory and calculate the pointers ourselves.

This method may be more involved for complex data structures, but it is the fastest one, because we only need to keep track of and deallocate a single pointer.

writeList :: Ptr List -> [Int] -> IO ()
writeList ptr l =
  case l of
    [] -> error "writeList: empty list"
    [x] -> do
      #{poke list, car} ptr x
      #{poke list, cdr} ptr nullPtr
    x:xs -> do
      #{poke list, car} ptr x
      let ptr' = plusPtr ptr #{size list}
      #{poke list, cdr} ptr ptr'
      writeList ptr' xs

mkList :: [Int] -> IO (Ptr List)
mkList l
  | null l = return nullPtr
  | otherwise = do
      ptr <- mallocBytes (length l * #{size list})
      writeList ptr l
      return ptr

way4 :: Int -> IO Int
way4 n = bracket (mkList [1..n]) free csum

Way 5: single-block allocation via allocaBytes + ContT

This is the same as Way 4, except using allocaBytes instead of mallocBytes. The two functions allocate the memory differently, so I thought I’d add this version to the benchmark.

mkList :: [Int] -> ContT r IO (Ptr List)
mkList l
  | null l = return nullPtr
  | otherwise = do
      ptr <- mallocBytesC (length l * #{size list})
      liftIO $ writeList ptr l
      return ptr

way5 :: Int -> IO Int
way5 n = runContT (liftIO . csum =<< mkList [1..n]) return

Way 6: managed

This is the same as Way 2, but using the managed library by Gabriel Gonzalez. (Thanks to reddit user /u/andriusst for bringing it up.)

mallocBytesC :: Int -> Managed (Ptr a)
mallocBytesC n = managed $ \k -> allocaBytes n k

mkList :: [Int] -> Managed (Ptr List)
mkList l =
  case l of
    [] -> return nullPtr
    x:xs -> do
      struct <- mallocBytesC #{size list}
      liftIO $ #{poke list, car} struct x
      xs_c <- mkList xs
      liftIO $ #{poke list, cdr} struct xs_c
      return struct

way6 :: Int -> IO Int
way6 n = with (liftIO . csum =<< mkList [1..n]) return

Managed is just a newtype wrapper around the polymorphic continuation monad (a.k.a. the Codensity monad) applied to IO. Because the underlying implementation is the same, the run time performance is exactly that of Way 2, but you get to work with simpler types: Managed instead of ContT r IO.

Is sum pure?

We expose the sum function from C as an IO function csum:

foreign import ccall "sum" csum :: Ptr List -> IO Int

The alternative is to expose it as a pure function:

foreign import ccall "sum" csum :: Ptr List -> Int

The Haskell FFI explicitly allows both declarations, and it may seem that a function summing up numbers deserves to be pure.

However, declaring csum as pure would break every single example above (after making the trivial changes to make it type check). Can you see why?

The answer is laziness. If we made csum pure and naively changed our code to compile, we might arrive at something like

bracket (mkList [1..n]) freeList (return . csum)

This tells the compiler to:

Call mkList [1..n];
Pass the result to the pure csum function, and promote the value to the IO type;
Call freeList;
Return the value from step (2).

You may think that the value returned from step (2), and therefore from the whole expression, is a number — the length of the list. But because of laziness, it’s not a number yet. It is a thunk, a delayed application of csum to the pointer value. This thunk will be forced at the next step — when we pass it to the print function, for example, or compare it with the expected value. Only then will csum try to do its work. But it will be too late, because by that time freeList will have run, and csum will discover that its pointer argument is no longer a valid pointer.

To work around this problem, we could replace return with evaluate, but this just shows us that our decision to declare csum was a mistake. It matters in what order mkList, csum, and freeList are evaluated, and the principled way to enforce this order in Haskell is the IO monad.

Benchmark results

The benchmark consists of allocating, summing, and deallocating a list of numbers from 1 to 100.

benchmarking way1
time                 3.301 μs   (3.277 μs .. 3.345 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 3.305 μs   (3.288 μs .. 3.341 μs)
std dev              76.34 ns   (44.80 ns .. 142.3 ns)
variance introduced by outliers: 27% (moderately inflated)

benchmarking way2
time                 2.183 μs   (2.147 μs .. 2.227 μs)
                     0.998 R²   (0.997 R² .. 1.000 R²)
mean                 2.161 μs   (2.145 μs .. 2.185 μs)
std dev              65.91 ns   (46.76 ns .. 97.04 ns)
variance introduced by outliers: 40% (moderately inflated)

benchmarking way3
time                 13.21 μs   (12.92 μs .. 13.49 μs)
                     0.997 R²   (0.996 R² .. 0.998 R²)
mean                 12.74 μs   (12.59 μs .. 12.93 μs)
std dev              574.0 ns   (437.5 ns .. 751.6 ns)
variance introduced by outliers: 54% (severely inflated)

benchmarking way4
time                 1.580 μs   (1.560 μs .. 1.605 μs)
                     0.999 R²   (0.998 R² .. 0.999 R²)
mean                 1.582 μs   (1.570 μs .. 1.601 μs)
std dev              51.28 ns   (37.48 ns .. 72.14 ns)
variance introduced by outliers: 44% (moderately inflated)

benchmarking way5
time                 1.434 μs   (1.418 μs .. 1.459 μs)
                     0.998 R²   (0.997 R² .. 0.998 R²)
mean                 1.511 μs   (1.487 μs .. 1.538 μs)
std dev              83.96 ns   (72.68 ns .. 95.63 ns)
variance introduced by outliers: 70% (severely inflated)

benchmarking way6
time                 2.155 μs   (2.135 μs .. 2.178 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 2.152 μs   (2.139 μs .. 2.171 μs)
std dev              52.54 ns   (40.20 ns .. 69.02 ns)
variance introduced by outliers: 30% (moderately inflated)

Haskell library in a C project

Wed, 26 Jul 2017 20:00:00 +0000

Haskell FFI (Foreign Function Interface) lets us call C code from Haskell and Haskell code from C.

Usually, people are more interested in calling C libraries from Haskell than vice versa. This is how we have bindings to operating systems (like POSIX and Windows APIs), graphics libraries (like GTK and Qt), and highly optimized code (like BLAS or ICU).

But at NStack, we are going in the opposite direction. We want to build a Haskell library that is going to be used by the programming languages that we support, such as Python.

Several places on the internet explain how to write FFI export declarations and initialize the GHC RTS from C: the GHC manual, the Haskell wiki, and a tutorial by Alex Petrov among others.

But none of them address the elephant in the room: how do you turn your Haskell code into a library? You know, an .so file that you can pass to a linker or load with dlopen.

This may seem like a trivial matter, so let’s review what exactly makes it complicated.

When ghc compiles a Haskell package that contains a library, by default it will produce both a dynamic (.so) and a static (.a) library. The dynamic libraries are used to execute Template Haskell code, while the static libaries are used when linking the final executable (again, by default).

These libraries are scattered around the file system, e.g.:

~/.stack/programs/x86_64-linux/ghc-8.0.2/lib/ghc-8.0.2/deepseq-1.4.2.0/libHSdeepseq-1.4.2.0-ghc8.0.2.so
~/.stack/snapshots/x86_64-linux/lts-8.11/8.0.2/lib/x86_64-linux-ghc-8.0.2/libHSwai-3.2.1.1-9yigkTgtHNLHh2mXrnIXo-ghc8.0.2.so
.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/build/libHStasty-0.11.2.1-4ogqtS3RxMEH205xhabqo-ghc8.0.2.so

A typical Haskell package will have hundreds of transitive dependencies, all of which you need to give to the linker. Furthermore, they need to come in the right (reverse topological) order.

Much of this complexity is managed by stack, cabal-install, or ghc --make, so you don’t notice it. Additionally, since by default Haskell depedencies are linked statically, you don’t have to worry about distributing them.

But these tools are designed to build executables, not shared objects, and I couldn’t make them do what I needed.

So I’ve written a perl script that does the following:

constructs the package dependency graph by repeatedly calling ghc-pkg field;
collects all compilation and linking options and writes them to two files, incopts.txt (“inc” stands for “include”) and ldopts.txt;
creates a lib directory and symlinks all the relevant .so files there so that they can be easily distributed.

Because the script uses the low-level ghc-pkg command, it can be used in any Haskell environment — be it stack, cabal-install, or even pure ghc. For instance, to use it with stack, I run

stack exec perl opts.pl libnstack

Here are a few things I’ve learned along the way while writing the script, for your education or amusement:

Each package has two library locations associated with it: library-dirs and dynamic-library-dirs. Except the rts package, which only has library-dirs containing both static and dynamic libraries.
The libraries listed in hs-libraries are divided into “Haskell” and “C” libraries. Their library names start with HS and C, respectively. The division has nothing to do with the language used to produce a library, because the rts, which is written in C, has the name HSrts. The way I understand it is that “Haskell” libraries differ among the ghc versions, while “C” libraries usually don’t. The only example of a “C” library is currently Cffi.

Why is libffi in hs-libraries and not in extra-libraries, together with libgmp, libm etc.? My guess is, to link it statically by default.
The hs-libraries field of the package description gives the static library name. To get the dynamic library name of a “Haskell” library, append -ghc8.0.2 (substitute your compiler version).

To get the dynamic library name of a “C” library (Cffi), strip that C. As the comment in compiler/main/Packages.hs explains:
```
 -- For non-Haskell libraries, we use the name "Cfoo". The .a
 -- file is libCfoo.a, and the .so is libfoo.so. That way the
 -- linker knows what we mean for the vanilla (-lCfoo) and dyn
 -- (-lfoo) ways.
```
Make sure you are linking against the threaded RTS, or else your signals won’t be handled properly, and an innocent Ctrl-C will likely result in a segfault. To link against the threaded RTS, replace HSrts with HSrts_thr.

HSrts is not the only library that has a _thr version; there’s also libCffi_thr.a, but it is identical to libCffi.a.
We’ve paid quite a bit of attention to libCffi, but it’s probably not even used on the major platforms like x86 or x86_64. See rts/Adjustor.c and mk/config.mk.
There’s a field for linker options, called ld-options, which is apparently only used by the rts package. It contains a bunch of flags like
```
-Wl,-u,base_GHCziTopHandler_runNonIO_closure
-Wl,-u,ghczmprim_GHCziTypes_Czh_con_info
-Wl,-u,base_GHCziInt_I8zh_static_info
...
```
I haven’t yet figured out what they do and why they are needed.

Thanks to Ben Gamari for answering some of my questions on IRC and Edward George for helping to debug the script.

Note on Cabal’s foreign libraries

Mikhail Glushenkov points out that, starting from Cabal 2.0, you can use a foreign-library component in a cabal file instead of the usual library.

While this feature is definitely an improvement, it doesn’t seem to address the issues described above and solved by my Perl script:

Link the resulting library using gcc or ld (not ghc).
Distribute the Haskell foreign library together with its dependencies.

Linux audio recording guide (PulseAudio or PipeWire)

Fri, 21 Jul 2017 20:00:00 +0000

This article explains how to record an audio stream on a Linux system running either PulseAudio or PipeWire. (The commands below are all PulseAudio commands, but they should work on a PipeWire system thanks to its PulseAudio compatibility.)

You can record both input streams (microphones) and output streams (whatever you are hearing in your headphones). However, each stream goes to a separate file. If you want to record a conversation, record both streams and mix them afterwards in an audio editor/DAW.

First, find out the PulseAudio name of the stream you want to record.

If you want to record a microphone, run

pactl --format=json list sources | jq '.[] | select(.monitor_source == "") | {name: .name, desc: .description}'

Here is an example output:

{
  "name": "alsa_input.pci-0000_06_00.6.analog-stereo",
  "desc": "Family 17h/19h HD Audio Controller Analog Stereo"
}
{
  "name": "alsa_input.usb-Audient_EVO4-00.analog-surround-40",
  "desc": "EVO4 Analog Surround 4.0"
}

The "name" component is what you need; the description is just there to help you figure out which stream is the right one.

If you want to record an output (e.g. a person you’re talking to), similarly run

pactl --format=json list sources | jq '.[] | select(.monitor_source != "") | {name: .name, desc: .description}'

Once you know the name of the stream you want to record, run

parecord --channels=1 --file-format=flac --device $STREAM_NAME filename.flac

When you are done recording, terminate the process by pressing Ctrl-C.

--channels=1 forces your recording to mono, which is usually what you want.

To see the list of supported file formats, run

parecord --list-file-formats

If you try to record a generic output, such as alsa_output.pci-0000_06_00.6.analog-stereo.monitor, you might also capture some unwanted output from other applications or notification sounds. To make sure only a specific application is recorded, here’s what you can do:

Create a fresh sink that will send its input to headphones, but to which no application is connected by default.
Direct the sound from the specific applications you want to record into this new sink.
Record the output of the new sink.

Find out the name of your output device by running

pactl --format=json list sinks | jq '.[] | {name: .name, desc: .description}'

In my case, it says

{
  "name": "alsa_output.pci-0000_06_00.6.analog-stereo",
  "desc": "Family 17h/19h HD Audio Controller Analog Stereo"
}
{
  "name": "alsa_output.usb-Audient_EVO4-00.analog-surround-40",
  "desc": "EVO4 Analog Surround 4.0"
}

Let’s say my headphones are connected to EVO 4. Then I would run:

OUTPUT_NAME=alsa_output.usb-Audient_EVO4-00.analog-surround-40
pactl load-module module-combine-sink sink_name=recording sink_properties=device.description=Recording slaves=$OUTPUT_NAME
parecord --channels=1 --file-format=flac --device recording.monitor filename.flac

Now, redirect the sound to the Recording sink:

Run the pavucontrol command (a graphical window will appear) and go to the “Playback” tab.
Start the application you’d like to record.
The application should appear in pavucontrol. If it doesn’t, make sure the application produces some sound. Unfortunately, until the application tries to play something, PulseAudio/PipeWire cannot “see” it.
Choose the Recording sink for the application as shown on the screenshot:

All vk.com IP addresses

Fri, 14 Jul 2017 20:00:00 +0000

Here’s the current list of IP addresses owned by the social network vk.com (“vkontakte”/“Вконтакте”):

CIDR	Base address	Netmask
87.240.128.0/18	87.240.128.0	255.255.192.0
93.186.224.0/20	93.186.224.0	255.255.240.0
95.142.192.0/20	95.142.192.0	255.255.240.0
95.213.0.0/17	95.213.0.0	255.255.128.0
185.32.248.0/22	185.32.248.0	255.255.252.0

You can use this list to block, unblock, or apply custom routing to the VK traffic.

Here is the Perl script I wrote to extract this information, in case you want to check whether this table is up-to-date or generate a similar table for a different website/company.

#!/usr/bin/perl -l

use strict;
use warnings;
use Net::Netmask;

my @inetnum;
my @ranges;

while (<>) {
  if (/^inetnum:\s*(\S+) - (\S+)$/) { @inetnum = ($1,$2); }
  if (/^mnt-by:\s*VKONTAKTE-NET-MNT$/) {
    push @ranges, range2cidrlist(@inetnum);
  }
}

for my $cidr (cidrs2cidrs(@ranges)) {
  print $cidr, "\t", $cidr->base(), "\t", $cidr->mask();
}

Usage (if you saved the script as ip_ranges.pl):

wget ftp://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz
zcat ripe.db.inetnum.gz | perl ip_ranges.pl

Can probability be greater than one?

Fri, 30 Jun 2017 20:00:00 +0000

Disclaimer: don’t take this article too seriously. Obviously, a probability greater than one violates many properties of probability (including Kolmogorov’s axioms), and the reasoning here is far from rigorous. I wrote this partly as a joke, partly as food for thought. I do think, however, that this line of reasoning has something to it and could probably be formalized by someone who has more motivation.

Have you ever wondered whether a probability could equal, say, 2?

The (frequentist) definition of probability is the limit of the ratio between the number of successes, $N_s$, and the number of trials, $N$:

\[ p = \lim_{N\to\infty}\frac{N_s}N \]

If the number of successes is allowed to be higher than the number of trials, then nothing prevents $p$ from exceeding 1.

Consider, for instance, the sleeping beauty problem:

Sleeping Beauty volunteers to undergo the following experiment and is told all of the following details: On Sunday she will be put to sleep. Once or twice, during the experiment, Beauty will be awakened, interviewed, and put back to sleep with an amnesia-inducing drug that makes her forget that awakening. A fair coin will be tossed to determine which experimental procedure to undertake: if the coin comes up heads, Beauty will be awakened and interviewed on Monday only. If the coin comes up tails, she will be awakened and interviewed on Monday and Tuesday. In either case, she will be awakened on Wednesday without interview and the experiment ends.

Any time Sleeping Beauty is awakened and interviewed she will not be able to tell which day it is or whether she has been awakened before. During the interview Beauty is asked: “What is your credence now for the proposition that the coin landed heads?”.

See also this video by Julia Galef, from which I originally found out about this problem.

Denote the events of the coin landing heads as $H$, the coin landing tails as $T$, and Beauty wakening up and being interviewed as $I$. Then the problem lies in finding the conditional probability $p(H|I)$.

Conditional probability problems of this kind are solved by the Bayes theorem:

\[ p(H|I)=\frac{p(I|H)p(H)}{p(I|H)p(H)+p(I|T)p(T)} \]

The prior probabilities $p(H)$ and $p(T)$ equal $1/2$ for a fair coin, so

\[ p(H|I)=\frac{p(I|H)}{p(I|H)+p(I|T)}. \]

The probability $p(I|H)$ of Beauty being interviewed under heads is 1: it happens with certainty.

But if the coin comes up tails, the interviewing happens twice as many times. Every time the experiment is conducted and the coin lands tails, the event $I$ occurs two times. The only logical possibility is to assign $p(I|T)$ the value of 2.

Therefore,

\[ p(H|I)=\frac{1}{1+2}=1/3. \]

I guess I’m a thirder then.

Generic unification

Sat, 17 Jun 2017 20:00:00 +0000

The unification-fd package by wren gayle romano is the de-facto standard way to do unification in Haskell. You’d use it if you need to implement type inference for your DSL, for example.

To use unification-fd, we first need to express our Type type as a fixpoint of a functor, a.k.a. an initial algebra.

For instance, let’s say we want to implement type inference for the simply typed lambda calculus (STLC). The types in STLC can be represented by a Haskell type

data Type = BaseType String
          | Fun Type Type

Note that Type cannot represent type variables.

Type can be equivalently represented as a fixpoint of a functor, TypeF:

-- defined in Data.Functor.Fixedpoint in unification-fd
newtype Fix f = Fix { unFix :: f (Fix f) }

data TypeF a = BaseType String
             | Fun a a
  deriving (Functor, Foldable, Traversable)

type Type = Fix TypeF

So Fix TypeF still cannot represent any type variables, but UTerm TypeF can. UTerm is another type defined in unification-fd that is similar to Fix except it includes another constructor for type variables:

-- defined in Control.Unification in unification-fd
data UTerm t v
    = UVar  !v               -- ^ A unification variable.
    | UTerm !(t (UTerm t v)) -- ^ Some structure containing subterms.

type PolyType = UTerm TypeF IntVar

UTerm, by the way, is the free monad over the functor t.

Unifiable

The Control.Unification module exposes several algorithms (unification, alpha equivalence) that work on any UTerm, provided that the underlying functor t (TypeF in our example) implements a zipMatch function:

class (Traversable t) => Unifiable t where
    -- | Perform one level of equality testing for terms. If the
    -- term constructors are unequal then return @Nothing@; if they
    -- are equal, then return the one-level spine filled with
    -- resolved subterms and\/or pairs of subterms to be recursively
    -- checked.
    zipMatch :: t a -> t a -> Maybe (t (Either a (a,a)))

zipMatch essentially tells the algorithms which constructors of our TypeF functor are the same, which are different, and which fields correspond to variables. So for TypeF it could look like

instance Unifiable TypeF where
  zipMatch (BaseType a) (BaseType b) =
    if a == b
      then Just $ BaseType a
      else Nothing
  zipMatch (Fun a1 a2) (Fun b1 b2) =
    Just $ Fun (Right (a1, b1)) (Right (a2, b2))
  zipMatch _ _ = Nothing

Now, I prefer the following style instead:

instance Unifiable TypeF where
  zipMatch a b =
    case a of
      BaseType a' -> do
        BaseType b' <- return b
        guard $ a' == b'
        return $ BaseType a'
      Fun a1 a2 -> do
        Fun b1 b2 <- return b
        return $ Fun (Right (a1, b1)) (Right (a2, b2))

Why? First, I really don’t like multi-clause definitions. But the main reason is that the second definition behaves more reliably when we add new constructors to TypeF. Namely, if we enable ghc warnings (-Wall) and extend TypeF to include tuples:

data TypeF a = BaseType String
             | Fun a a
             | Tuple a a
  deriving (Functor, Foldable, Traversable)

… we’ll get a warning telling us not to forget to implement zipMatch for tuples:

warning: [-Wincomplete-patterns]
    Pattern match(es) are non-exhaustive
    In a case alternative: Patterns not matched: (Tuple _ _)

If we went with the first version, however, we would get no warning, because it contains a catch-all clause

  zipMatch _ _ = Nothing

As a result, it is likely that we forget to update zipMatch, and our tuples will never unify.

This is a common mistake people make when implementing binary operations in Haskell, so I just wanted to point it out. But other than that, both definitions are verbose and boilerplate-heavy.

And it goes without saying that in real-life situations, the types we want to unify tend to be bigger, and the boilerplate becomes even more tedious.

For instance, I’ve been working recently on implementing type inference for the nstack DSL, which includes tuples, records, sum types, optionals, arrays, the void type, and many primitive types. Naturally, I wasn’t eager to write zipMatch by hand.

Generic Unifiable

Generic programming is a set of techniques to avoid writing boilerplate such as our implementation of zipMatch above.

Over the years, Haskell has acquired a lot of different generic programming libraries.

For most of my generic programming needs, I pick uniplate. Uniplate is very simple to use and reasonably efficient. Occasionally I have a problem that requires something more sophisticated, like generics-sop to parse YAML or traverse-with-class to resolve names in a Haskell AST.

But none of these libraries can help us to implement a generic zipMatch.

Consider the following type:

data TypeF a = Foo a a
             | Bar Int String

A proper zipMatch implementation works very differently for Foo and Bar: Foo has two subterms to unify whereas Bar has none.

But most generics libraries don’t see this difference between Foo and Bar. They don’t distinguish between polymoprhic and non-polymorphic fields. Instead, they treat all fields as non-polymorphic. From their point of view, TypeF Bool is exactly equivalent to

data TypeF = Foo Bool Bool
           | Bar Int String

Luckily, there is a generic programming library that lets us “see” type parameters. Well, just one type parameter, but that’s exactly enough for zipMatch. In other words, this library provides a generic representation for type constructors of kind * -> *, whereas most other libraries only concern themselves with ordinary types of kind *.

What is that library called? base.

Seriously, starting from GHC 7.6 (released in 2012), the base library includes a module GHC.Generics. The module consists of:

Several types (constants K1, parameters Par1, sums :+:, products :*:, compositions :.:) out of which we can build different algebraic types of kind * -> *.
A class for representable algebraic data types, Generic1:
```
class Generic1 f where
  type Rep1 f :: * -> *
  from1  :: f a -> (Rep1 f) a
  to1    :: (Rep1 f) a -> f a
```
The associated type synonym Rep1 maps an algebraic data type like TypeF to an isomorphic type composed out of the primitives like K1 and :*:. The functions from1 and to1 allow converting between the two.

The compiler itself knows how to derive the Generic1 instance for eligible types. Here is what it looks like:

{-# LANGUAGE DeriveFunctor, DeriveFoldable, DeriveTraversable #-}
{-# LANGUAGE DeriveGeneric #-}

import GHC.Generics (Generic1)

data TypeF a = BaseType String
             | Fun a a
             | Tuple a a
  deriving (Functor, Foldable, Traversable, Generic1)

So, in order to have a generic Unifiable instance, all I had to do was:

Implement Unifiable for the primitive types in GHC.Generics.
Add a default zipMatch implementation to the Unifiable class.

You can see the details in the pull request.

Complete example

Here is a complete example that unifies a -> (c, d) with c -> (a, b -> a).

{-# OPTIONS_GHC -Wall #-}
{-# LANGUAGE DeriveFunctor, DeriveFoldable,
             DeriveTraversable, DeriveGeneric,
             DeriveAnyClass, FlexibleContexts
  #-}

import Data.Functor.Identity
import Control.Unification
import Control.Unification.IntVar
import Control.Unification.Types
import Control.Monad.Trans.Except
import Control.Monad.Trans.Class
import qualified Data.Map as Map
import GHC.Generics

data TypeF a = BaseType String
             | Fun a a
             | Tuple a a
  deriving (Functor, Foldable, Traversable, Show, Generic1, Unifiable)
  --                                              ^^^^^^^^^^^^^^^^^^^
  --                                           the magic happens here

unified :: IntBindingT TypeF Identity
  (Either
    (UFailure TypeF IntVar)
    (UTerm TypeF String))
unified = runExceptT $ do
  a_var <- lift freeVar
  b_var <- lift freeVar
  c_var <- lift freeVar
  d_var <- lift freeVar
  let
    a = UVar a_var
    b = UVar b_var
    c = UVar c_var
    d = UVar d_var
    
    term1 = UTerm (Fun a (UTerm $ Tuple c d))
    term2 = UTerm (Fun c (UTerm $ Tuple a (UTerm $ Fun b a)))

  result <- applyBindings =<< unify term1 term2

  -- replace integer variable identifiers with variable names
  let
    all_vars = Map.fromList
      [(getVarID a_var, "a")
      ,(getVarID b_var, "b")
      ,(getVarID c_var, "c")
      ,(getVarID d_var, "d")
      ]

  return $ fmap ((all_vars Map.!) . getVarID) result

main :: IO ()
main = print . runIdentity $ evalIntBindingT unified

Output:

Right (Fun "c" (Tuple "c" (Fun "b" "c")))

On friendly contributing policies

Mon, 12 Jun 2017 20:00:00 +0000

Neil Mitchell gave a talk at ZuriHac about drive-by contributions. I did not attend ZuriHac this year, but Neil published the slides of his talk. While I was skimming through them, one caught my attention:

The quote on the bottom-right is taken from the haskell-src-exts contributing policy, which I wrote back when I was its maintainer.

As I said, I didn’t have the chance to attend the talk, and the video does not seem to be released yet, so I can only guess the message of this slide (perhaps Neil or someone who was present will clarify) — but it looks to me that this quote is an example of an unfriendly message that is contrasted to the first, welcoming, message.

If that’s the case, this doesn’t seem fair to me: the quote is cherry-picked from a section that starts with

So, you’ve fixed a bug or implemented an extension. Awesome!

We strive to get every such pull request reviewed and merged within a month.

For best results, please follow these guidelines:

You could argue whether this is friendly enough or not, but you have to admit that it is at least considerate of contributors’ time and interests.

But what about those two particular sentences on the slide? Are they rude? Is it because they don’t have “please” in them?

I guess that, taken in isolation, they do seem rude for that reason, but they are part of a list of instructions, and you can make a list of instructions only so much polite. In particular, I don’t think that simply adding “please” in front of every imperative would make it any nicer — but I am not a native English speaker and could be wrong.

On the other hand, if I wrote that policy as a list of polite requests (“Could you please not put multiple unrelated changes in a single pull request?”), I feel that it would take more time and effort for a potential contributor to comprehend and extract the essence of what’s written. That would be disrespectful to the contributor’s time. Besides, if you have 10+ polite requests in a row, it starts to look as a caricature.

That said, I am open to constructive criticism. If you have ideas about how I could have written it better, please drop me a line.

Update. Neil has written a nice detailed response, which I greatly appreciate.

Word vs. Int

Mon, 05 Jun 2017 20:00:00 +0000

When dealing with bounded non-negative integer quantities in Haskell, should we use Word or Int to represent them?

Some argue that we shoud use Word because then we automatically know that our quantities are non-negative.

because there's many things that shouldn't be negative by semantic. there's no such as -5 coins in your wallet, or a body weight being -1kg.
— Vincent Hanquez (@vincenthz) June 5, 2017

There is a famous in typed functional programming circles maxim that says “make illegal states unrepresentable”. Following this maxim generally leads to code that is more likely correct, but in each specific instance we should check that we are indeed getting closer to our goal (writing correct code) instead of following a cargo cult.

So let’s examine whether avoiding unrepresentable negative states serves us well here.

If our program is correct and never arrives at a negative result, it does not matter whether we use Int or Word. (I’m setting overflow issues aside for now.)

Thus, we only need to consider a case when we subtract a bigger number from a smaller number because of a logic flaw in our program.

Here is what happens depending on what type we use:

> 2 - 3 :: Int
-1
> 2 - 3 :: Word
18446744073709551615

Which answer would you prefer?

Even though technically -1 doesn’t satisfy our constraints and 18446744073709551615 does, I would choose -1 over 18446744073709551615 any day, for two reasons:

There is a chance that some downstream computation will recognize a negative number and report an error.

A stock exchange won’t let me buy -1 shares, and the engine won’t let me set the speed to -1 km/h. (These are terrible examples, I know, but hopefully they illustrate my point.)

Will those systems also reject 18446744073709551615 shares or km/h? If they are well designed, yes, but I’d rather not test this in production.
For a human, it is easier to notice the mistake if the answer does not make any sense at all than if the answer kinda makes sense.

If an experienced programmer sees an unexpectedly huge number like 18446744073709551615, she will easily connect it to an underflow, although it’s an extra logical step she has to make. A less experienced programmer might spend quite a bit of time figuring this out.

In any case, I don’t see any advantage of replacing an invalid number such as -1 with a technically-valid-but-meaningless number like 18446744073709551615.

Ben Millwood said it well:

I moreover feel like, e.g. length :: [a] -> Word (or things of that ilk) would be even more of a mistake, because type inference will spread that Word everywhere, and 2 - 3 :: Word is catastrophically wrong. Although it seems nice to have an output type for non-negative functions that only has non-negative values, in fact Word happily supports subtraction, conversion from Integer, even negation (!) without a hint that anything has gone amiss. So I just don’t believe that it is a type suitable for general “positive integer” applications.

Moritz Kiefer makes another great point:

One problem that I’ve run into quite a few times with Word is [0 .. n - 1] which is quite common if you’re mapping over array indices. That alone has mostly resulted in me being really careful when I use Word.

Here Moritz is referring to the fact that the expression [0 .. n-1] :: [Word] results in the list of all integers 0 <= i < n unless n is 0, in which case n-1 wraps to maxBound.

Natural

There is a way to get the best of both worlds, though: the Natural type from Numeric.Natural.

It is an arbitrary-precision integer, so we don’t have to worry about overflow.
It is a non-negative type, so the invalid state is not representable.
It raises an exception upon underflow, making the errors in the code prominent:
```
> 2 - 3 :: Natural
*** Exception: arithmetic underflow
```

There are perhaps a couple of valid use cases for Word that I can think of, but they are fairly exotic. (I am not talking here about the types such as Word32 or Word64, which are indespensible for bit manipulation.) Most of the time we should prefer either Int or Natural to Word.

Universally stateless monads

Sun, 04 Jun 2017 20:00:00 +0000

Background: monad-control and stateless monads

The monad-control package allows to lift IO functions such as

forkIO :: IO () -> IO ThreadId

catch :: Exception e => IO a -> (e -> IO a) -> IO a

allocate :: MonadResource m => IO a -> (a -> IO ()) -> m (ReleaseKey, a)

to IO-based monad stacks such as StateT Int (ReaderT Bool IO).

The core idea of the package is the associated type StM, which, for a given monad m and result type a, calculates the “state” of m at a.

The “state” of a monad is whatever the “run” function for this monad returns.

For instance, for StateT Int IO Char, we have

runStateT :: StateT Int IO Char -> Int -> IO (Char, Int)

The result type of this function (minus the outer monad constructor, IO, which is always there) is (Char, Int), and that is what StM (StateT Int IO) Char should expand to:

> :kind! StM (StateT Int IO) Char
StM (StateT Int IO) Char :: *
= (Char, Int)

In this case, StM m a is not the same as a; it contains a plus some extra information.

In other cases, StM m a may not contain an a at all; for instance

> :kind! StM (ExceptT Text IO) Char
StM (ExceptT Text IO) Char :: *
= Either Text Char

and we cannot always extract a Char from Either Text Char.

For some monads, though, StM m a reduces precisely to a. I call such monads “stateless”. A notable example is the reader monad:

> :kind! StM (ReaderT Int IO) Bool
StM (ReaderT Int IO) Bool :: *
= Bool

Definition. A monad m is stateless at type a iff StM m a ~ a.

Note that a monad like ReaderT (IORef Int) IO is also stateless, even though one can use it to implement stateful programs.

The important feature of stateless monads is that we can fork them without duplicating the state and terminate them without losing the state. The monad-control package works best with stateless monads: it is less tricky to understand, and you can do some things with stateless monads that are hard or impossible to do with arbitrary MonadBaseControl monads.

Universally stateless monads

When both the monad m and the result type a are known, the compiler can expand the associated synonym StM m a and decide whether StM m a ~ a.

However, there are good reasons to keep the monad m polymorphic and instead impose the constraints (e.g. MonadReader Int m) that m must satisfy.

In this case, the compiler cannot know a priori that m is stateless, and we need to explicitly state that in the function signature. In Taking advantage of type synonyms in monad-control, I showed one such example: running a web application with access to the database. In order to convince the compiler that m is stateless, I needed to add the constraint

StM m ResponseReceived ~ ResponseReceived

to the type signature.

As you can see, this doesn’t quite say “monad m is stateless”; instead it says “monad m is stateless at type a” (where a is ResponseReceived in the above example).

This is fine if we only use monad-control at one result type. But if we use monad-control functions at many different types, the number of constraints required quickly gets out of hand.

As an example, consider the allocate function from resourcet’s Control.Monad.Trans.Resource:

allocate :: MonadResource m => IO a -> (a -> IO ()) -> m (ReleaseKey, a)

As the module’s documentation says,

One point to note: all register cleanup actions live in the IO monad, not the main monad. This allows both more efficient code, and for monads to be transformed.

In practice, it is often useful for the register and cleanup actions to live in the main monad. monad-control lets us lift the allocate function:

{-# LANGUAGE FlexibleContexts, TypeFamilies, ScopedTypeVariables #-}
import Control.Monad.Trans.Control
import Control.Monad.Trans.Resource

allocateM
  :: forall m a . (MonadBaseControl IO m, MonadResource m,
                   StM m a ~ a, StM m () ~ (), StM m (ReleaseKey, a) ~ (ReleaseKey, a))
  => m a -> (a -> m ()) -> m (ReleaseKey, a)
allocateM acquire release =
  liftBaseWith
    (\runInIO -> runInIO $ allocate
      (runInIO acquire)
      (runInIO . release))

This small function requires three different stateless constraints — constraints of the form StM m x ~ x — and two additional stateless constraints per each additional type a at which we use it.

These constraints are artefacts of the way type synonyms work in Haskell; StM m a is not really supposed to depend on a. If a monad is stateless, it is (pretty much always) stateless at every result type.

Definition. A monad m is universally stateless iff for all a, StM m a ~ a.

In order to capture this universal statelessness in a single constraint, we can use Forall from the constraints package.

First, we need to transform the constraint StM m a ~ a to a form in which it can be partially applied, as we want to abstract over a. Simply saying

type StatelessAt m a = StM m a ~ a

won’t do because type synonyms need to be fully applied: StatelessAt m is not a valid type.

We need to use a trick to create a class synonym:

class    StM m a ~ a => StatelessAt m a
instance StM m a ~ a => StatelessAt m a

Now StatelessAt is a legitimate class constraint (not a type synonym), and so we can abstract over its second argument with Forall:

type Stateless m = Forall (StatelessAt m)

Now we only need to include the Stateless m constraint, and every time we need to prove that StM m a ~ a for some result type a, we wrap the monadic computation in assertStateless @a (...), where assertStateless is defined as follows:

assertStateless :: forall a m b . Stateless m => (StM m a ~ a => m b) -> m b
assertStateless action = action \\ (inst :: Stateless m :- StatelessAt m a)

The type signature of assertStateless is crafted in such a way that we only need to specify a, and m is inferred to be the “current” monad. We could have given assertStateless a more general type

assertStateless :: forall m a r . Stateless m => (StM m a ~ a => r) -> r

but now we have to apply it to both m and a.

As an example of using assertStateless, let’s rewrite the lifted allocate function to include a single stateless constraint:

allocateM
  :: forall m a . (MonadBaseControl IO m, MonadResource m, Stateless m)
  => m a -> (a -> m ()) -> m (ReleaseKey, a)
allocateM acquire release =
  assertStateless @a $
  assertStateless @() $
  assertStateless @(ReleaseKey, a) $
  liftBaseWith
    (\runInIO -> runInIO $ allocate
      (runInIO acquire)
      (runInIO . release))

Here, assertStateless generated all three StM m x ~ x constraints for us on the fly, from the single universally-quantified constraint Stateless m.

Stateless monad transformers

Let’s say we are writing a function that works in some stateless monad, m:

foo :: (MonadBaseControl IO m, MonadResource m, Stateless m) => m ()
foo = do ...

But locally, it adds another layer on top of m:

foo :: (MonadBaseControl IO m, MonadResource m, Stateless m) => m ()
foo = do
  thing <- getThing
  flip runReaderT thing $ do
    ...

And somewhere in there we need to allocate something:

foo :: (MonadBaseControl IO m, MonadResource m, Stateless m) => m ()
foo = do
  thing <- getThing
  flip runReaderT thing $ do
    resource <- allocateM acq rel
    ...

The compiler won’t accept this, though:

• Could not deduce: StM
                      m (Data.Constraint.Forall.Skolem (StatelessAt (ReaderT () m)))
                    ~
                    Data.Constraint.Forall.Skolem (StatelessAt (ReaderT () m))
    arising from a use of ‘allocateM’
  from the context: (MonadBaseControl IO m,
                     MonadResource m,
                     Stateless m)
    bound by the type signature for:
               foo :: (MonadBaseControl IO m, MonadResource m, Stateless m) =>
                      m ()

In order to run allocateM in the inner environment, ReaderT Thing m, we need to satisfy the constraint Stateless (ReaderT Thing), which is different from the Stateless m that we have in scope.

If the acq and rel actions do not need to access the thing, we can avoid the problem by lifting the action to the outer environment:

foo :: (MonadBaseControl IO m, MonadResource m, Stateless m) => m ()
foo = do
  thing <- getThing
  flip runReaderT thing $ do
    resource <- lift $
      -- this happens in m
      allocateM acq rel
    ...

But what if acq and rel do need to know the thing?

In that case, we need to prove to the compiler that for all m, Stateless m implies Stateless (ReaderT Thing). This should follow from the fact that ReaderT e is itself a “stateless transformer”, meaning that it doesn’t change the state of the monad that it transforms. As with Stateless, we put this in the form of partially-applicable class and then abstract over a (and m):

class    StM (t m) a ~ StM m a => StatelessTAt t (m :: * -> *) a
instance StM (t m) a ~ StM m a => StatelessTAt t m a

type StatelessT t = ForallV (StatelessTAt t)

Now we need to prove that StatelessT t and Stateless m together imply Stateless (t m). In the notation of the constraints package, this statement can be written as

statelessT :: forall t m . (StatelessT t, Stateless m) :- Stateless (t m)

How to prove it in Haskell is not completely obvious, and I recommend that you try it yourself. I also posted a simplified version of this exercise on twitter the other day.

Anyway, here is one possible answer:

statelessT = Sub $ forall prf
  where
    prf :: forall a . (StatelessT t, Stateless m) => Dict (StatelessAt (t m) a)
    prf = Dict \\ (instV :: StatelessT t :- StatelessTAt t m a)
               \\ (inst  :: Stateless m  :- StatelessAt m a)

Finally, here is a function analogous to assertStateless that brings the Stateless (t m) constraint into scope:

liftStatelessT
  :: forall t m b . (StatelessT t, Stateless m)
  => (Stateless (t m) => (t m) b) -> (t m) b
liftStatelessT action = action \\ statelessT @t @m

And here is a minimal working example that demonstrates the usage of liftStatelessT:

foo :: (MonadBaseControl IO m, MonadResource m, Stateless m) => m ()
foo = do
  flip runReaderT () $ liftStatelessT $ do
    resource <- allocateM (return ()) (const $ return ())
    return ()

monad-unlift

After this article was originally published, MitchellSalad pointed out the monad-unlift library, which Michael Snoyman announced two years earlier, and which I was not aware of.

Indeed, Stateless and monad-unlift (v0.2.0) are strikingly similar.

(“Stateless” is the name of the code and the module described in this article. You can find the contents of the module below. I have not published it on github or hackage.)

But here are some important differences.

the interface

With Stateless, we use the normal MonadBaseControl functions (like liftBaseWith) and then turn StM m a into a with assertStateless.

With monad-unlift, we call askUnliftBase or askRunBase instead of liftBaseWith.

For comparison, here is allocateM implemented with both approaches:

-- Stateless
allocateM
  :: forall m a . (MonadBaseControl IO m, MonadResource m, Stateless m)
  => m a -> (a -> m ()) -> m (ReleaseKey, a)
allocateM acquire release =
  assertStateless @a $
  assertStateless @() $
  assertStateless @(ReleaseKey, a) $
  liftBaseWith
    (\runInIO -> runInIO $ allocate
      (runInIO acquire)
      (runInIO . release))

-- monad-unlift
allocateM
  :: forall m a . (MonadBaseUnlift IO m, MonadResource m)
  => m a -> (a -> m ()) -> m (ReleaseKey, a)
allocateM acquire release = do
  UnliftBase runInIO <- askUnliftBase
  allocate
    (runInIO acquire)
    (runInIO . release)

The monad-lift version is simpler and more concise despite (but, in fact, thanks to) the UnliftBase wrapper.

On the other hand, Stateless integrates better with the other code built on top of monad-control. If we have a third-party “lifted” library whose function returns m (StM m a), Stateless allows us to simplify that down to m a. With monad-unlift, we would have to re-implement the whole function to use different primitives.

dealing with transformers

The approach to transformers also differs between the libraries. With Stateless, we stay on the transformer level and prove that it is stateless:

foo :: forall m . (MonadBaseControl IO m, MonadResource m, Stateless m) => m ()
foo = do
  flip runReaderT () $ liftStatelessT $ do
    resource <- allocateM acq rel
    return ()
  where
    acq :: ReaderT () m ()
    acq = return ()
    rel :: () -> ReaderT () m ()
    rel _ = return ()

With monad-unlift, we bring everything down into the untransformed monad:

foo :: forall m . (MonadBaseUnlift IO m, MonadResource m) => m ()
foo = do
  flip runReaderT () $ do
    Unlift run <- askUnlift
    resource <- lift $ allocateM (run acq) (run . rel)
    return ()
  where
    acq :: ReaderT () m ()
    acq = return ()
    rel :: () -> ReaderT () m ()
    rel _ = return ()

If you call allocateM many times in the same function, “running” everything with monad-unlift may become tedious, but otherwise both approaches are workable.

Here is a case where monad-unlift would not work, however. Consider a function

bar :: (MonadBaseUnlift IO m, MonadReader () m) => m ()

The signature seems reasonable: the function needs to run in a stateless monad so that it can do something like allocateM internally; it also needs access to some configuration parameter, which I replaced with () for simplicity.

We want to call bar from a monad-polymorphic function foo:

foo :: MonadBaseUnlift IO m => m ()

We can’t call bar directly because the MonadReader constraint will not be satisfied. But if we add a ReaderT layer to satisfy it,

foo :: MonadBaseUnlift IO m => m ()
foo = runReaderT bar ()

… then bar’s MonadBaseUnlift constraint is no longer satisfied.

There is just no reasonable way to do this with monad-unlift, as far as I can tell. With Stateless, this becomes trivial:

bar :: (Stateless m, MonadReader () m) => m ()
bar = return ()

foo :: (Monad m, Stateless m) => m ()
foo = runReaderT (liftStatelessT bar) ()

unliftio

Just a couple of weeks after this article was originally published, Michael announced another library in this space: unliftio.

There are two main differences between unliftio and monad-unlift/Stateless:

The base monad is always IO, as the package name implies.
Instead of relying on monad-control to provide the instances for various transformers, and on libraries such as lifted-base and lifted-async to provide the lifted versions of the standard functions, it builds its own ecosystem.

I considered doing roughly the same when I wrote Stateless, but decided against it, as it meant writing more code. But now that Michael bit the bullet, there is no reason not to take advantage of his work.

There are plenty of reasons to prefer the specialized version: simpler and easier to understand types, better type error messages, better type inference, easier deriving (due to the absence of associated types), and no complications with monad transformers.

So, unliftio is what I would recommend today over Stateless or monad-unlift.

Complete code for Stateless

-- (c) Roman Cheplyaka, 2017. License: MIT
{-# LANGUAGE GADTs, ConstraintKinds, MultiParamTypeClasses, FlexibleInstances,
             ScopedTypeVariables, RankNTypes, AllowAmbiguousTypes,
             TypeApplications, TypeOperators, KindSignatures,
             UndecidableInstances, UndecidableSuperClasses #-}

module Stateless where

import Data.Constraint
import Data.Constraint.Forall
import Control.Monad.Trans.Control

class    StM m a ~ a => StatelessAt m a
instance StM m a ~ a => StatelessAt m a

-- | A constraint that asserts that a given monad is stateless
type Stateless m = Forall (StatelessAt m)

-- | Instantiate the stateless claim at a particular monad and type
assertStateless :: forall a m b . Stateless m => (StM m a ~ a => m b) -> m b
assertStateless action = action \\ (inst :: Stateless m :- StatelessAt m a)

class    StM (t m) a ~ StM m a => StatelessTAt t (m :: * -> *) a
instance StM (t m) a ~ StM m a => StatelessTAt t m a

-- | A statement that a monad transformer doesn't alter the state type
type StatelessT t = ForallV (StatelessTAt t)

-- | A proof that if @t@ is a stateless transformer and @m@ is a stateless monad,
-- then @t m@ is a stateless monad
statelessT :: forall t m . (StatelessT t, Stateless m) :- Stateless (t m)
statelessT = Sub $ forall prf
  where
    prf :: forall a . (StatelessT t, Stateless m) => Dict (StatelessAt (t m) a)
    prf = Dict \\ (instV :: StatelessT t :- StatelessTAt t m a)
               \\ (inst  :: Stateless m  :- StatelessAt m a)

-- | Derive the 'Stateless' constraint for a transformed monad @t m@
liftStatelessT
  :: forall t m b . (StatelessT t, Stateless m)
  => (Stateless (t m) => (t m) b) -> (t m) b
liftStatelessT action = action \\ statelessT @t @m

Why your podcast needs https

Mon, 22 May 2017 20:00:00 +0000

Modern browsers warn users when they attempt to send data over an unencrypted http connection. This seems to have been effective, and most dynamic websites today are served over https.

But what if you are setting up a simple static blog or podcast website? Is it worth bothering with https, or is it above your paranoia level?

There are several reasons to use https even for a simple static site: for instance, to prevent ad injection by ISPs. You can easily imagine something even more nefarious, such as an evil twin hotspot in a popular San Francisco coffee shop invisibly adding malicious code to snippets that patrons copy-paste from unsecured blogs.

Yesterday I was at the Euston station in London waiting for a train. There were advertisements for free WiFi all over the place, to which I gratefully connected. I had recently discovered a new podcast, and I queued several episodes to listen to during the 2.5 hour ride.

The episodes downloaded very quickly, but when I tried to listen to them, they would not play.

It turned out that the free WiFi at Euston isn’t free internet; it’s a local network where you can watch ads and read magazines. All traffic, including requests for the mp3 files from my podcatcher, were redirected to their captive portal.

If the links in the podcast’s RSS feed were https links, the program would probably refuse to download the files when presented with no or invalid TLS certificate. (To be fair, it could have also looked at the content type of the files, which was probably text/html.)

But instead, the program simply downloaded the webpage thinking it was an mp3 file. So I had to undo this later by figuring out which episodes were corrupted (some I had downloaded earlier, and they were valid) and how to trigger their re-downloading.

So if you have a podcast, configure https on your host and make all links (especially in the RSS feed) https links. If you don’t do that, some of your listeners may hear this instead of your voice.

Convert time interval to number in R

Thu, 18 May 2017 20:00:00 +0000

Let’s say you have two time values in R, t1 and t2, and you want to compute the distance between them.

A natural thing to try would be

as.numeric(t1-t2)

This should probably return a number in seconds, but it might be in some other units, so you test just to be safe:

> as.numeric(as.POSIXct("2017-05-17 19:00:10")-as.POSIXct("2017-05-17 19:00:00"))
[1] 10

“Ok,” you think, “seconds it is,” and proceed to use an expression like the above in your code.

You just fell victim to an incredibly nasty “feature” of the R date/time API. The as.numeric and other similar functions actually choose the units based on the length of the interval by default:

> as.numeric(as.POSIXct("2007-05-07 09:00:10")-as.POSIXct("2007-05-07 09:00:00"))
[1] 10
> as.numeric(as.POSIXct("2007-05-07 09:10:00")-as.POSIXct("2007-05-07 09:00:00"))
[1] 10
> as.numeric(as.POSIXct("2007-05-07 19:00:00")-as.POSIXct("2007-05-07 09:00:00"))
[1] 10
> as.numeric(as.POSIXct("2007-05-17 09:00:00")-as.POSIXct("2007-05-07 09:00:00"))
[1] 10

To get a consistent behavior, you need to specify the units, e.g.:

> as.numeric(as.POSIXct("2007-05-17 09:00:00")-as.POSIXct("2007-05-07 09:00:00"), units="secs")
[1] 864000

The acceptable units values are "auto", "secs", "mins", "hours", "days", and "weeks".

Please help me with these open source Haskell projects

Thu, 04 May 2017 20:00:00 +0000

Here are some pull requests and issues that have been awaiting my attention for months or years.

Sadly, I haven’t found and probably won’t find enough time for them. So I am asking you for help.

If you can work on and close at least one of the items below, that would be awesome.

Thanks to people who have helped so far:

Alexey Zabelin @alexeyzab — immortal: add wait

Update (2017-06-14). I now have processed most of the PRs here myself.

Pull requests awaiting reviews

These are pull requests submitted to my projects. For each one, I describe what needs to be done.

ansi-terminal: add save, restore, report cursor

https://github.com/UnkindPartition/ansi-terminal/pull/18

Needs a review and testing on different platforms.

ansi-terminal: compatibility with Win32-2.5.0.0 and above

https://github.com/UnkindPartition/ansi-terminal/pull/21

Needs review and testing.

tasty: build with GHCJS

https://github.com/UnkindPartition/tasty/pull/150

This one actually has little to do with tasty or ghcjs; it only needs a PR for the clock package to fall back to time.

tasty: make `--quickcheck-show-replay` always show the seed (not just on failure)

https://github.com/UnkindPartition/tasty/pull/143

Need to figure out how to fix this without going back to random (see this comment).

temporary: canonicalized versions of `withSystemTempDirectory` and `withTempDirectory`

https://github.com/UnkindPartition/temporary/pull/2

Needs another pair of eyes — especially regarding the mysterious withCanonicalizedSystemTempDirectory.

tasty: add soft timeouts

https://github.com/UnkindPartition/tasty/pull/89

When a QuickCheck test runs into a timeout, Tasty does not report the input values generated by QuickCheck for the interrupted run.

This PR is probably affected by this change in exception handling — you’ll need to figure out how exactly and update the pull request appropriately.

immortal: add wait

Done on 2017-05-07 by Alexey Zabelin

https://github.com/UnkindPartition/immortal/pull/2

For graceful shutdown, I wait for threads to finish in my servers main loop. I would like to do the same with Immortal threads.

What needs to be done:

Rebase
Add a test

tasty-golden: add golden test for text files

https://github.com/UnkindPartition/tasty-golden/pull/15

I think the point of this PR is to add an internal diff implementation, but I am not sure.

If you understand and like the idea, please update the PR and make a new case for it; otherwise I’ll probably close it.

tasty: Silence a few GHC warnings

https://github.com/UnkindPartition/tasty/pull/122

Needs rebase and review.

Issues that need to be fixed

Here are a couple of issues that I care about and was going to fix myself, but can’t find the time.

stack: no package names in haddock index

https://github.com/commercialhaskell/stack/issues/2886

xmonad: notifications pop up on top of xscreensaver

https://github.com/xmonad/xmonad/issues/74

Generic zipWith

Tue, 25 Apr 2017 20:00:00 +0000

In response to the traverse-with-class 1.0 announcement, user Gurkenglas asks:

Can you use something like this to do something like gzipWith (+) :: (Int, Double) -> (Int, Double) -> (Int, Double)?

There are two separate challenges here:

How do we traverse two structures in lockstep?
How do we make sure that the values we are combining are of the same type?

Because traverse-with-class implements Michael D. Adams’s generic zipper, I first thought that it would suffice to traverse the two values simultaneously. That didn’t quite work out. That zipper is designed to traverse the structure in all four directions: not just left and right, but also up and down. Therefore, if we want to traverse an (Int, Double) tuple with a Num constraint, all possible substructures — including (Int, Double) itself — must satisfy that constraint. The way this manifests itself is through Rec c constraints, which cannot be satisfied for tuples without defining extra Num instances.

It is possible to design a restricted zipper that would only travel left and right and will not impose any unnecessary constraints. But because we need only a simple one-way traversal, we can get away with something even simpler — a free applicative functor. (Indeed, a free applicative is a building block in Adams’s zipper.)

This is simple and beautiful: because a free applicative functor is an applicative functor, we can gtraverse with it; and because a free applicative functor is essentially a heterogeneous list, we can zip two such things together.

Another way we could approach this is by using Oleg’s Zipper from any Traversable, which is based on the continuation monad. I haven’t tried it, but I think it should work, too.

Now we arrive at the second challenge. In traverse-with-class, when we traverse a heterogeneous value, we observe each field as having an existential type exists a . c a => a. If the type of (+) was something like (Num a1, Num a2) => a1 -> a2 -> a1 — as it is in many object-oriented languages — it would be fine. But in Haskell, we can only add two Num values if they are of the same type.

Packages like one-liner or generics-sop use a type-indexed generic representation, so we can assert field-wise type equality of two structures at compile time. traverse-with-class is not typed in this sense, so we need to rely on run-time type checks via Typeable.

The full code for gzipWith is given below. Note that relying on Ørjan’s free applicative has two important consequences:

We zip from right to left, so that gzipWith @Num (+) [1,2,3] [1,2] evaluates to [3,5], not [2,4].
For GTraversable instances that are right-associated (e.g. the standard GTraversable instance for lists), the complexity is quadratic.

I believe that both of these issues can be resolved, but I don’t have the time to spend on this at the moment.

{-# OPTIONS_GHC -Wall #-}
{-# LANGUAGE ScopedTypeVariables, MultiParamTypeClasses, FlexibleInstances,
             ConstraintKinds, RankNTypes, AllowAmbiguousTypes, TypeApplications,
             UndecidableInstances, GADTs, UndecidableSuperClasses,
             FlexibleContexts, TypeOperators #-}

import Data.Typeable
import Data.Generics.Traversable

-- TypeableAnd c is a synonym for (c a, Typeable a)
class    (c a, Typeable a) => TypeableAnd c a
instance (c a, Typeable a) => TypeableAnd c a

-- Ørjan Johansen’s free applicative functor
data Free c a
  = Pure a
  | forall b. (c b, Typeable b) => Snoc (Free c (b -> a)) b

instance Functor (Free c) where
  fmap f (Pure x) = Pure $ f x
  fmap f (Snoc lft x) = Snoc (fmap (f .) lft) x

instance Applicative (Free c) where
  pure = Pure
  tx <*> Pure e = fmap ($ e) tx
  tx <*> Snoc ty az = Snoc ((.) <$> tx <*> ty) az

unit :: TypeableAnd c b => b -> Free c b
unit = Snoc (Pure id)

toFree :: forall c a . GTraversable (TypeableAnd c) a => a -> Free c a
toFree = gtraverse @(TypeableAnd c) unit

fromFree :: Free c a -> a
fromFree free =
  case free of
    Pure a -> a
    Snoc xs x -> fromFree xs x

zipFree :: (forall b . c b => b -> b -> b) -> Free c a -> Free c a -> Free c a
zipFree f free1 free2 =
  case (free1, free2) of
    (Pure a1, _) -> Pure a1
    (_, Pure a2) -> Pure a2
    (Snoc xs1 (x1 :: b1), Snoc xs2 (x2 :: b2)) ->
      case (eqT :: Maybe (b1 :~: b2)) of
        Nothing -> error "zipFree: incompatible types"
        Just Refl -> Snoc (zipFree f xs1 xs2) (f x1 x2)

gzipWith
  :: forall c a . GTraversable (TypeableAnd c) a
  => (forall b . c b => b -> b -> b)
  -> a -> a -> a
gzipWith f a1 a2 = fromFree $ zipFree f (toFree @c a1) (toFree @c a2)

zippedTuple :: (Int, Double)
zippedTuple = gzipWith @Num (+) (1, 1) (3, pi)
-- (4,4.141592653589793)

traverse-with-class 1.0 release

Sun, 23 Apr 2017 20:00:00 +0000

I have released the 1.0 version of traverse-with-class. This library generalizes many Foldable and Traversable functions to heterogeneous containers such as records.

For instance, you can apply Show to all fields and collect the results:

{-# LANGUAGE TemplateHaskell, MultiParamTypeClasses, FlexibleInstances,
             ConstraintKinds, UndecidableInstances, TypeApplications #-}

import Data.Generics.Traversable
import Data.Generics.Traversable.TH

data User a = User
 { name :: String
 , age  :: Int
 , misc :: a
 }

deriveGTraversable ''User

allFields = gfoldMap @Show (\x -> [show x]) $ User "Alice" 22 True
-- ["\"Alice\"","22","True"]

You also get a free zipper for your data types.

The main change in the version 1.0 is that the constraint with which the traversal is conducted is specified via a visible type application. Type applications weren’t available when I originally wrote this library in 2013, so in that version I used implicit parameters to pass around the annoying proxies.

Thanks to Hao Lian for his help with this transition.

Right after I published this blog post, I saw this tweet:

Days since last mailing list discussion of Foldable tuples:

0

RESTART THE CLOCK!
— Michael Snoyman (@snoyberg) April 23, 2017

Guess what, traverse-with-class provides the sensible Foldable-like instance for tuples:

{-# LANGUAGE FlexibleInstances, TypeApplications #-}

import Data.Generics.Traversable

-- U is a trivial constraint satisfied by all types
class U a
instance U a

tupleLength = gfoldl' @U (\c _ -> c + 1) 0 (1::Int, True)
-- returns 2

Disable weird indentation for R in vim

Mon, 03 Apr 2017 20:00:00 +0000

By default, vim indents multi-line function calls in R in the following strange way:

some_function(
              arg1,
              arg2,
              arg3)

There are two issues with this style:

It steals horizontal space, so if some of the arguments are themselves complex expressions, the lines will quickly become too long.
If you rename some_function or change it to a different function, the whole block needs to be re-aligned.

A saner indentation is

some_function(
  arg1,
  arg2,
  arg3)

where the indentation amount is small and fixed (say, two spaces).

To make vim indent R code this way, add to your .vimrc:

let r_indent_align_args = 0

For more information, see :help ft-r-indent.

Deploying MathJax

Sun, 02 Apr 2017 20:00:00 +0000

Update (2020-03-21): the instructions in this post do not apply for the recently released MathJax-3.0, and I haven’t found a similar optimization recipe for it.

MathJax have just announced that they will shut down their CDN by the end of this month.

Although they suggest an alternative CDN, I have long wanted to host a copy of MathJax myself — this has a nice side-effect of being able to write math-heavy articles while offline (I have a local nginx server running on my laptop that serves a copy of this site).

The major obstacle has been the huge size of a default MathJax installation. It is huge both in terms of the size in bytes (inflating backups) and in terms of the number of files (slowing down jekyll). In their blog post, MathJax developers admit this problem and point to the optimization guide. The guide, unfortunately, contains too much text and too few commands.

Therefore, I want to document here my process of installing MathJax 2.7.0 — so that I can repeat this in the future, but also in the hope that this will be useful for others.

These commands leave only:

English locale (which is built-in)
TeX woff fonts
TeX input
HTML-CSS output

wget https://github.com/mathjax/MathJax/archive/2.7.0.zip
unzip MathJax-2.7.0.zip
rm MathJax-2.7.0.zip
cd MathJax-2.7.0
rm -rf docs test unpacked .gitignore README-branch.txt README.md bower.json \
  CONTRIBUTING.md LICENSE package.json composer.json .npmignore .travis.yml \
  config/ fonts/HTML-CSS/TeX/png/ localization/ \
  extensions/MathML extensions/asciimath2jax.js extensions/jsMath2jax.js \
  extensions/mml2jax.js extensions/toMathML.js
find fonts/HTML-CSS/ -mindepth 1 -maxdepth 1 ! -name TeX -exec rm -rf {} \+
find fonts -mindepth 3 -maxdepth 3 ! -name woff -exec rm -rf {} +
find jax/input/ -mindepth 1 -maxdepth 1 ! -name TeX -exec rm -rf {} \+
find jax/output/ -mindepth 1 -maxdepth 1 ! -name HTML-CSS -exec rm -rf {} \+
find jax/output/HTML-CSS/fonts -mindepth 1 -maxdepth 1 ! -name TeX -exec rm -rf {} \+

This brings the installation from 32069 files and 181MiB down to 182 files and 1.7MiB.

A MathJax config consistent with this installation should look something like:

MathJax.Hub.Config({
  extensions: ["tex2jax.js"],
  jax: ["input/TeX", "output/HTML-CSS"],
  "HTML-CSS": {
    availableFonts: ["TeX"],
    imageFont: null
  },
  MathMenu: {
   showRenderer: false,
   showFontMenu: false,
   showLocale: false
  }
});

Increase the open files limit on Linux

Sun, 26 Mar 2017 20:00:00 +0000

Each process on Linux has several limits associated with it, such as the maximum number of files it can open simultaneously. You can find out your current open files limit by running

ulimit -Sn # soft limit; can be raised up to the hard limit
ulimit -Hn # hard limit

To see all limits, run

ulimit -Sa # soft limits
ulimit -Ha # hard limits

The way you can adjust these limits depends on the particular Linux system (e.g. whether it is systemd-based, and possibly even on the version of systemd) and on the way you logged into the system (via console, gdm, lightdm etc.)

Here I describe a few steps that can help you to increase the open files limit. It’s hard to predict which steps will be relevant, but if you follow all of them, there’s a good chance you will succeed.

Similar instructions should work for other limits, too.

PAM

Edit the file /etc/security/limits.conf and add the following line:

* - nofile 20000

where 20000 is the desired limit. The * means all users, and the - means set both soft and hard limits. See limits.conf(5).

You may want to replace the * with a specific user name. Moreover, to change the limits for root, you may need to write root instead of *.

Check that there are no conflicting declarations under /etc/security/limits.d/*.conf, as those files take precedence.

Next, to ensure that these settings are applied, locate the file under /etc/pam.d that corresponds to your login method (/etc/pam.d/login for console, /etc/pam.d/lightdm for lightdm and so on) and add the following line unless it is already there:

session required pam_limits.so

For the changes to take effect, it should be sufficient to re-login.

On Fedora 25, the PAM settings alone seemed to work for the console login but not for the lightdm login.

systemd

If you are on a systemd-based system, try editing both /etc/systemd/system.conf and /etc/systemd/user.conf and adding the following line under the [Manager] section (see systemd-system.conf(5)):

DefaultLimitNOFILE=20000

Then reboot the system.

I found out about this setting from a comment by Ewan Leith, but changing user.conf alone didn’t work for me; I had to change both user.conf and system.conf.

Haskell without GMP

Fri, 10 Mar 2017 20:00:00 +0000

When you compile a Haskell program with GHC, by default your program is linked against the GMP (GNU Multiple Precision Arithmetic) library. GHC uses GMP to implement Haskell’s arbitrary-precision Integer type.

Because GMP is distributed under the L/GPL licenses, this presents a problem if you want to link the executable statically and distribute it without the implications of LGPL.

Here I’ll show how to compile a Haskell program without GMP. The process consists of two steps:

Install a GHC compiled with integer-simple instead of GMP. You may need to compile such a GHC yourself.
Make sure your dependencies do not use GMP.

These instructions are geared towards stack, but it should be clear how to adapt them to other workflows.

Install GHC with integer-simple

integer-simple is a pure Haskell implementation of the subset of GMP functionality.

Because the Integer type is provided by the base library, and the base library is compiled at the same time as GHC itself, we need a different build of GHC to support integer-simple — although that may change at some point.

At the time of writing, FP Complete distributes integer-simple builds for several recent versions of GHC, but only for Windows. To check, look at the current version of stack-setup-2.yaml and search for “integersimple”.

Thus, on Windows you can say

stack setup --ghc-variant=integersimple 8.0.2

and it will download and install the GHC 8.0.2 based on integer-simple.

If you are inside a stack project, add

ghc-variant: integersimple

to stack.yaml so that stack knows which compiler flavor to use. Also, in this case you don’t need to give stack setup the GHC version or --ghc-variant; these will be taken from stack.yaml.

If there is no precompiled integer-simple GHC for your platform or desired GHC version, you’ll have to build it yourself as I describe below.

Compile GHC with integer-simple

These instructions were tested with GHC 8.0.2 on Linux and macOS.

Check the system requirements
Get the GHC source by either cloning the git repo or downloading the source tarball.
Save the template mk/build.mk.sample as mk/build.mk:
```
cp mk/build.mk.sample mk/build.mk
```
Now, add the following line somewhere in mk/build.mk:
```
INTEGER_LIBRARY=integer-simple
```
While editing that file, also choose the build profile by uncommenting one of the BuildFlavour = lines.
- To test the process, use BuildFlavour = quick.
- Once you are happy with the result, run make distclean and rebuild with BuildFlavour = perf.
Another option I found useful to set in mk/build.mk is
```
BUILD_SPHINX_PDF=NO
```
Otherwise, I get errors because I don’t have various exotic TeX packages installed.
Follow the standard build instructions, except the final make install command.
Run make binary-dist to generate the release tarball.

Download the current stack-setup-2.yaml and add a record for your release, such as

linux64-integersimple-tinfo6:
    8.0.2:
        url: "/home/user/ghc/ghc-8.0.2-x86_64-unknown-linux.tar.xz"
        content-length: 114017964
        sha1: ad38970c4431d44fef38c4696847ba491ef24332

Now you can follow the instructions from the previous section, except replace the stack-setup-2.yaml url with the path or url of your own stack-setup-2.yaml file.

Make sure your dependencies do not use GMP

Some packages depend on GMP through the integer-gmp package.

Fortunately, such packages usually have a Cabal flag to remove this dependency or replace it with integer-simple. The flag itself is usually called integer-gmp or integer-simple.

There are different ways to set these flags. With stack, you can declare the flags in stack.yaml as follows:

extra-deps:
- text-1.2.2.1
- hashable-1.2.5.0
- scientific-0.3.4.10
- integer-logarithms-1.0.1
- cryptonite-0.22

flags:
  text:
    integer-simple: true
  hashable:
    integer-gmp: false
  scientific:
    integer-simple: true
  integer-logarithms:
    integer-gmp: false
  cryptonite:
    integer-gmp: false

The above YAML snippet can be easily turned into a custom snapshot and shared among multiple stack projects if needed.

References

Group data by month in R

Wed, 22 Feb 2017 20:00:00 +0000

I often analyze time series data in R — things like daily expenses or webserver statistics. And just as often I want to aggregate the data by month to see longer-term patterns.

Doing this in base R is a bit awkward, and the internet is littered with working but terrible solutions using strftime or similar date-to-string conversions.

The real gem is the function floor_date from the lubridate package. I found out about it from this old StackOverflow answer by Hadley Wickham. As the name implies, it can be used to round each date down to the month boundary, so that dates in the same month are rounded down to the same date.

Let’s look at an example.

library(dplyr)
library(lubridate)
set.seed(2017)
options(digits=4)

Say these are your daily expenses for 2016:

(expenses <- data_frame(
  date=seq(as.Date("2016-01-01"), as.Date("2016-12-31"), by=1),
  amount=rgamma(length(date), shape = 2, scale = 20)))

## # A tibble: 366 × 2
##          date amount
##        <date>  <dbl>
## 1  2016-01-01  75.42
## 2  2016-01-02  28.14
## 3  2016-01-03  51.12
## 4  2016-01-04  26.12
## 5  2016-01-05  42.09
## 6  2016-01-06  40.99
## 7  2016-01-07  45.59
## 8  2016-01-08  57.55
## 9  2016-01-09  22.98
## 10 2016-01-10  14.50
## # ... with 356 more rows

Then you can summarize them by month like this:

expenses %>% group_by(month=floor_date(date, "month")) %>%
   summarize(amount=sum(amount))

## # A tibble: 12 × 2
##         month amount
##        <date>  <dbl>
## 1  2016-01-01 1200.9
## 2  2016-02-01 1002.9
## 3  2016-03-01 1237.6
## 4  2016-04-01 1120.8
## 5  2016-05-01 1276.6
## 6  2016-06-01 1404.6
## 7  2016-07-01  972.6
## 8  2016-08-01 1245.7
## 9  2016-09-01 1020.2
## 10 2016-10-01  986.2
## 11 2016-11-01 1106.3
## 12 2016-12-01 1235.2

floor_date lets you round dates to various time periods from seconds to years and also multiples of these periods, e.g.

expenses %>% group_by(month=floor_date(date, "14 days")) %>%
   summarize(amount=sum(amount))

## # A tibble: 36 × 2
##         month amount
##        <date>  <dbl>
## 1  2016-01-01  550.0
## 2  2016-01-15  462.1
## 3  2016-01-29  188.8
## 4  2016-02-01  568.2
## 5  2016-02-15  335.1
## 6  2016-02-29   99.6
## 7  2016-03-01  675.7
## 8  2016-03-15  459.8
## 9  2016-03-29  102.2
## 10 2016-04-01  458.3
## # ... with 26 more rows

Theory behind RSEM

Sun, 29 Jan 2017 20:00:00 +0000

In this article, I will walk through and try to explain a 2009 paper RNA-Seq gene expression estimation with read mapping uncertainty by Bo Li, Victor Ruotti, Ron M. Stewart, James A. Thomson, and Colin N. Dewey.

I will also occasionally refer to a 2011 paper by Bo Li and Colin N. Dewey, RSEM: accurate transcript quantification from RNA-Seq data with or without a reference genome.

Together, the two papers explain the theory behind the software package for estimating gene and isoform expression levels RSEM.

Motivation

An RNA-Seq experiment proceeds roughly as follows:

A class of RNA molecules (e.g., mRNA) is extracted from a biological sample.
The molecules are fragmented. The size of RNA fragments is in the hundreds of nucleotides.
A random sample of the RNA fragments is sequenced. The obtained sequences are called reads and usually have a fixed length from 50 to 250 nucleotides.

Typically (but not always), the RNA is reverse-transcribed to DNA either before or after fragmentation, and it is the DNA that gets sequenced, but that’s irrelevant for us here.

The goal of the RNA-Seq data analysis is, given the reads and a list of known gene isoforms (all possible RNA sequences), estimate the relative number of RNA molecules (transcripts) of each isoform in the original biological sample.

So, for instance, given two isoforms

(isoform 1) AAAAAAAAAAA
(isoform 2) UUUUUUUUUUU

and three reads

(read 1)    AAAAAAA
(read 2)    UUUUUUU
(read 3)    AAAAAAA

we might estimate that rougly $2/3$ of all transcripts in the biological sample were from isoform 1 and $1/3$ were from isoform 2.

A simple estimation method is to count the reads that align to each isoform and then adjust the counts for the number of reads and isoform sizes.

However, not all reads align to a single isoform. Isoforms of the same gene are highly similar: they all are built from the same small set of segments (exons). Different genes may also be highly similar. More than a half of the reads in any given experiment may map to more than one isoform. Such reads are called multireads.

We could just ignore multireads and consider only the reads that unambiguously map to one isoform. One problem with this is discarding a lot of useful data. But more importantly, without a careful adjustment, discarding multireads introduces a bias: we will underestimate the abundance of genes that have many active isoforms or other similar genes, as these genes are more likely to produce multireads.

The paper describes a probabilistic model and a computational method to estimate isoform abundances based on all available reads, including multireads.

Simplifications

As in the original paper, we will assume single-end reads and ignore quality scores. Besides, we will make two addional assumptions compared to the paper:

The protocol is strand-specific; that is, we will assume the orientation of the reads is known just so that we have fewer variables to model.
All reads come from a known list of isoforms. Therefore, we will not consider the “noise” isoform ($i=0$ in the paper).

Notation and variables

$N$ is the total number of reads (called library size); $n=1,\ldots,N$.
$M$ is the number of known isoforms; $i=1,\ldots,M$.
$L$ is the read length.
$l_i$ is the length of isoform $i$; $j=1,\ldots,l_i$.
$R_n$ is the sequence of read $n$.
Read $n$ is assumed to start at a position $S_n\in[1,l_i]$ in the isoform $G_n=i\in[1,M]$.
$\tau_i$ is the fraction of transcripts that belong to isoform $i$ out of all transcripts in the sample. When multiplied by million, it is the transcripts per million (TPM) measure.
$\theta_i$ is the prior probability that any single read is derived from isoform $i$: $\theta_i=p(G_n=i)$. Because longer isoforms are expected to produce proportionally more fragments and reads, the relationship between $\tau_i$ and $\theta_i$ is: \[\theta_i=\frac{\tau_i\cdot l_i}{\sum_{k=1}^{M}\tau_{k}\cdot l_{k}},\] \[\tau_i=\frac{\theta_i/l_i}{\sum_{k=1}^{M}\theta_{k}/l_{k}}.\]

In other words, $\theta_i\propto \tau_i\cdot l_i$ subject to $\sum_i \theta_i = \sum_i \tau_i = 1$.
The symbol $\propto$ used above means “proportional to”. The formula $\theta_i\propto \tau_i\cdot l_i$ means that $\theta_i/(\tau_i\cdot l_i)$ is constant and does not depend on $i$. Similarly, $p(\theta)\propto 1$ means that $\theta$ has a constant (uniform) probability density.
Vectors of parameters are denoted by letters without indices:
- $R=(R_1, \ldots, R_N)$
- $G=(G_1, \ldots, G_N)$
- $S=(S_1, \ldots, S_N)$
- $\tau=(\tau_1, \ldots, \tau_M)$
- $\theta=(\theta_1, \ldots, \theta_M)$
$\mathbb{I}$ is the indicator function: $\mathbb{I}_p$ is equal to $1$ if $p$ is true and $0$ otherwise.

Thus, we have five vector-valued random variables (of different dimensions!): $\theta, \tau, G, S, R$. Of these, $R$ is the observed data, and $\theta$ and $\tau$ are the unknown parameters that we wish to estimate from the data. $G$ and $S$ are unobserved variables that we are not directly interested in, but they are necessary to build the model.

Vectors $\tau$ and $\theta$ are equivalent in the sense that they carry the same information, just on different scales. Once we estimate $\theta$, we can get an equivalent estimate of $\tau$, and vice versa, using the conversion formulae given above.

We call $\theta$ a random variable in the Bayesian sense. It is random because its value is not known and some values of $\theta$ are more consistent with the data than the others. There is no particular random process that generates $\theta$, so it is not random in the classical (frequentist) sense.

Probabilistic approach

What does an estimate of $\theta$ (or $\tau$, which is usually more interesting) look like? The most general form is the full posterior probability distribution $p(\tau|R)$. In practice, we need to summarize the distribution to make sense of it. Some useful summaries are:

The maximum a posteriori (MAP) value, or the mode of the posterior distribution, is the value of $\tau$ for which $p(\tau|R)$ has the greatest possible value. This is, roughly speaking, the most likely value of $\tau$ under our model given the observed data.
An uncertainty interval $(\tau^1, \tau^2)$ at level $\gamma$ gives a plausible range of values that $\tau$ might take: \[p(\tau^1 < \tau < \tau^2|R)=\gamma.\]
If we are interested in one or a handful of genes or isoforms, we may want to plot the marginal distributions $p(\tau_i|R)$.
If we are interested in the differential expression of isoforms $i$ and $k$ within the same experiment, we can compute \[p(\tau_i > \tau_k|R).\] Or, we can compare expression in two independent experiments $(\tau, R)$ and $(\tau',R')$ by computing \[p(\tau_i > \tau_i'|R,R').\]

Generative model

We begin by formulating a generative model, a probabilistic process that conceptually generates our data.

Li et al. suggest a model where $G_n$, $S_n$, and $R_n$ are generated for every $n$ as follows:

Draw $G_n$ from the set $\{1,\ldots,M\}$ of all isoforms according to the probabilities $\theta$: \[p(G_n=i|\theta)=\theta_i.\]
Choose where on the isoform the read will start according to the read start position distribution (RSPD) $p(S_n|G_n)$.

The authors suggest two uniform distributions: for poly(A)+ RNA \[p(S_n=j|G_n=i)=\frac{1}{l_i} \cdot \mathbb{I}_{1\leq j\leq l_i}\] and for poly(A)- RNA \[p(S_n=j|G_n=i)=\frac{1}{l_i-L+1} \cdot \mathbb{I}_{1\leq j\leq (l_i-L+1)}.\]

The exact form of the RSPD doesn’t matter as long as it is known in advance. The paper also shows how to estimate the RSPD from data; we will not cover that here.
The read $R_n$ is generated by sequencing the isoform $G_n$ starting from the position $S_n$. If sequencing was perfect, this would be deterministic: $R_n$ would contain exactly the bases $S_n, S_n+1, \ldots, S_n+L-1$ from $G_n$. In reality, sequencing errors happen, and we can read a slightly different sequence. Hence $R_n$ is also a random variable with some conditional distribution $p(R_n|G_n,S_n)$.

As with RSPD, we don’t care about the particular form of the conditional probability $p(R_n|G_n,S_n)$. In the simplest case, it would assign a fixed probability such as $10^{-3}$ to each mismatch between the read and the isoform. More realistically, the error probability would increase towards the end of the read. Better yet, we could extend the model to take quality scores into account.

Note that this model does not accurately represent the physics of an RNA-Seq experiment. In particular, it does not model fragmentation. Essentially, the model assumes that the fragment size is constant and equal to the read length. This is addressed in the 2011 paper, although the model there is still backwards: they first pick an isoform and then a fragment from it, whereas in reality we pick a fragment at random from a common pool, and that determines the isoform. This is the reason why $\theta$ does not represent the isoform frequencies and has to be further normalized by the isoform lengths to obtain $\tau$.

A generative model does not have to match exactly the true data generation process as long as the joint distribution $p(G,S,R|\theta)$ fits the reality. Here, the authors saw a way to simplify a model without sacrificing its adequacy much.

This model describes the following Bayesian network:

We can write the joint probability as \[p(G,S,R|\theta)=\prod_{n=1}^N p(G_n,S_n,R_n|\theta)=\prod_{n=1}^N p(G_n|\theta)p(S_n|G_n)p(R_n|G_n,S_n)\label{JOINT}\tag{JOINT}.\]

Finding $\theta$

How do we get from $p(G,S,R|\theta)$ to $p(\theta|R)$?

We start by marginalizing out the unknowns $G_n$ and $S_n$:

\[\begin{align*} p(R|\theta)=\prod_{n=1}^N p(R_n|\theta) & = \prod_{n=1}^N\sum_{i=1}^M\sum_{j=1}^{l_i} p(R_n,G_n=i,S_n=j|\theta) \\ &=\prod_{n=1}^N\sum_{i=1}^M\sum_{j=1}^{l_i} p(R_n|G_n=i,S_n=j) p(S_n|G_n) \theta_i . \end{align*}\]

Then apply Bayes’ theorem to find $\theta$:

\[p(\theta|R)=\frac{p(R|\theta)p(\theta)}{p(R)}.\]

We could expand $p(R)$ as $\int p(R|\theta)p(\theta)d\theta$, but since $p(R)$ does not depend on $\theta$, we won’t need to calculate it.

Choosing the prior

$p(\theta)$ is the prior probability density of $\theta$. Let’s pretend that we do not have any prior information on the isoform expression levels. Then it may seem reasonable to assume a uniform prior probability density $p(\theta)\propto 1$.

There is a subtle issue with this. Recall that the isoform expression levels are represented by the vector $\tau$, not $\theta$. The vector $\theta$ is “inflated” by the isoform lengths to account for the effect that longer isoforms produce more reads.

Therefore, the prior $p(\theta)\propto 1$ corresponds to the (unfounded) assumption that shorter isoforms are somehow a priori more expressed than longer ones.

A better prior for $\theta$ might be the one for which the vector \[\tau(\theta)=(\tau_1(\theta),\ldots,\tau_M(\theta))=\left(\frac{\theta_1/l_1}{\sum_{k=1}^M \theta_k/l_k},\ldots,\frac{\theta_M/l_M}{\sum_{k=1}^M \theta_k/l_k}\right)\] is uniformly distributed on the unit simplex. The probability density of such distribution is

\[p(\theta)=(n-1)!\cdot\frac{\partial(\tau_1,\ldots,\tau_{M-1})}{\partial(\theta_1,\ldots,\theta_{M-1})},\]

where the “fraction” on the right is the Jacobian determinant. (We parameterize $\theta$ and $\tau$ distributions by the first $M-1$ vector components, viewing $\theta_M=1-\sum_{k=1}^{M-1}\theta_k$ and $\tau_M=1-\sum_{k=1}^{M-1} \tau_k$ as functions.)

How much do the different priors affect the results? It is not clear. Li et al. use the uniform prior on $\theta$ implicitly in the 2009 paper by computing the maximum likelihood estimate and explicitly in the 2011 paper, when computing credible intervals. In the latter paper, they note:

CIs estimated from data simulated with the mouse Ensembl annotation were less accurate (Additional file 7). We investigated why the CIs were less accurate on this set and found that many of the CIs were biased downward due to the Dirichlet prior and the larger number of transcripts in the Ensembl set.

I haven’t tested yet whether the uniform $\tau$ prior improves these CIs.

In this article, we will follow the lead of Li et al. and assume $p(\theta)\propto 1$.

EM explanation

The EM (“expectation-maximization”) algorithm is used to find the maximum a posteriori estimate of $\theta$. As Li et al. point out, the EM algorithm can be viewed as repeated “rescuing” of multireads. Let’s see what that means.

In this section, we will assume that a read either maps to an isoform perfectly or does not map to it at all, and that it can map to each isoform only once (though it can map to several isoforms at the same time). This is just to simplify the presentation. The algorithm and its derivation, which are presented in the next section, can handle the general case.

Intuition

First, consider a case of only two expressed isoforms ($M=2$). In general, there will be some number of reads, say, $N_1$, that map only to isoform 1, some other number, $N_2$, of reads that map only to isoform 2. Finally, there will be $N_{12}=N-N_1-N_2$ reads that map equally well to both isoforms.

The easiest way to estimate $\theta$ is by considering only unambiguously mapped reads. This leads to the “zeroth” approximations

\[\begin{align*} \theta^{(0)}_1&=N_1/(N_1+N_2),\\ \theta^{(0)}_2&=N_2/(N_1+N_2). \end{align*}\]

This is the best we can do considering just $N_1$ and $N_2$. To improve our estimate, we need to extract some information from $N_{12}$.

Consider a particular multiread $n$. Can we guess to which isoform it belongs? No, it maps equally well to both. But can we guess how many of the multireads come from each isoform? The more expressed an isoform is, the more reads we expect it to contribute to $N_{12}$. How do we know how much each isoform is expressed? We don’t — if we knew, we wouldn’t need the algorithm — but we can make a reasonable guess based on the estimate $\theta^{(0)}$.

We need to be careful, though. Would it be valid to estimate the fraction of ambiguous reads that come from isoform 1 as $\theta^{(0)}_1$?

Suppose that isoform 1 is much shorter than isoform 2: $l_1 \ll l_2$. Now consider any particular ambiguous read $R_n$. Does the fact that $l_1 \ll l_2$ increase the probability that this read comes from the longer isoform, i.e. that $G_n=2$?

No — because, according to our assumptions made earlier in this section, $R_n$ maps to either isoform only once. The number of all fragments produced by isoform $i$ is proportional to $\theta_i$; but the number of fragments capable of producing a particular read, $R_n$, is proportional to the transcript abundance, $\tau_i$. The longer an isoform, the more fragments it will produce, but also the lower fraction of these fragments will look anything like $R_n$.

Therefore, we partition the $N_{12}$ ambiguous reads between the two isoforms in proportion $\tau^{(0)}_1:\tau^{(0)}_2$, where $\tau^{(0)}$ is derived from $\theta^{(0)}$ according to the usual formula (see Notation and variables).

Partitioning the multireads gives us the updated counts: $N_1 + N_{12}\cdot\tau^{(0)}_1$ for isoform 1 vs. $N_2 + N_{12}\cdot\tau^{(0)}_2$ for isoform 2. These updated counts lead us to a new estimate of $\theta$, $\theta^{(1)}$: \[\theta^{(1)}_i = \frac{N_i+N_{12}\cdot\tau^{(0)}_i}{N}.\]

And the cycle repeats. We can repeat this procedure until we notice that $\theta^{(r+1)}$ does not differ much from $\theta^{(r)}$.

Simulation

For a higher number of isoforms, the computations are analogous. To see how this works out in practice, consider the following simulated example with 3 isoforms. You can download the R script performing the simulation.

The true values of $l$ and $\theta$ are:

\[ \begin{array}{c|rr} i & l_i & \theta_i \\ \hline 1 & 300 & 0.60 \\ 2 & 1000 & 0.10 \\ 3 & 2000 & 0.30 \\ \end{array} \]

The counts obtained from a thousand RNA-Seq reads are:

\[ \begin{array}{lll} N_1 = 111 & N_{12} = 69 & N_{123} = 144 \\ N_2 = 26 & N_{13} = 311 \\ N_3 = 186 & N_{23} = 153 \\ \end{array} \]

Amazingly, under the assumptions introduced in this section, we don’t need to know the actual reads and how they map to the isoforms — we only need the six numbers above.

The first five successive approximations for $\theta$ made by the EM algorithm are:

\[ \begin{array}{r|rrrrr} i & \theta^{(0)}_i & \theta^{(1)}_i & \theta^{(2)}_i & \theta^{(3)}_i & \theta^{(4)}_i \\ \hline 1 & 0.34 & 0.53 & 0.58 & 0.59 & 0.59 \\ 2 & 0.08 & 0.07 & 0.07 & 0.08 & 0.08 \\ 3 & 0.58 & 0.40 & 0.34 & 0.33 & 0.32 \\ \end{array} \]

The zeroth approximation, $\theta^{(0)}$, is not very accurate, but then it quickly gets much better:

EM derivation

Now suppose our current estimate for $\theta$ is $\theta^{(r)}=\theta^*$; let’s see how we can improve it to get $\theta^{(r+1)}$.

We start with the following algebraic identity:

\[p(\theta|R)=\frac{p(\theta,G,S|R)}{p(G,S|\theta,R)}.\]

Apply logarithms:

\[ \log p(\theta|R)=\log p(\theta,G,S|R) - \log p(G,S|\theta,R)\tag{EM1}\label{EM1}. \]

Define the following operator, $E^*$, which takes the expectation of a function $X(G,S)$ with respect to $G$ and $S$ under the distribution $p(G,S|\theta^*,R)$: \[E^*(X)=\sum_G\sum_S X(G,S)\cdot p(G,S|\theta^*,R).\]

Applying this operator to both sides of $\ref{EM1}$ gives:

\[\log p(\theta|R)=E^*(\log p(\theta,G,S|R)) - E^*(\log p(G,S|\theta,R)).\tag{EM2}\label{EM2}\]

It is important to realize that we are not approximating anything yet or introducing any new assumptions (e.g. that $G$ and $S$ follow a certain distribution). Everything up to this point is algebraic identities.

Now, our goal is to find $\theta$ that is a better estimate than $\theta^*$: $p(\theta|R) > p(\theta^*|R)$. That means increasing the right hand side of $\ref{EM2}$ compared to its value at $\theta=\theta^*$.

The significance of the conditional distribution used in $E^*$ is that the term $- E^*(\log p(G,S|\theta,R))$ will be increased no matter what we set $\theta$ to. This is because $- E^*(\log p(G,S|\theta,R))$ is the cross entropy between the distributions $p(G,S|\theta^*,R)$ and $p(G,S|\theta,R)$. By Gibbs’ inequality, the cross entropy is at its minimum at $\theta=\theta^*$.

Thus, we only need to increase — ideally, maximize — the other term in $\ref{EM2}$, $E^*(\log p(\theta,G,S|R))$. This is called the expected complete-data log-likelihood. Now it’s time to exploit the Bayesian network of our generative model:

\[ p(\theta,G,S|R) = \frac{p(R,G,S,\theta)}{p(R)} = \frac{p(R|G,S)p(S|G)p(G|\theta)p(\theta)}{p(R)}, \]

\[\begin{align*} E^*(\log p(\theta,G,S|R)) & = E^*(\log p(R|G,S))\\ &+E^*(\log p(S|G))\\ &+E^*(\log p(G|\theta))\\ &+\log p(\theta)\\ &-\log p(R). \end{align*}\]

In the above sum, assuming the uniform prior, only one term depends on $\theta$: $E^*(\log p(G|\theta))$.

\[\begin{align*} E^*(\log p(G|\theta))&=E^*(\log\prod_{n=1}^N p(G_n|\theta))\\ & =\sum_{n=1}^N E^*\log p(G_n|\theta) \\ & =\sum_{n=1}^N\sum_{i=1}^M \log\theta_i \cdot p(G_n=i|\theta^*, R_n) \\ & =\sum_{i=1}^M a_i\log \theta_i, \end{align*}\]

where the coefficients $a_i$ can be computed by applying Bayes’ theorem and $\ref{JOINT}$:

\[\begin{align*} a_i&=\sum_{n=1}^N p(G_n=i|\theta^*, R_n)\\ &=\sum_{n=1}^N \frac {\sum_{j=1}^{l_i} p(R_n|G_n=i,S_n=j)\cdot \tau_i^*} {\sum_{k=1}^M\sum_{j=1}^{l_k} p(R_n|G_n=k,S_n=j)\cdot \tau_k^*} \end{align*}\]

We need to maximize $f(\theta)=\sum_i a_i\log \theta_i$ subject to $g(\theta)=\sum_i\theta_i=1$ and $\theta_i\geq 0$.

The method of Lagrange multipliers gives the following necessary condition for the extremum of $f(\theta)$:

\[\frac{\partial f}{\partial \theta_i}=\lambda\frac{\partial g}{\partial \theta_i},\]

or $\theta_i=a_i/\lambda$. Since $\sum_i\theta_i=1$, $\lambda=\sum_i a_i$ and $\theta_i=a_i/\sum_k a_k$.

The function $f$ is concave (as a weighted sum of logarithms) and goes to $-\infty$ at the boundary $\theta_i=0$ when $a_i>0$, therefore, \[\theta_i=\frac{a_i}{\sum_{k=1}^M a_k}\] is the global maximum of $f$ under the constraints and should be taken as the next estimate of the true $\theta$.

Approximation via alignment

Consider the probabilities $p(R_n|G_n=i,S_n=j)$ that are used to compute $a_i$. The probability that $R_n$ could have originated from isoform $i$ at position $j$ exponentially decreases with every mismatch between the sequences of the read and the isoform.

For any given read, most of the pairs $(i,j)$ will result in unrelated dissimilar sequences, which therefore contribute almost nothing to the sum $\sum_{j=1}^{l_i} p(R_n|G_n=i,S_n=j)\cdot \tau_i^*$.

If we approximate these tiny probabilities by $0$, we can replace the whole sum with the sum over only those $j$ for which $R_n$ aligns well to isoform $i$ at position $j$.

The only concern here is to keep the denominator from getting close to $0$. In our case, the denominator is always greater than the numerator, so the only thing we need to worry about is if all the terms get approximated by $0$, that is, the read does not map to anything. This is one of the reasons why Li et al. introduce the noise isoform, which never gets replaced by $0$. An alternative is simply to ignore the reads that do not map anywhere.

How much space does an 8-bit integer occupy in C and Haskell?

Wed, 25 Jan 2017 20:00:00 +0000

How much space does an unsigned 8-bit integer occupy in C and Haskell?

Neither the C99 standard nor the Haskell2010 standard specifies such low-level details, so the answer could in theory be anything. To have something to work with, let’s make the following assumptions:

architecture: x86-64
ABI/calling conventions: System V
C compiler: GCC 6.3
Haskell compiler: GHC 8.0

C

In C, the unsigned 8-bit integer type is called uint8_t. It is defined in the header stdint.h. Its width is guaranteed to be exactly 8 bits; thus, its size is 1 byte.

But how much space does it really occupy? That depends on two factors

whether it is a function argument or return value, or a (local or global) variable
whether it is part of an array or struct

Function arguments and return values

According to the AMD64 System V ABI, the first 6 integer arguments are passed via registers and the rest are passed on the stack. If a function returns a single integer value, it is passed back in a register. Since the integer registers are 64-bit wide, when a uint8_t value is passed in a register, it effectively occupies 8 bytes.

To illustrate, consider this function:

uint8_t plus(uint8_t a, uint8_t b) {
  return a+b;
}

GCC generates the following code:

lea    (%rsi,%rdi,1),%eax
retq

The two arguments are passed in the 64-bit registers %rsi and %rdi. Although the result is written to a 32-bit register %eax, it is part of the 64-bit register %rax, and the other 32 bit of that register cannot be reused easily while %eax is occupied.

What about the arguments passed through the stack? The ABI dictates that their sizes, too, are rounded up to 8 bytes. This allows to preserve stack alignment without complicating the calling conventions.

Example:

uint8_t plus(uint8_t a, uint8_t b, uint8_t c,
             uint8_t d, uint8_t e, uint8_t f,
             uint8_t g) {
  return a+g;
}

translates into

mov    %edi,%eax
add    0x8(%rsp),%al
retq

We see that the g argument is 8 bytes below the stack boundary, (%rsp). These whole 8 bytes are dedicated to our tiny int.

When uint8_t’s are part of a struct or similar, they occupy one byte each. Curiously, if the struct is 16 bytes or smaller, the uint8_t’s will be packed into registers!

struct twobytes {
  uint8_t a;
  uint8_t b;
};

uint8_t plus(struct twobytes p) {
  return p.a+p.b;
}

compiles into

mov    %edi,%eax
movzbl %ah,%eax
add    %edi,%eax
retq

Both bytes are passed inside %edi, and the intermediate 1-byte %ah register is used to take them apart.

Local and global variables

Like function arguments, local variables can reside in registers or on the stack. But unlike function arguments, local variables are not constrained by calling conventions; the compiler can do whatever it wants.

When an 8-bit local variable is stored in a register, it effectively occupies the whole 64-bit register, as there is only one 8-bit “subregister” per general-purpose register (unlike in x86).

What happens to the local uint8_t variables stored on the stack? We can compile this test program to find out:

uint8_t plus(uint8_t a, uint8_t b) {
  volatile uint8_t c = a+b;
  return c;
}

add    %edi,%esi
mov    %sil,-0x1(%rsp)
movzbl -0x1(%rsp),%eax
retq

The volatile keyword is needed to force the compiler to store the local variable c on the stack rather than in a register. As we see, c is stored at -0x1(%rsp), so 1 byte is enough here. This is because there is no alignment requirement for 8-bit integers. The same is true for global variables.

Haskell

In Haskell, the unsigned 8-bit integer type is called Word8. Its canonical module according to the standard is Data.Word, but in GHC, it is originally defined in GHC.Word and then re-exported from Data.Word.

Word8 is a boxed type. The space occupied by every boxed type in Haskell consists of two parts: the header and the payload. Here is a helpful picture from the GHC wiki:

Note that stuff on the bottom of the picture — the info table and the entry code — is read-only static data shared among all instances of the given type and even across multiple copies of the same program, so we don’t count it towards the space occupied by a value.

The header is a structure that (on x86-64) normally consists of 8 bytes — a pointer to the entry code for the object.

The value of our byte is stored in the payload. But how exactly? Let’s look at the definition of Word8 in GHC.Word:

-- Word8 is represented in the same way as Word. Operations may assume
-- and must ensure that it holds only values from its logical range.

data Word8 = W8# Word#
-- ^ 8-bit unsigned integer type

Word# is an unboxed machine-word-sized unsigned integer, i.e. a 64-bit integer for x86-64.

In total, a Word8 lying around occupies 16 bytes. When computing with Word8’s inside some kind of inner loop, they will normally be unboxed into Word#’s and passed around in 8-byte registers or in 8-byte cells on the (Haskell) stack — more or less like in C.

Thus, during computation, Haskell is not that different from C. But what about storage? Can multiple Word8’s be packed together densely?

TwoBytes

Say, we need a structure, TwoBytes, consisting of two Word8’s. We intend to use it as a key and/or element type in a large dictionary, so we’d like to keep it as compact as possible. (Note that Data.Map already adds a 48 bytes overhead per key/value.)

If we declare TwoBytes in the most naive way

data TwoBytes = TwoBytes Word8 Word8

the structure will occupy 56 bytes! TwoBytes would consist of a header (8 bytes) and a payload consisting of two pointers (8 bytes each), each pointing to a Word8 (16 bytes each).

A more efficient way to declare TwoBytes is

data TwoBytes = TwoBytes {-# UNPACK #-} !Word8
                         {-# UNPACK #-} !Word8

This makes the fields strict and unpacked, so that the two bytes are stored directly in TwoBytes’s payload. This occupies 24 bytes — “only” 12 bytes per Word8. Compared to a single Word8, we see some economy, but it only amortizes the header. No matter how many Word8’s we put together, the size won’t get below 8 bytes per Word8.

To pack bytes together, we can use an unboxed vector:

data TwoBytes = TwoBytes {-# UNPACK #-} !(Vector Word8)

To see how much memory this structure occupies, we need to see the definition of Vector and the underlying ByteArray:

-- | Unboxed vectors of primitive types
data Vector a = Vector {-# UNPACK #-} !Int
                       {-# UNPACK #-} !Int
                       {-# UNPACK #-} !ByteArray -- ^ offset, length, underlying byte array
data ByteArray = ByteArray ByteArray#

The runtime representation of ByteArray# is a pointer to the StgArrBytes structure defined in includes/rts/storage/Closures.h:

typedef struct {
    StgHeader  header;
    StgWord    bytes;
    StgWord    payload[FLEXIBLE_ARRAY];
} StgArrBytes;

The space required for a ByteArray# is 8 bytes for the header, 8 bytes for the length, and the payload, rounded up to whole words (see stg_newByteArrayzh in rts/PrimOps.cmm) — so 8 bytes in our case, 24 in total.

The size of Vector, therefore, is 8 bytes for the header, 16 bytes for the offset and length (needed to provide O(1) slicing for vectors), 8 bytes for the pointer to the ByteArray#, and 24 bytes for ByteArray# itself; total of 56 bytes.

This is the opposite of the previous definition in that the representation is asymptotically efficient, requiring 1 byte per Word8, but the upfront cost makes it absolutely impractical for TwoBytes.

Even if we cut out the middleman and used ByteArray directly:

data TwoBytes = TwoBytes {-# UNPACK #-} !ByteArray

… it would only get us to 40 bytes.

The most frugal approach for the case of two bytes is to define

data TwoBytes = TwoBytes {-# UNPACK #-} !Word

(16 bytes) and do packing/unpacking by hand. This is a rare case where a Haskell programmer needs to write code that a C compiler would generate (recall two bytes packed into %edi) and not the other way around.

If GHC provided a Word8# unboxed type, we could use the earlier defined

data TwoBytes = TwoBytes {-# UNPACK #-} !Word8
                         {-# UNPACK #-} !Word8

which would still occupy 16 bytes but be more conventient to work with than a single Word. But that’d require a major change to the compiler, and it’s probably not worth the hassle.

Summary

In both C and Haskell, a byte-sized integer occupies 8 bytes when it is actively worked upon (i.e. kept in a register) and 1 byte when many of them are stored in an array/vector.

However, when storing single Word8’s or small structures like TwoBytes, Haskell is not as memory-efficient. Primarily this is because idiomatic Haskell relies heavily on pointers and everything is word-aligned.

Getting Random Things Done: fetching a random card from Trello

Sun, 22 Jan 2017 20:00:00 +0000

I am a big fan of randomness. Often when I struggle with a decision, be it what to have for lunch or how to name a baby, I pick a ~~magic ball~~ random number generator and do whatever it tells me.

Randomness is especially helpful in deciding what to do next. This is a decision we make many times a day, and it can consume a lot of mental energy.

In Getting things done, David Allen suggests a four-criteria model for choosing actions. Four criteria! That sounds like a lot of work in itself. Why don’t we delegate this decision to chance?

Since I use Trello as my GTD system, I developed a simple UNIX shell script to fetch random Trello cards from a particular list (such as “Next actions” or “To read”). If you would like to try it out, follow the instructions below.

How-to

First, go to https://trello.com/app-key and copy the API key it generates. That page also has a link to generate a token “if you are looking to build an application for yourself” — and that’s exactly what you are doing. So go ahead and generate a token.

Save the API key and the token you’ve got to shell variables:

TRELLO_KEY=317130d1b90d72ac17ef53d59ba1bd81
TRELLO_TOKEN=65136e476c86f3688d33440acc7ba10681c9eb756e6557e78d119e28eec8e1bc

Next, you need to find out the id of the list you are interested in. Even the official docs admit that

One of the trickier parts of using the Trello API for simple use cases is finding a List ID that belongs to a user.

Follow their instructions to get the list id. Then create a shell variable to hold it:

TRELLO_LIST=a117f78b518937b5d958fcc8

Once you have the right data in the right variables, simply run the following pipeline in your shell prompt:

curl -Ns "https://api.trello.com/1/lists/$TRELLO_LIST/cards/open?key=$TRELLO_KEY&token=$TRELLO_TOKEN" |
  jq -r '.[] | "\(.name): \(.desc)\n" | @base64' |
  shuf -n 1 |
  base64 --decode

(The base64 encoding/decoding is needed to handle correctly descriptions that span multiple lines.)

You can put the above commands into a file and make it an executable script. For instance, I have scripts random-trello-action and random-trello-paper.

Nested monadic loops may cause space leaks

Tue, 10 Jan 2017 20:00:00 +0000

Consider the following trivial Haskell program:

main :: IO ()
main = worker

{-# NOINLINE worker #-}
worker :: (Monad m) => m ()
worker =
  let loop = poll >> loop
  in loop

poll :: (Monad m) => m a
poll = return () >> poll

It doesn’t do much — except, as it turns out, eat a lot of memory!

% ./test +RTS -s & sleep 1s && kill -SIGINT %1
     751,551,192 bytes allocated in the heap                                               
   1,359,059,768 bytes copied during GC
     450,901,152 bytes maximum residency (11 sample(s))
       7,166,816 bytes maximum slop
             888 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      1429 colls,     0 par    0.265s   0.265s     0.0002s    0.0005s
  Gen  1        11 colls,     0 par    0.701s   0.703s     0.0639s    0.3266s

  INIT    time    0.000s  (  0.000s elapsed)
  MUT     time    0.218s  (  0.218s elapsed)
  GC      time    0.966s  (  0.968s elapsed)
  EXIT    time    0.036s  (  0.036s elapsed)
  Total   time    1.223s  (  1.222s elapsed)

  %GC     time      79.0%  (79.2% elapsed)

  Alloc rate    3,450,267,071 bytes per MUT second

  Productivity  21.0% of total user, 21.0% of total elapsed

These nested loops happen often in server-side programming. About a year ago, when I worked for Signal Vine, this happened to my code: the inner loop was a big streaming computation; the outer loop was something that would restart the inner loop should it fail.

Later that year, Edsko de Vries blogged about a very similar issue.

Recently, Sean Clark Hess observed something similar. In his case, the inner loop waits for a particular AMQP message, and the outer loop calls the inner loop repeatedly to extract all such messages.

So why would such an innocent-looking piece of code consume unbounded amounts of memory? To find out, let’s trace the program execution on the STG level.

Background: STG and IO

The runtime model of ghc-compiled programs is described in the paper Making a Fast Curry: Push/Enter vs. Eval/Apply for Higher-order Languages. Here is the grammar and the reduction rules for the quick reference.

It is going to be important that the IO type in GHC is a function type:

newtype IO a = IO (State# RealWorld -> (# State# RealWorld, a #))

Here are a few good introductions to the internals of IO: from Edsko de Vries, Edward Z. Yang, and Michael Snoyman.

Our program in STG

Let’s see now how our program translates to STG. This is a translation done by ghc 8.0.1 with -O -ddump-stg -dsuppress-all:

poll_rnN =
    sat-only \r srt:SRT:[] [$dMonad_s312]
        let { sat_s314 = \u srt:SRT:[] [] poll_rnN $dMonad_s312; } in
        let { sat_s313 = \u srt:SRT:[] [] return $dMonad_s312 ();
        } in  >> $dMonad_s312 sat_s313 sat_s314;

worker =
    \r srt:SRT:[] [$dMonad_s315]
        let {
          loop_s316 =
              \u srt:SRT:[] []
                  let { sat_s317 = \u srt:SRT:[] [] poll_rnN $dMonad_s315;
                  } in  >> $dMonad_s315 sat_s317 loop_s316;
        } in  loop_s316;

main = \u srt:SRT:[r2 :-> $fMonadIO] [] worker $fMonadIO;

This is the STG as understood by ghc itself. In the notation of the fast curry paper introduced above, this (roughly) translates to:

main = THUNK(worker monadIO realWorld);

worker = FUN(monad ->
  let {
    loop = THUNK(let {worker_poll_thunk = THUNK(poll monad);}
                 in then monad worker_poll_thunk loop);
  } in loop
);

poll = FUN(monad ->
  let {
    ret_thunk = THUNK(return monad unit);
    poll_poll_thunk = THUNK(poll monad);
  }
  in then monad ret_thunk poll_poll_thunk
);

monadIO is the record (“dictionary”) that contains the Monad methods >>=, >>, and return for the IO type. We will need return and >> (called then here) in particular; here is how they are defined:

returnIO = FUN(x s -> (# s, x #));
thenIO = FUN(m k s ->
  case m s of {
    (# new_s, result #) -> k new_s
  }
);
monadIO = CON(Monad returnIO thenIO);
return = FUN(monad ->
  case monad of {
    Monad return then -> return
  }
);
then = FUN(monad ->
  case monad of {
    Monad return then -> then
  }
);

STG interpreters

We could run our STG program by hand following the reduction rules listed above. If you have never done it, I highly recommend performing several reductions by hand as an exercise. But it is a bit tedious and error-prone. That’s why we will use Bernie Pope’s Ministg interpreter. My fork of Ministg adds support for unboxed tuples and recursive let bindings necessary to run our program.

There is another STG interpreter, stgi, by David Luposchainsky. It is more recent and looks nicer, but it doesn’t support the eval/apply execution model used by ghc, which is a deal breaker for our purposes.

We run Ministg like this:

ministg --noprelude --trace --maxsteps=100 --style=EA --tracedir leak.trace leak.stg

Ministg will print an error message saying that the program hasn’t finished running in 100 steps — as we would expect, — and it will also generate a directory leak.trace containing html files. Each html file shows the state of the STG machine after a single evaluation step. You can browse these files here.

Tracing the program

Steps 0 through 16 take us from main to poll monadIO, which is where things get interesting, because from this point on, only code inside poll will be executing. Remember, poll is an infinite loop, so it won’t give a chance for worker to run ever again.

Each iteration of the poll loop consists of two phases. During the first phase, poll monadIO is evaluated. This is the “pure” part. No IO gets done during this part; we are just figuring out what is going to be executed. The first phase runs up until step 24.

On step 25, we grab the RealWorld token from the stack, and the second phase — the IO phase — begins. It ends on step 42, when the next iteration of the loop begins with poll monadIO.

Let’s look at the first phase in more detail. In steps 18 and 19, the let-expression

let {
  ret_thunk = THUNK(return monad unit);
  poll_poll_thunk = THUNK(poll monad);
}
in then monad ret_thunk poll_poll_thunk

is evaluated. The thunks ret_thunk and poll_poll_thunk are allocated on the heap at addresses $3 and $4, respectively.

Later these thunks will be evaluated/updated to partial applications: $3=PAP(returnIO unit) on step 35 and $4=PAP(thenIO $7 $8) on step 50.

We would hope that these partial applications will eventually be garbage-collected. Unfortunately, not. The partial application $1=PAP(thenIO $3 $4) is defined in terms of $3 and $4. $1 is the worker_poll_thunk, the “next” instance of the poll loop invoked by worker.

This is why the leak doesn’t occur if there’s no outer loop. Nothing would reference $3 and $4, and they would be executed and gc’d.

IO that doesn’t leak

The memory leak is a combination of two reasons. As we discussed above, the first reason is the outer loop that holds on to the reference to the inner loop.

The second reason is that IO happens here in two phases: the pure phase, during which we “compute” the IO action, and the second phase, during which we run the computed action. If there was no first phase, there would be nothing to remember.

Consider this version of the nested loop. Here, I moved NOINLINE to poll. (NOINLINE is needed because otherwise ghc would realize that our program doesn’t do anything and would simplify it down to a single infinite loop.)

main :: IO ()
main = worker

worker :: (Monad m) => m ()
worker =
  let loop = poll >> loop
  in loop

{-# NOINLINE poll #-}
poll :: (Monad m) => m a
poll = return () >> poll

In this version, ghc would inline worker into main and specialize it to IO. Here is the ghc’s STG code:

poll_rqk =
    sat-only \r srt:SRT:[] [$dMonad_s322]
        let { sat_s324 = \u srt:SRT:[] [] poll_rqk $dMonad_s322; } in
        let { sat_s323 = \u srt:SRT:[] [] return $dMonad_s322 ();
        } in  >> $dMonad_s322 sat_s323 sat_s324;

main1 =
    \r srt:SRT:[r3 :-> main1, r54 :-> $fMonadIO] [s_s325]
        case poll_rqk $fMonadIO s_s325 of _ {
          (#,#) ipv_s327 _ -> main1 ipv_s327;
        };

Here, poll still runs in two phases, but main1 (the outer loop) doesn’t. This program still allocates memory and runs not as efficient as it could, but at least it runs in constant memory. This is because the compiler realizes that poll_rqk $fMonadIO is not computing anything useful and there’s no point in caching that value. (I am actually curious what exactly ghc’s logic is here.)

What if we push NOINLINE even further down?

main :: IO ()
main = worker

worker :: (Monad m) => m ()
worker =
  let loop = poll >> loop
  in loop

poll :: (Monad m) => m a
poll = do_stuff >> poll

{-# NOINLINE do_stuff #-}
do_stuff :: Monad m => m ()
do_stuff = return ()

STG:

do_stuff_rql =
    sat-only \r srt:SRT:[] [$dMonad_s32i] return $dMonad_s32i ();

$spoll_r2SR =
    sat-only \r srt:SRT:[r54 :-> $fMonadIO,
                         r2SR :-> $spoll_r2SR] [s_s32j]
        case do_stuff_rql $fMonadIO s_s32j of _ {
          (#,#) ipv_s32l _ -> $spoll_r2SR ipv_s32l;
        };

main1 =
    \r srt:SRT:[r3 :-> main1, r2SR :-> $spoll_r2SR] [s_s32n]
        case $spoll_r2SR s_s32n of _ {
          (#,#) ipv_s32p _ -> main1 ipv_s32p;
        };

This code runs very efficiently, in a single phase, and doesn’t allocate at all.

Of course, in practice we wouldn’t deliberately put these NOINLINEs in our code just to make it inefficient. Instead, the inlining or specialization will fail to happen because the function is too big and/or resides in a different module, or for some other reason.

Arities

Arities provide an important perspective on the two-phase computation issue. The arity of then is 1: it is just a record selector. The arity of thenIO is 3: it takes the two monadic values and the RealWorld state token.

Arities influence what happens at runtime, as can be seen from the STG reduction rules. Because thenIO has arity 3, a partial application is created for thenIO ret_thunk poll_poll_thunk. Let’s change the arity of thenIO to 2, so that no PAPs get created:

thenIO = FUN(m k ->
  case m realWorld of {
    (# new_s, result #) -> k
  }
);

(this is similar to how unsafePerformIO works). Now we no longer have PAPs, but our heap is filled with the same exact number of BLACKHOLEs.

More importantly, arities also influence what happens during compile time: what shape the generated STG code has. Because then has arity 1, ghc decides to create a chain of thens before passing the RealWorld token. Let’s change (“eta-expand”) the poll code as if then had arity 4, without actually changing then or thenIO or their runtime arities:

# added a dummy argument s
poll = FUN(monad s ->
  let {
    ret_thunk = THUNK(return monad unit);
    poll_poll_thunk = THUNK(poll monad);
  }
  in then monad ret_thunk poll_poll_thunk s
);
# no change in then or thenIO
then = FUN(monad ->
  case monad of {
    Monad return then -> then
  }
);
thenIO = FUN(m k s ->
  case m s of {
    (# new_s, result #) -> k new_s
  }
);

This code now runs in constant memory!

Therefore, what inlining/specialization does is that it lets the compiler to see the true arity of a function such as then. (Of course, it would also allow the compiler to replace then with thenIO.)

Conclusions

Let me tell you how you can avoid any such space leaks in your code by following a simple rule:

I don’t know.

In some cases, -fno-full-laziness or -fno-state-hack help. In this case, they don’t.

In 2012, I wrote why reasoning about space usage in Haskell is hard. I don’t think anything has changed since then. It is a hard problem to solve. I filed a ghc bug #13080 just in case the ghc developers might figure out a way how to address this particular issue.

Most of the time everything works great, but once in a while you stumble upon something like this. Such is life.

Thanks to Reid Barton for pointing out that my original theory regarding this leak was incomplete at best.

optparse-applicative quick start

Fri, 30 Dec 2016 20:00:00 +0000

When I need to write a command-line program in Haskell, I invariably pick Paolo Capriotti’s optparse-applicative library.

Unfortunately, the minimal working example is complicated enough that I cannot reproduce it from memory, and the example in the README is very different from the style I prefer.

So I decided to put up a template here for a program using optparse-applicative. I am going to copy it into all of my future projects, and you are welcome to do so, too.

import Options.Applicative
import Control.Monad (join)

main :: IO ()
main = join . customExecParser (prefs showHelpOnError) $
  info (helper <*> parser)
  (  fullDesc
  <> header "General program title/description"
  <> progDesc "What does this thing do?"
  )
  where
    parser :: Parser (IO ())
    parser =
      work
        <$> strOption
            (  long "string_param"
            <> short 's'
            <> metavar "STRING"
            <> help "string parameter"
            )
        <*> option auto
            (  long "number_param"
            <> short 'n'
            <> metavar "NUMBER"
            <> help "number parameter"
            <> value 1
            <> showDefault
            )

work :: String -> Int -> IO ()
work _ _ = return ()

Matching country names by local alignment

Sun, 11 Dec 2016 20:00:00 +0000

The other day, I was looking at the Global Terrorism Database published on Kaggle. I wanted to see if the number of terrorist attacks across countries correlated with the population of those countries. The dataset itself didn’t contain the population figures, so I downloaded The World Bank’s population data and tried to merge the two datasets together.

Integrating datasets from different providers is rarely easy. In this case, there were two issues:

The terrorism database gives data about historic countries, while the World Bank maps everything to modern countries.

The terrorism database tells you about the terror acts performed in the Soviet Union in 1978 or Czechoslovakia in 1972, but the World Bank won’t tell you how many people lived in those countries. Instead, it will tell you how many people lived in what today are Russia, Ukraine, Czech Republic, or Slovakia.

It might be possible to match modern countries and historic ones (e.g. calculate the population of the Soviet Union by adding up the population of all Soviet republics), but I was more interested in modern trends, so I decided to ignore the historic countries.
Some of the modern countries have different spellings. A country may have a common name (Russia, Macedonia) and a formal compound name (the Russian Federation, the former Yugoslav Republic of Macedonia). The terrorism database uses the common names, whereas the World Bank uses the formal names, often with idiosyncratic abbreviations, such as “Macedonia, FYR”.

In this article, I compare several methods of matching alternative country spellings. The winner turns out to be local alignment — a method well known in bioinformatics but for some reason rarely used outside of it.

Fuzzy join

Merging datasets based on ill-defined textual values is such a common task that there is a special R package for it, fuzzyjoin. The fuzzyjoin package treats strings as equal if the distance between them is within a user-specified limit. The string distance can be calculated in ten different ways by the stringdist package.

For the task at hand, we will not use fuzzyjoin. fuzzyjoin’s mode of operation is to have a fixed similarity threshold and treat everything similar enough as equal. Whatever distance limit we pick, for some names it will be too tight and there won’t be a single match, and for other names it will be too lax and we will get more than one match.

It makes sense to expect exactly one matching country in the World Bank data for each country from the Global Terrorism Database. So instead of picking an arbitrary threshold, for every country in the first dataset we will pick the best matching name from the second one.

Distance-based methods

Which of the 10 string distance metrics should we use to match country names? First, I tried the classic Levenshtein distance and a couple of others metrics offered by stringdist. They could guess some of the countries but not that many. In order to compare the performance of different metrics, I had to compile the reference table of true matches. (I used the algorithm’s output as a starting point, so it wasn’t too much work.)

Global Terrorism Database	World Bank
Venezuela	Venezuela, RB
Egypt	Egypt, Arab Rep.
Iran	Iran, Islamic Rep.
West Bank and Gaza Strip	West Bank and Gaza
Syria	Syrian Arab Republic
South Korea	Korea, Rep.
Bahamas	Bahamas, The
Hong Kong	Hong Kong SAR, China
Laos	Lao PDR
Republic of the Congo	Congo, Rep.
Yemen	Yemen, Rep.
Russia	Russian Federation
Ivory Coast	Cote d’Ivoire
Bosnia-Herzegovina	Bosnia and Herzegovina
Brunei	Brunei Darussalam
Macedonia	Macedonia, FYR
Gambia	Gambia, The
North Korea	Korea, Dem. People’s Rep.
Macau	Macao SAR, China
Kyrgyzstan	Kyrgyz Republic
Democratic Republic of the Congo	Congo, Dem. Rep.
East Timor	Timor-Leste

Now let’s see how well each metric from stringdist can reconstruct the truth.

methods <- c("osa", "lv", "dl", "hamming", "lcs", "qgram", "cosine", "jaccard", "jw", "soundex")
distMatches <- data_frame(method=methods) %>% group_by(method) %>% do({
  guess <- names2[apply(stringdistmatrix(true_matches$name1, names2, .$method), 1, which.min)]
  mutate(true_matches, name2.guess=guess)
}) %>% ungroup

distMatches %>% group_by(method) %>%
  summarise(score=mean(name2==name2.guess)) %>%
  arrange(desc(score))

	method	score
1	jw	0.50
2	cosine	0.36
3	jaccard	0.32
4	soundex	0.27
5	lcs	0.23
6	qgram	0.23
7	dl	0.14
8	lv	0.14
9	osa	0.14
10	hamming	0.00

Here, score is the fraction of countries successfully matched by a given method. The highest-scoring method, Jaro–Winkler distance, got only half of the countries right. Let’s see what mistakes it made:

distMatches %>% filter(method=="jw",name2!=name2.guess)

Global Terrorism Database	World Bank	jw’s guess
Iran	Iran, Islamic Rep.	Ireland
Syria	Syrian Arab Republic	Serbia
South Korea	Korea, Rep.	South Africa
Republic of the Congo	Congo, Rep.	Puerto Rico
Ivory Coast	Cote d’Ivoire	Honduras
Gambia	Gambia, The	Zambia
North Korea	Korea, Dem. People’s Rep.	North America
Macau	Macao SAR, China	Malta
Kyrgyzstan	Kyrgyz Republic	Kazakhstan
Democratic Republic of the Congo	Congo, Dem. Rep.	Dominican Republic
East Timor	Timor-Leste	Ecuador

Why do these distance metrics perform so poorly? In order to transform “East Timor” to “Timor-Leste”, they need to remove 5 characters on the left and append 6 on the right, which makes a total of 11 edits. On the other hand, to transform “East Timor” to “Ecuador”, they keep the leading “E” and trailing “or” and replace only 7 letters in between. These metrics ignore the similarities and focus on the differences.

Local alignment

A similar problem arises in bioinformatics. Let’s say we are interested in the similarities between the human and bee genomes. Naturally, these genomes are quite different, and if we perform global alignment (as in Levenshtein and most other distance metrics), the relatively few conserved genes will be lost in the mismatch noise.

For this reason, bioinformaticians developed algorithms for local alignment, a similarity metric that rewards similarity without punishing for irrelevant differences. In R, a simple local alignment algorithm is included in Biostring’s pairwiseAlignment function.

localMatches <- true_matches %>%
  mutate(name1 %>%
    sapply(function(n)
      pairwiseAlignment(names2, n, type="local", scoreOnly=T) %>% which.max) %>%
    names2[.])
localMatches %>% summarize(score=mean(name2==name2.guess))

The local alignment method guessed 77% of the countries, compared to only 50% countries guessed by the best distance metric. Here are the countries it didn’t guess:

Global Terrorism Database	World Bank	Local alignment’s guess
South Korea	Korea, Rep.	South Asia
Republic of the Congo	Congo, Rep.	Central African Republic
North Korea	Korea, Dem. People’s Rep.	Middle East & North Africa
Democratic Republic of the Congo	Congo, Dem. Rep.	Central African Republic
East Timor	Timor-Leste	East Asia & Pacific (excluding high income)

Every mistake is caused by a lengthy generic qualifier such as South, North, East, or Republic.

Why isn’t local alignment included in the stringdist package? Well, here’s the thing: strictly speaking, it is not a distance. Any distance metric must obey the triangle inequality:

\[d(x,z)\leq d(x,y)+d(y,z).\]

The local alignment score does not satisfy it: the phrase “local alignment” is highly (locally) similar to both “local” and “alignment”, but “local” and “alignment” are not similar at all.

Does this matter? In some applications, perhaps. But for country names matching, it sure looks like local alignment beats any of the “proper” distance metrics.

And yet local alignment didn’t guess everything it could. I experimented briefly with a multi-alignment algorithm, but, at least in this case, it performed only a little bit better.

Accuracy of quantile estimation through sampling

Wed, 30 Nov 2016 20:00:00 +0000

In Bayesian data analysis, a typical workflow is:

Draw samples $\theta$ from the posterior distribution $p(\theta|y)$ using Markov chain Monte Carlo.
Compute something of interest based on the drawn samples.

For instance, computing the $0.025$ and $0.975$ quantiles from the samples of a scalar parameter $\theta$ yields a $95\%$ credible interval for $\theta$.

But how accurate is the quantile estimation? The accuracy depends on many factors:

number of samples
posterior distribution
the quantile being estimated
estimation method
accuracy metric

In this article, we consider a fixed posterior distribution $(\mathcal{N}(0,1))$ and accuracy metric (root-mean-square error (RMSE)), and investigate how accuracy varies with the number of samples, quantile, and estimation method.

Accuracy depending on number of samples and quantile

There are many ways to compute a sample quantile. We’ll return to them later. For now, we’ll stick with the simplest method: quantile $p$ is estimated as $k$th order statistic (that is, the $k$th-smallest sample), where:

$k$ is defined as $n\cdot p$ rounded to the nearest even number, except when $n\cdot p$ rounds down to zero, in which case $k=1$;
$n$ is the total number of samples;
$p$ is the quantile to be estimated (such as $0.025$).

In R, this can be computed as

quantile(xs, p, names=F, type=3)

Let $F(x)$ and $f(x)$ be the cumulative distribution function (cdf) and probability density function (pdf) of the posterior distribution $\mathcal{N}(0,1)$. Then the density function of the $k$th order statistic is (see here)

\[f_k(x) = n {n-1\choose k-1}F(x)^{k-1}f(x)(1-F(x))^{n-k}.\]

The true $p$th quantile is given by $q=F^{-1}(p)$.

The root-mean-square error of approximating $q$ with $x\sim f_k$ is

\[\sqrt{\int_{-\infty}^{\infty}(x-q)^2f_k(x)dx}.\]

This would be challenging to calculate analytically, but numerically it’s not hard at all. Here are the above definitions translated to R:

quantileLogDensity <- function(p, n) {
  k <- max(round(n*p),1)
  coef <- lchoose(n-1, k-1) + log(n)
  function(x) {
    coef +
        dnorm(x, log = T) +
        pnorm(x, log.p = T)*(k-1) +
        pnorm(x, log.p = T, lower.tail = F)*(n-k)
}}

quantileDensity <- function(p, n)function(x)
  exp(quantileLogDensity(p,n)(x))

rmse <- function(p, n) {
  q <- qnorm(p)
  qd <- quantileDensity(p, n)
  sqrt(integrate(function(x){qd(x) * (q-x)^2}, -Inf, Inf)$value)
}

Note that everything is computed through logarithms to avoid over- and underflow.

And here is the result:

Accuracy depending on method

The method we used above to estimate the population quantile based on sample’s ordered statistics is not the only one. There is a class of quantile estimators based on various weighted sums of order statistics.

As these estimators are already implemented in R’s quantile function as different “types”, we’ll save some time and compare them by simulation instead of computing the integrals.

simulationRmse <- function(p, n, method) {
  replicate(10, {
    qs <- replicate(300, quantile(rnorm(n), p, type = method, names = F))
    sqrt(mean((qs-qnorm(p))^2))})
}

Since RMSE is now a random quantity, we estimate it 10 times and display as a boxplot to visualize the uncertainty.

Conclusions

The estimates of more extreme quantiles (such as $0.025$) are less accurate. See also Are 50% confidence intervals more robustly estimated than 95% confidence intervals?
The naive quantile estimator (type 3) is not great on smaller sample sizes; there are many better alternatives. R’s default type 7 fares well.
Increasing the number of samples improves accuracy (albeit with diminishing returns) and can compensate for an extreme quantile.
For a parameter that is approximately normally distributed with standard deviation $\sigma$, to estimate its $0.025$ quantile within $0.1\sigma$, we need about $750$ samples. For the $0.25$ quantile we need only $200$ samples.

This article was inspired by an exercise from Bayesian Data Analysis (exercise 1, chapter 10).

RNA-Seq normalization explained

Mon, 28 Nov 2016 20:00:00 +0000

RNA-Seq (short for RNA sequencing) is a type of experiment that lets us measure gene expression. The sequencing step produces a large number (tens of millions) of cDNA¹ fragment sequences called reads. Every read represents a part of some RNA molecule in the sample².

Then we assign (“map”) every read to one of the isoforms and count how many reads each isoform has got.

All else being equal, the more abundant an isoform is, the more fragments from it are likely to be sequenced. Therefore, we can use read counts as a proxy for isoform abundance.

However, since “all else” is never equal, the counts need to be adjusted to be comparable across isoforms, samples, and experiments. Here we will explore these adjustments and why they are necessary.

RPK

Consider the following mapped reads from an RNA-Seq experiment. Which isoform is more abundant, the red one or the yellow one?

The yellow isoform has got more reads assigned to it, but it is also much longer than the red one. The longer the isoform, the more fragments (and, therefore, reads) we should expect it to generate.

To be able to compare read counts across isoforms, we divide the counts by the isoform length. It is also customary to multiply the number by $1000$, obtaining reads per kilobase:

\[RPK_i=10^3\cdot\frac{n_i}{l_i},\]

where $n_i$ is the number of reads mapped to isoform $i$, and $l_i$ is the length of that isoform.

RPKM and RPM

Consider the data from two RNA-Seq experiments. In which one is the red isoform more expressed?

If we rely on raw counts, the first experiment has produced more red fragments, and so we may conclude that the red isoform is more expressed there.

Computing $RPK$ won’t change anything since we are comparing expression of the same isoform.

But the first experiment has produced more total fragments sequenced, and the higher the overall number of reads is, the higher count we should expect for any given isoform.

To compare counts across experiments, we should further normalize by the total fragment count (usually expressed in millions). Thus, reads per kilobase per million is computed as

\[RPKM_i=10^9\cdot\frac{n_i}{l_i\cdot \sum_j n_j}.\]

If we do not intend to compare abundances across isoforms, then simply reads per million will do:

\[RPM_i=10^6\cdot\frac{n_i}{\sum_j n_j}.\]

TPM

Here we consider RNA-Seq data from two different tissues. For simplicity, let’s make a (completely unrealistic) assumption that in each tissue, only two isoforms are expressed: red and yellow in tissue 1, and red and green in tissue 2. We are interested in the difference in expression of the red isoform in the two tissues.

The number of reads for the red isoform is the same in both cases. Since we are comparing the expression of the same isoform, the $RPK$ values will be identical too. Furthermore, the total number of reads is the same between the experiments, so even the $RPKM$ values won’t show any difference. But is there one?

Imagine for a second that there is an equal number of red and yellow transcripts in the tissue 1 sample. Since the red isoform is longer than the yellow one, it yields more fragments, and so we should observe more red reads than yellow ones. Yet we see more yellow reads. This means that the number of red transcripts is significantly lower than the number of yellow transcripts.

If we conduct the same thought experiment with tissue 2, we would expect somewhat more green fragments than red ones, and this is what we observe. The relative abundance of the red isoform in tissue 2 is probably close to $50\%$ and thus is higher than in tissue 1.

The formula for $RPKM_i$ takes into account the length of isoform $i$ only. But the lengths of other isoforms clearly have an impact on the relative number of isoform $i$’s fragments.

The metric called $TPM$, or transcripts per million, directly measures the relative abundance of transcripts. To estimate it from the count data, notice that the $RPK$ values are proportional to the abundances of isoforms within a single experiment. Thus the abundance of isoform $i$ per million transcripts can be estimated as

\[TPM_i=10^6\cdot\frac{RPK_i}{\sum_j RPK_j}=10^6\cdot\frac{n_i/l_i}{\sum_j n_j/l_j}.\]

Relative vs. absolute expression

So far we have been estimating the relative abundance, i.e. what proportion of transcripts in a sample belong to a particular isoform. Can we estimate the absolute abundance from RNA-Seq data?

Consider the two cells drawn below. The colored squiggly lines represent individual transcripts of the corresponding isoforms.

Cell B has twice as many transcripts of each isoform as cell A. If we conduct RNA-Seq experiments in the two cells, we would get samples from essentially the same distribution. We wouldn’t be able to tell the cells apart based on their RNA-Seq.

Here’s a trick: during library preparation, we add a known amount of an artificial RNA or DNA that is not produced by the studied organism (the blue squiggle below), then we can compare all abundances against it. This artificially introduced material is called a spike-in.

If we regard the spike-in as isoform $0$, with the known absolute abundance of $T_0$ transcripts, then the absolute abundance of isoform $i$ can be estimated as

\[T_i = T_0\cdot\frac{RPK_i}{RPK_0}=T_0\cdot\frac{n_i/l_i}{n_0/l_0}.\]

Dangers of relative expression

Whether we care about absolute or relative expression depends on the biological question in hand. However, looking at relative expression alone can produce unexpected results.

Suppose again that only two isoforms are being expressed, red and yellow. In condition A, the two isoforms are equally expressed. In condition B, the yellow isoform’s expression doubles, while the red isoform’s expression is not affected at all.

Now let’s look at the relative expression:

Based on this chart, we might conclude that the red isoform is also differentially expressed between the two conditions. Technically this is true, as long as we are talking about relative expression, but this is only a consequence of the overexpression of the yellow isoform.

Make ~/.pam_environment work again

Tue, 08 Nov 2016 20:00:00 +0000

In the early days of Linux, when we used to start X with startx, your GUI environment was directly inherited from your console environment. So if you wanted to set or change an environment variable, you’d just put it in ~/.profile.

Nowadays we use display managers such as gdm, and the recommended way to set environment variables is in ~/.pam_environment, which is read by the pam_env module. The syntax of that file is

PATH OVERRIDE=/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin
LANG OVERRIDE=en_US.UTF-8

After a recent update I noticed that even this stopped working. Turns out that this feature was marked as a security issue and disabled by default.

On a typical laptop or personal computer, where there are no hostile local users, you can enable it back.

Check whether your PAM configuration is managed by authselect

On newer systems (at least the RedHat-derived ones; not sure about the others) authselect is used to manage the configuration files in /etc/pam.d. To check if your system uses authselect, run

authselect check

On a system that uses authselect, you will get something like

Current configuration is valid.

On a system that doesn’t use authselect, you’ll get either an error message indicating that there’s no authselect installed (e.g. authselect: No such file or directory) or some kind of message from authselect telling you that it’s not being used.

If authselect is used

Create a custom authselect profile:

authselect create-profile minimal_with_env -b minimal --symlink-meta --symlink-pam

Edit the file /etc/authselect/custom/minimal_with_env/password-auth. Find the line that says
```
auth       required    pam_env.so
```
and change it to
```
auth       required    pam_env.so user_readenv=1
```

Switch to the new profile:

authselect select custom/minimal_with_env

If authselect is not used

Find the file in /etc/pam.d that corresponds to your display manager, such as /etc/pam.d/lightdm for lightdm. If you are logging in via the tty directly, skipping a display manager, see /etc/pam.d/login, but note that e.g. on Fedora 31 the setting is delegated to /etc/pam.d/system-auth, which is then sourced.

Find the line in that file that says

auth       required    pam_env.so

and change it to

auth       required    pam_env.so user_readenv=1

Restart your display manager.

You may also need to disable SELinux; otherwise you may be seeing errors like these in journalctl:

Dec 05 10:47:31 jerry audit[69035]: AVC avc:  denied  { read } for  pid=69035 comm="login" name=".pam_environment" dev="dm-2" ino=12063987 scontext=system_u:system_r:local_login_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1
Dec 05 10:47:31 jerry audit[69035]: AVC avc:  denied  { open } for  pid=69035 comm="login" path="/home/roman/.pam_environment" dev="dm-2" ino=12063987 scontext=system_u:system_r:local_login_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1

Electoral vote distributions are polynomials

Fri, 28 Oct 2016 20:00:00 +0000

In his article Electoral vote distributions are Monoids, Gabriel Gonzalez poses and answers the following question based on 538’s data:

what would be Hillary’s chance of winning if each state’s probability of winning was truly independent of one another?

To answer the question, Gabriel devises a divide-and-conquer algorithm. He computes probability distributions over vote counts in subsets of all states and then combines them. He also observes that vote distributions form a monoid.

Here I want to share an algebraic perspective on vote counting and show why distributions form a monoid.

Let $p_i$ be the probability of Hillary’s victory in state $i$, and $n_i$ be the number of electoral college votes for that state, where $i=1,\ldots,N$, and $N$ is the total number of states (and districts; see Gabriel’s post for details).

Then a vote distribution is a collection of probabilities $q_k$ that Hillary will get exactly $k$ votes:

\[ \newcommand{\p}[1]{\mathrm{Pr}\{#1\}} \begin{equation} q_k = \p{\text{number of votes for H.Clinton} = k},\;k=1,\ldots,\sum_{i=1}^N n_i. \end{equation} \]

Consider the following polynomial:

\[Q(x)=\prod_{i=1}^N\left(p_i x^{n_i}+(1-p_i)\right).\]

This is a product of $N$ brackets, one for each state. If we expanded it, we would get $2^N$ terms. Every such term takes either $p_i x^{n_i}$ or $1-p_i$ from each bracket and multiplies them. Every such term corresponds to a particular election outcome: if Hillary won in a particular state, take $p_i x^{n_i}$ from the corresponding bracket; otherwise, take $1-p_i$.

For example, if an election outcome means that Hillary won in states $1,4,\ldots$ and lost in states $2,3,\ldots$, then the corresponding term is

\[ p_1 x^{n_1}(1-p_2)(1-p_3)p_4 x^{n_4}\ldots=p_1(1-p_2)(1-p_3)p_4\ldots x^{n_1+n_4+\ldots}. \]

Notice that $p_1(1-p_2)(1-p_3)p_4\ldots$ is exactly the probability of the outcome (under the independence assumption) and $n_1+n_4+\ldots$ is the number of votes for Hillary under that outcome.

Since the power of $x$ in each term is the number of votes for Hillary, outcomes that result in the same number of votes, say $k$, correspond to like terms. If we combine them, their probabilities (terms’ coefficients) will add up. To what? To $q_k$, the total probability of Hillary getting $k$ votes.

Therefore,

\[Q(x) = \sum_{k}q_kx^k.\]

Deriving the final vote distribution $q_k$ from $p_i$ and $n_i$ is just expanding and reducing $Q(x)$ from $\prod_{i=1}^N\left(p_i x^{n_i}+(1-p_i)\right)$ to $\sum_{k}q_kx^k$.

As Gabriel notes, doing this in the direct way would be inefficient. His divide-and-conquer approach directly translates to expanding $Q(x)$: divide all brackets into two groups, recursively expand the groups, combine the results.

Under this formulation, it becomes obvious that vote distributions form a proper monoid: it is just a monoid of polynomials under multiplication.

Mean-variance ceiling

Thu, 20 Oct 2016 20:00:00 +0000

Today I was playing with the count data from a small RNA-Seq experiment performed in Arabidopsis thaliana.

At some point, I decided to look at the mean-variance relationship for the fragment counts. As I said, the dataset is small; there are only 3 replicates per condition from which to estimate the variance. Moreover, each sample is from a different batch. I wasn’t expecting to see much.

But there was a pattern in the mean-variance plot that was impossible to miss.

It is a nice straight line that many points lie on, but none dare to cross. A ceiling.

The ceiling looked mysterious at first, but then I found a simple explanation. The sample variance of $n$ numbers $a_1,\ldots,a_n$ can be written as

\[\sigma^2=\frac{n}{n-1}\left(\frac1n\sum_{i=1}^n a_i^2-\mu^2\right),\]

where $\mu$ is the sample mean. Thus,

\[\frac{\sigma^2}{\mu^2}=\frac{\sum a_i^2}{(n-1)\mu^2}-\frac{n}{n-1}.\]

For non-negative numbers, $n^2\mu^2=(\sum a_i)^2\geq \sum a_i^2$, and

\[\frac{\sigma^2}{\mu^2}\leq\frac{n^2}{n-1}-\frac{n}{n-1}=n.\]

This means that on a log-log plot, all points $(\mu,\sigma^2)$ lie on or below the line $y=2x+\log n$.

Moreover, the points that lie exactly on the line correspond to the samples where all $a_i$ but one are zero. In other words, those are gene-condition combinations where the gene’s transcripts were registered in a single replicate for that condition.

The rule of 17 in volleyball

Wed, 19 Oct 2016 20:00:00 +0000

Scott Adams, the author of Dilbert, writes in his book “How to Fail at Almost Everything and Still Win Big”:

Recently I noticed that the high-school volleyball games I attended in my role as stepdad were almost always won by the team that reached seventeen first, even though the winning score is twenty-five and you have to win by two.

It’s common for the lead to change often during a volleyball match, and the team that first reaches seventeen might fall behind a few more times before winning, which makes the pattern extra strange.

Good observation, Scott! But why could it be so?

Scott offers two possible explanations. One is psychological: the leading team has a higher morale while the losing team feels defeated. The other is that perhaps the coach of the losing team sees this as an opportunity to let his bench players on the court.

While these reasons sound plausible to me, there is a simpler logical explanation. It would hold even if the players and coaches were robots.

Imagine that you enter a gym where a game is being played. You see the current score: 15:17. If you know nothing else about the teams except their current score, which one do you think is more likely to win the set?

There are two reasons to think it is the leading team:

The score by itself doesn’t offer much evidence that the leading team is stronger or in a better shape. However, if one of the teams is stronger, it is more likely to be the leading team.
Even without assuming anything about how good the teams are, the leading team at this moment is up for an easier task: it needs only 8 points to win, whereas the team behind needs 10 points.

To quantify the reliability of Scott Adams’s “rule of 17”, I wrote a simple simulation in R:

sim.one <- function(prob, threshold) {
  score <- c(0,0)
  leader <- NA
  serving <- 1
  while (all(score < 25) || abs(diff(score)) < 2) {
    winner <-
      if (as.logical(rbinom(1,1,prob[[serving]])))
        serving
      else
        3 - serving
    score[[winner]] <- score[[winner]] + 1
    serving <- winner
    if (is.na(leader) && any(score == threshold)) {
      leader <- which.max(score)
    }
  }
  return(c(leader, which.max(score)))
}

Here prob is a 2-dimensional vector $(p_1,p_2)$, where $p_i$ is the probability of team $i$ to win their serve against the opposing team. The function simulates a single set and returns two numbers: which team first scored threshold (e.g. 17) points and which team eventually won. If the two numbers are equal, the rule worked in this game.

Then I simulated a game 1000 times for each of many combinations of $p_1$ and $p_2$ and calculated the fraction of the games where the rule worked. Here’s the result:

When $p_1=p_2$, the reliability of the rule is independent of the values of $p_1$ and $p_2$ (within the tested limits of $0.3$ and $0.7$) and equals approximately $81\%$. This is entirely due to reason 2: all else being equal, the leading team has a head start.

When teams are unequal, reason 1 kicks in, and for large inequalities, the reliability of the rule approaches $1$. For instance, when $p_1=0.3$ and $p_2=0.7$, the rule works about $99\%$ of the time.

Is there anything magical about the number 17? No, we would expect the rule to work for any threshold at least to some extent. The reliability would grow from somewhere around $50\%$ for the threshold of $1$ to almost $100\%$ for the threshold of $25$.

And indeed, this is what we observe (for $p_1=p_2$):

This reminds me of men’s gold medal match at the 2012 London Olympics, where Russia played against Brazil. Russia loses the first two sets. A game lasts until one of the teams wins 3 sets in total, so Russia cannot afford to lose a single set now. In the third set, Brazil continues to lead, reaching 17 (and then 18) points while Russia has 15. Several minutes later, Brazil leads 22:19.

And then, against all odds, the Russian team wins that set 29:27, then the two following sets, and gets the gold.

How to prepare a good pull request

Sun, 18 Sep 2016 20:00:00 +0000

A pull request should have a specific goal and have a descriptive title. Do not put multiple unrelated changes in a single pull request.
Do not include any changes that are irrelevant to the goal of the pull request.

This includes refactoring or reformatting unrelated code and changing or adding auxiliary files (.gitignore, .travis.yml etc.) in a way that is not related to your main changes.
Make logical, not historical commits.

Before you submit your work for review, you should rebase your branch (git rebase -i) and regroup your changes into logical commits.

Logical commits achieve different parts of the pull request goal. Each commit should have a descriptive commit message. Logical commits within a single pull request rarely overlap in the lines of code they touch.

If you want to amend your pull request, rewrite the branch and force-push it instead of adding new (historical) commits or creating a new pull request.
Make clean commits. Run git diff --check origin/master, and if it reports any issues like trailing whitespace or missing newlines at the end of the file, fix them.

.gitignore

My .gitignore policy is that the project-specific .gitignore file should only contain patterns specific for this project. For instance, if a test suite generates files *.out, this pattern belongs to the project’s .gitignore.

If a pattern is standard across a wide range of projects (e.g. *.o, or .stack-work for Haskell projects), then it belongs to the user-specific ~/.gitignore.

stack.yaml

(This section is specific to Haskell.)

My policy is to track stack.yaml inside the repo for applications, but not for libraries.

The rationale is that for an application, stack.yaml provides a useful bit of metainformation: which snapshot the app is guaranteed to build with. Additionally, non-programmers (or non-Haskell programmers) may want to install the application, and the presence of stack.yaml makes it easy for them.

These benefits do not apply to libraries. And the cost of including stack.yaml is:

The snapshot version gets out of date quickly, so you need to update this file regularly.
This file is often changed temporarily (e.g. to test a specific version of a dependency), and if it is tracked, you need to pay attention not to commit those changes by accident.

UPDATE (2017-06-01): I now include stack.yaml in some of my library repos. The reason is that it is needed for stack’s recommended travis configuration.

A case for static linking in scientific computing

Fri, 09 Sep 2016 20:00:00 +0000

When researchers run scientific software on high-performance clusters, they often experience problems with shared libraries, such as this one:

bcftools: /lib64/libz.so.1: version `ZLIB_1.2.5.2' not found

Or this one:

eagle: error while loading shared libraries: libhts.so.1: cannot open shared object file: No such file or directory

Popular advice points them in the direction of LD_LIBRARY_PATH, but a simple and robust solution—static linking—is often overlooked.

In this article, I explain the background behind static and dynamic linking, demonstrate the advantages of static linking, address some of the objections against static linking, and give instructions on how to prepare static binaries.

What is static and dynamic linking?

The word linking itself refers to the process of assembling a complete program from libraries of subprograms¹.

Static linking occurs as the last stage of the compilation process; the required libraries are embedded in the final binary file of your program.

Some (or even all) of the libraries may not be included in the binary at this stage. In this case, when we attempt to run the program, we need dynamic linking in order to find the libraries and make the missing subroutines accessible to the program. This second type of libraries is called dynamic, or shared, libraries. The files with these libraries are usually named libsomething.so on Linux and something.dll on Windows.

The rules that an operating system follows when it searches for dynamic libraries are complex. And simply having a library in the right place is not enough; it needs to be the same version of the library that was used during the compilation, or at least a different version with the same ABI. (So no, you shouldn’t ln -s /usr/lib/libsomething.so.2 /usr/lib/libsomething.so.1 when a program doesn’t work, contrary to another popular piece of advice one can find on the Internet.)

Linux distributions, most of which dynamically link the software they distribute, manage this by engaging qualified package maintainers and by having tools and centralized infrastructure to build and distribute packages.

But the world of scientific software is not there yet. And if you fail to take care of your dynamic libraries, the result can vary from a program refusing to start (as we saw earlier), to a program crash in the middle of operation, to a hard to detect and diagnose case of data corruption.

This is why I think that scientific software should be linked statically by default.

Advantages of static linking

Reproducibility. When a program is linked statically, it executes the same algorithm wherever it is run. A dynamic executable executes the code from the version of the dynamic library that happens to be installed on a particular computing node.

Note that the static executable doesn’t contain the metainformation about the library versions used to build it. You should record that information when you compile the software, and ideally use a binary repository manager for your builds.

But replacing dynamic linking with static linking by itself dramatically increases the probability that your program will run tomorrow in the same way as it runs today.

Ease of distribution. Suppose that you want your colleague to run the program you have compiled. If you link statically, you only need to distribute a single binary. If you link dynamically, you need to distribute the binary, all dynamic libraries that it depends on, and the instructions on how to make the binary find its libraries.

Portability. Static linking ensures that you can compile the program on your Ubuntu laptop and run it on a Debian or CentOS cluster. With dynamic linking, you’d have to compile it directly on a cluster or in an identical environment.

That said, you still need to ensure that both systems use the same or compatible architectures (the majority of scientific computing happens on x86-64 anyway), the same OS (probably Linux) and not too different kernel versions (or libc versions if you link libc dynamically).

No problems with finding libraries. Since no dynamic libraries are needed, the OS cannot fail to find them. No more cannot open shared object file messages. No more LD_LIBRARY_PATH tricks.

Isn’t static linking considered harmful?

Ulrich Drepper says it is:

There are still too many people out there who think (or even insist) that static linking has benefits. This has never been the case and never will be the case.

Ulrich certainly knows about this stuff much more than I ever hope to. But as you can tell from the above quote, he is sometimes a bit extreme in his judgment.

There is no shortage of knowledgeable people who disagree with him on this issue.

But more importantly, he looks at linking from a very different perspective. For many years, he was employed by Red Hat. He was one of those people who knew a lot about dealing with dynamic libraries and maintained a centralized repository of packages that worked well together in a controlled environment.

It is understandable that he would not care about any of the advantages I list above (though this is different from claiming that there has never been and never will be any benefits to static linking).

But what about the advantages of the dynamic linking that Ulrich describes in his article?

Centralized bug/security fixes.

Security issues matter less for scientific software because it is not exposed to the outside world.
HPC cluster users don’t benefit from centralized bug fixes because usually they don’t have the permissions to install software system-wide. Every user of the same cluster or node would still be responsible for their own updates.
The scale is very different. If you are Red Hat, re-linking hundreds or thousands of binaries every time there is an update in a library is a significant burden. If you are a researcher, you deal maybe with a dozen or two programs, and you may not have to update them often.
Even when centralized updates are possible (e.g. if you can request libraries to be installed centrally and then link against them), scientists would not want them because they are directly at odds with reproducibility.

More efficient use of physical memory through sharing the code.

In high-performance computing, the size of the libraries is usually negligible compared to the size of the data being processed.
When the number of running processes is small, and they don’t have many common dependencies, there’s not much opportunity for sharing.
On the other hand, sometimes multiple copies of the same executable are run in parallel. This happens with software that is not capable of multithreading or cannot exploit it efficiently. Well, in this case, the OS actually can share the code across the processed because it is exactly the same.
When there’s little sharing of code between processes, static linking can sometimes be more memory-efficient. This is because static linking only embeds the object files (i.e. parts of a library) that are actually used by the application, whereas dynamic linking has to load the entire library into memory.

Security measures like load address randomization—see above.

Some features of glibc require dynamic linking. Ulrich, by the way, was one of the core developers of glibc—just in case you were wondering why he considers this a problem of static linking and not a problem of glibc.

Fortunately, most scientific software doesn’t perform character conversions or go to the network. It just crunches numbers. You don’t need dynamic linking for that.

Licensing considerations. I am not a lawyer, but as far as I can tell, this should concern you only if the software is closed-source (or distributed under a license incompatible with GPL) and some of those dependencies are licensed under LGPL. In that case, those dependencies must be linked dynamically, although the other ones can still be linked statically.

I am not sure why Ulrich writes “(L)GPL”, since, to my knowledge, GPL itself does not make a distinction between static and dynamic linking, but I am happy to be corrected.

Tools and hacks like ltrace, LD_PRELOAD, LD_PROFILE, LD_AUDIT don’t work. Oh well.

OK, how do I do this?

Unfortunately, most of the scientific software I come across is linked dynamically by default. Otherwise, I wouldn’t be writing this article.

Convincing the build system

Read the installation instructions. They usually can be found in a file named README, INSTALL, or on the website. If they mention static linking, congratulations.

If not, also try looking inside the Makefile (or whatever build system the software uses). If there is a configure script, try ./configure --help. There could be a target for static linking that the author has not documented.

If the build system doesn’t support static linking out of the box, you will have to modify the linker flags. If a Makefile is well-written, this should be as simple as LDFLAGS=-static make or make LDFLAGS=-static (these are different; try them in this order).

If that doesn’t work, edit the Makefile. Locate the rule that does linking, i.e. produces the final executable.

It usually looks like this:

program: $(OBJS)
    gcc -o program $(OBJS)

Note that gcc could also be g++, or ld, or hidden behind a variable such as $(CC), $(CXX), or $(LD). The variable $(OBJS) could also be named differently, or be a literal list of .o, .c, or .cpp files. But the program is usually exactly the name of the final program (or, again, a variable that expands to one).

Once you located this rule, try adding a -static flag to the gcc command line:

program: $(OBJS)
    gcc -static -o program $(OBJS)

In many cases it will be enough to perform static linking.

Sometimes you need to get more creative. One tool, for instance, explicitly built itself as a shared library. This is incompatible with a (global) -static flag set as part of $(LDFLAGS). The way I solved this was to specify a target explicitly, i.e. make prog1 prog2, so that it wouldn’t attempt to build a dynamic library and fail.

Dependencies

In order to statically link a program, you need to have its dependencies available as static libraries. These are files names as libsomething.a.

If this is a library available in your distribution:

For Debian and derived distros (e.g. Ubuntu), static libraries usually reside in the libsomething-dev package. If you’ve done dynamic linking of the program, you probably have this package installed already because it also contains the header files.
For Red Hat and derived distributions (Fedora, CentOS; not sure about other rpm-based distros), static libraries are often placed in separate packages named libsomething-static.

If this is a third-party library, you’ll need to get a static version of it or compile it from source, following the same instructions that you are reading right now.

Verifying the result

How do you check that you got a static binary? Try running file and ldd on it.

For a dynamic binary, you’ll get something like this:

% ldd ./eagle 
    linux-vdso.so.1 (0x00007ffd47d87000)
    libhts.so.1 => not found
    libboost_program_options.so.1.49.0 => not found
    libboost_iostreams.so.1.49.0 => not found
    libz.so.1 => /lib64/libz.so.1 (0x00007fe77a445000)
    libopenblas.so.0 => /lib64/libopenblas.so.0 (0x00007fe778133000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe777f17000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fe777b90000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fe777886000)
    libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fe777658000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe777441000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe77707e000)
    libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007fe776d4d000)
    /lib64/ld-linux-x86-64.so.2 (0x000055f7b3885000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fe776b49000)
    libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007fe776908000)

% file ./eagle
./eagle: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.26, BuildID[sha1]=af18461c835d6f0209754b78c639581c67ed1443, stripped

For a static binary, you’ll see this instead:

% ldd ./Minimac3
    not a dynamic executable

% file ./Minimac3
./Minimac3: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.26, BuildID[sha1]=edf443bb3e695b3f421b0d162ca30bb0f422c2ad, stripped

By the way, if file says that your binary is “not stripped”, run the strip command on it:

% strip eagle

This will significantly reduce its on-disk size and make it faster to copy to the cluster.

To avoid confusion, I should note that I am talking here about software written in compiled languages such as C, C++, or Fortran. I am not talking about interpreted languages (Perl, Python, Ruby) or bytecode-compiled languages (Java, Scala).↩︎

Extract the first n sequences from a FASTA file

Tue, 23 Aug 2016 20:00:00 +0000

A FASTA file consists of a series of biological sequences (DNA, RNA, or protein). It looks like this:

>gi|173695|gb|M59083.1|AETRR16S Acetomaculum ruminis 16S ribosomal RNA
NNTAAACAAGAGAGTTCGATCCTGGCTCAGGATNAACGCTGGCGGCATGCCTAACACATGCAAGTCGAAC
GGAGTGCTTGTAGAAGCTTTTTCGGAAGTGGAAATAAGTTACTTAGTGGCGGACGGGTGAGTAACGCGTG

>gi|310975154|ref|NR_037018.1| Acidaminococcus fermentans strain VR4 16S ribosomal RNA gene, partial sequence
GGCTCAGGACGAACGCTGGCGGCGTGCTTAACACATGCAAGTCGAACGGAGAACTTTCTTCGGAATGTTC
TTAGTGGCGAACGGGTGAGTAACGCGTAGGCAACCTNCCCCTCTGTTGGGGACAACATTCCGAAAGGGAT

There probably exist dozens of python scripts to extract the first $n$ sequences from a FASTA file. Here I will show an awk one-liner that performs this task, and explain how it works.

Here it is (assuming the number of sequences is stored in the environment variable NSEQS):

awk "/^>/ {n++} n>$NSEQS {exit} {print}"

This one-liner can read from standard input (e.g. as part of a pipe), or you can append one or more file names to the end of the command, e.g.

awk "/^>/ {n++} n>$NSEQS {exit} {print}" file.fasta

An awk script consists of one or more statements of the form pattern { actions }. The input is read line-by-line, and if the current line matches the pattern, the corresponding actions are executed.

Our script consists of 3 statements:

/^>/ {n++} increments the counter each time a new sequence is started. /.../ denotes a regular expression pattern, and ^> is a regular expression that matches the > sign at the beginning of a line.

An uninitialized variable in awk has the value 0, which is exactly what we want here. If we needed some other initial value (say, 1), we could have added a BEGIN pattern like this: BEGIN {n=1}.
n>$NSEQS {exit} aborts processing once the counter reaches the desired number of sequences.
{print} is an action without a pattern (and thus matching every line), which prints every line of the input until the script is aborted by exit.

A shorter and more cryptic way to write the same is

awk "/^>/ {n++} n>$NSEQS {exit} 1"

Here I replaced the action-without-pattern by a pattern-without-action. The pattern 1 (meaning “true”) matches every line, and when the action is omitted, it is assumed to be {print}.

Docker configuration on Fedora

Thu, 18 Aug 2016 20:00:00 +0000

If you need to change the docker daemon options on Fedora, take a look at these files:

# ls /etc/sysconfig/docker*
/etc/sysconfig/docker
/etc/sysconfig/docker-network
/etc/sysconfig/docker-storage
/etc/sysconfig/docker-storage-setup

In my case, I needed to change the container base size, so I put the following in /etc/sysconfig/docker-storage:

DOCKER_STORAGE_OPTIONS="--storage-opt dm.basesize=20G"

These files are then sourced in /etc/systemd/system/multi-user.target.wants/docker.service, and the variables (such as DOCKER_STORAGE_OPTIONS) are passed to the docker daemon.

Does it matter if Hask is (not) a category?

Sun, 07 Aug 2016 20:00:00 +0000

Andrej Bauer raises a question whether Hask is a real category. I think it’s a legitimate question to ask, especially by a mathematician or programming languages researcher. But I want to look closer at how a (probably negative) answer to this question would affect Haskell and its community.

To illustrate the fallacy of assuming blindly that Hask is a category, Andrej tells an anecdote (which I find very funny):

I recall a story from one of my math professors: when she was still a doctoral student she participated as “math support” in the construction of a small experimental nuclear reactor in Slovenia. One of the physicsts asked her to estimate the value of the harmonic series $1+1/2+1/3+\cdots$ to four decimals. When she tried to explain the series diverged, he said “that’s ok, let’s just pretend it converges”.

Presumably here is what happened:

The physicists came up with a mathematical model of a nuclear reactor.
The model involved the sum of the harmonic series.
Andrej’s math professor tried to explain that the series diverged and therefore something was wrong with the model.

When we try to model a phenomenon, we should watch out for two types of problems:

The model itself is erroneous.
The model itself is fine; but the phenomenon we are describing does not meet all of the model’s assumptions.

The first type of problem means that the people who built the model couldn’t get their math right. That’s too bad. We let mathematicians to gloss over the messy real world, to impose whatever assumptions they want, but in return we expect a mathematically rigorous model upon which we can build. In Andrej’s story, hopefully the math support lady helped the physicists build a better model that didn’t rely on the convergence of the harmonic series.

But at some point the model has to meet the real world; and here, the issues are all but inevitable. We know that all models are wrong (meaning that they don’t describe the phenomenon ideally, not that they are erroneous) — but some are useful.

Physicists, for example, often assume that they are dealing with isolated systems, while being perfectly aware that no such system exists (except, perhaps, for the whole universe, which would be impossible to model accurately). Fortunately, they still manage to design working and safe nuclear reactors!

Consider Hask. Here, the abstraction is the notion of a category, and the phenomenon is the programming language Haskell. If types and functions of Haskell do not form a proper category, we have the second type of modelling problem. The foundation — the category theory — is, to the best of my knowledge, widely accepted among mathematicians as a solid theory.

Since category theory is often used to model other purely mathematical objects, such as groups or vector spaces, mathematicians may get used to a perfect match between the abstraction and the phenomenon being described. Other scientists (including computer scientists!) can rarely afford such luxury.

Usefulness is the ultimate criterion by which we should judge a model. We use monads in Haskell not because they are a cool CT concept, but because we tried them and found that they solve many practical problems. Comonads, which from the CT standpoint are “just” the dual of monads, have found much fewer applications, not because we found some kind of theoretical problems with them — we simply didn’t find that many problems that they help address. (To be fair, we tried hard, and we did manage to find a few.)

There are people who, inspired by some category theory constructions, come up with novel algorithms, data structures, or abstractions for Haskell. For these discoveries to work, it is neither necessary nor sufficient that they correspond perfectly to the original categorical abstractions they were derived from. And as long as playing with the “Hask category” yields helpful intuition and working programming ideas, we are going to embrace it.

Debugging a CUPS Forbidden error

Fri, 08 Jul 2016 20:00:00 +0000

When I try to install a printer on a fresh Fedora 24 system through the CUPS web interface (http://localhost:631/admin/), I get

Add Printer Error

Unable to add printer:

    Forbidden

Here’s the relevant part of the config file, /etc/cups/cupsd.conf:

<Limit CUPS-Add-Modify-Printer CUPS-Delete-Printer CUPS-Add-Modify-Class CUPS-Delete-Class CUPS-Set-Default CUPS-Get-Devices>
  AuthType Default
  Require user @SYSTEM
  Order deny,allow
</Limit>

Now look at man cupsd.conf for the explanation:

Require user {user-name|@group-name} ...
     Specifies  that  an authenticated user must match one of the named
     users or be a member of one of the named groups.  The  group  name
     "@SYSTEM" corresponds to the list of groups defined by the System‐
     Group directive in the cups-files.conf(5) file.   The  group  name
     "@OWNER" corresponds to the owner of the resource, for example the
     person that submitted a print job.

Let’s look at /etc/cups/cups-files.conf and find out what those groups are:

SystemGroup sys root

Alright, so the solution is:

sudo usermod -a -G sys feuerbach

(where feuerbach is my username).

Frankly, I think it’s a bug that an admin user cannot add a printer by default, but luckily it’s not hard to fix.

Install Fedora Linux on an encrypted SSD

Tue, 28 Jun 2016 20:00:00 +0000

I just replaced the SSD in my laptop with a bigger one and installed a fresh Fedora Linux on it, essentially upgrading from F23 to F24.

Here are a few notes which could be useful to others and myself in the future.

Verifying the downloaded image

How do you verify the downloaded image? You verify the checksum.

How do you verify the checksum? You check its gpg signature.

How do you verify the authenticity of the gpg key? You could just check the fingerprint against the one published on the website above, but this is hardly better than trusting the checksum, since they both come from the same source.

Here’s a better idea: if you already have a Fedora system, you have the keys at /etc/pki/rpm-gpg.

In my case, I imported /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-primary (yes, my F23 system already contained the F24 signing keys), and was able to check the checksum signature.

This protects you against a scenario when getfedora.org is compromised and the checksums/signatures/keys are replaced there.

Installing from a USB partition

Turned out the only optical disc in my house was damaged, and I didn’t have a USB stick big enough to burn the Fedora image either.

I did have an external USB drive with some free space on it, but it contained a lot of data, so I couldn’t just make it one big ISO partition.

There are several instructions on how to create bootable USB partitions, but most of them look fragile and complicated.

Luckily, Fedora makes this super easy.

Install the RPM package livecd-tools (which is a packaged version of this repo)
Create a partition big enough for the ISO and format it. Unlike many other instructions that tell you to use FAT, this one works with ext[234] just fine.
livecd-iso-to-disk Fedora-Workstation-Live-x86_64-24-1.2.iso /dev/sdb1

Setting up disk encryption

I was impressed by how easy it was to set up full disk encryption. I just checked the box “Encrypt my data” in the installer, and it used a very sensible partitioning scheme close to what I used to set up manually before:

Unencrypted /boot partition
Encrypted partition with LVM on top of it
- Three logical volumes on the encrypted LVM: root, /home, and swap.

The only thing that I had to do was to enable TRIM support:

For LVM: set issue_discards = 1 in /etc/lvm/lvm.conf.
For cryptsetup: change none to discard in /etc/crypttab.
Enable weekly trims systemctl enable fstrim.timer && systemctl start fstrim.timer

Predicting a coin toss

Tue, 14 Jun 2016 20:00:00 +0000

I flip a coin and it comes up heads. What is the probability it will come up heads the next time I flip it?

“Fifty percent,” you say. “The coin tosses are independent events; the coin doesn’t have a memory.”

Now I flip a coin ten times, and ten times in a row it comes up heads. Same question: what is the probability it will come up heads the next time?

You pause for a second, if only because you are not used to seeing a coin land heads ten times in a row.

But, after some hesitation, you convince yourself that this is no different from the first experiment. The coin still has got no memory, and the chances are still 50-50.

Or you become suspicious that something is not right with the coin. Maybe it is biased, or maybe it has two heads and no tails. In that case, your answer may be something like 95% for heads, where the remaining 5% account for the chance that the coin is only somewhat biased and tails are still possible.

This sounds paradoxical: coin tosses are independent, yet the past outcomes influence the probability of the future ones. We can explain this by switching from frequentist to Bayesian statistics. Bayesian statistics lets us model the coin bias (the probability of getting a single outcome of heads) itself as a random variable, which we shall call $\theta$. It is random simply because we don’t know its true value, and not because it varies from one experiment to another. Consequently, we will update its probability distribution after every experiment because we gain more information, not because it affects the coin itself.

Let $X_i\in\{H,T\}$ be the outcome of $i$th toss. If we know $\theta$, we automatically know the distribution of $X_i$:

\[p(X_i=H|\theta)=\theta.\]

As before, the coin has no memory, so for any given $\theta$, the tosses are independent: $p(X_i \wedge X_j|\theta)=p(X_i|\theta)p(X_j|\theta)$. But they are independent only when conditioned on $\theta$, and that resolves the paradox. If we don’t assume that we know $\theta$, then $X$s are dependent, because the earlier observations affect what we know about $\theta$, and $\theta$ affects the probability of the future observations.

A model with conditionally independent variables is called a Bayesian network or probabilistic graphical model, and it can be represented by a directed graph as shown on the right. The arrows point from causes to effects, and the absence of an edge indicates conditional independence.

Based on our evidence of 10 heads in a row, Bayes’ theorem lets us estimate the distribution of $\theta$. All we need is a prior distribution – what did we think about the coin before we tossed it?

For coin tossing and other binary problems, it is customary to take the Beta distribution as the prior distribution, as it makes the calculations very easy. Often such choice is justified, but in our case it would be a terrible one. Almost all coins we encounter in our lives are fair. To make the beta distribution centered at $\theta=0.5$ and low variance, we would need to set its parameters, $\alpha$ and $\beta$, to large equal numbers. The resulting distribution would assign non-trivial probability only to low deviations from $\theta=0.5$, and the distribution would be barely affected by our striking evidence.

Instead, let’s engineer our prior distribution from scratch. Double-sided coins may be rare in everyday life, but they are easy to buy on eBay. When someone approaches us out of the blue and starts flipping coins, there’s a fair chance they’ve got one of those. Still, we believe in humanity, so let’s assign a point probability of just $1\%$ to $\theta=0$ and $\theta=1$. What about biased coins, such as coins with $\theta=0.83$? Turns out they are unlikely to exist. Nevertheless, the Bayesian statistics teaches to be reluctant to assign zero probabilities to events, since then no amount of evidence can prove us wrong. So let’s take $0.1\%$ and spread it uniformly across the interval $[0;1]$. The remaining $97.9\%$ will be the probability of a fair coin.

Formally, our prior distribution over $\theta$ can be specified by its probability density as

\[ p(\theta)=0.979\delta(\theta-0.5)+0.01\delta(\theta)+0.01\delta(\theta-1)+0.001, \]

where $\delta$ is the Dirac delta function used to specify point probabilities.

Let $D$ refer to the event that $X_i=H$, $i=1,2,\ldots,10$. Then $p(D|\theta)=\theta^{10}$. By Bayes’ theorem,

\[ p(\theta|D)=\frac{p(D|\theta)p(\theta)}{\int_0^1 p(D|\theta)p(\theta)d\theta} = \frac{\theta^{10}p(\theta)}{\int_0^1 \theta^{10}p(\theta)d\theta}. \]

Now we need to do a bit of calculation by hand:

\[ \begin{multline} \int_0^1 \theta^{10}p(\theta)d\theta=0.979\cdot0.5^{10}+0.01\cdot 1^{10}+0.01 \cdot 0^{10} + 0.001\int_0^1 \theta^{10}d\theta \\ = 9.56\cdot 10^{-4} + 0.01 + 9.09\cdot 10^{-5}=0.0110; \end{multline} \] \[ p(\theta|D)=0.087\delta(\theta-0.5)+0.905\delta(\theta-1)+0.091\theta^{10}. \]

Thus, we are $90.5\%$ sure that the coin is double-headed, but we also allow $8.7\%$ for pure coincidence and $0.8\%$ for a biased coin.

Now back to our question: how likely is it that the next toss will produce heads?

\[ \begin{multline} p(X_{11}=H|D) = \int_0^1 p(X_{11}=H|D,\theta)p(\theta|D)d\theta = \int_0^1 \theta \, p(\theta|D)d\theta \\ = 0.087\cdot 0.5+0.905\cdot 1+0.091\cdot \int_0^1\theta^{11}d\theta = 0.956. \end{multline} \]

Very likely indeed. Notice, by the way, how we used the conditional independence above to replace $p(X_{11}=H|D,\theta)$ with $p(X_{11}=H|\theta)=\theta$.

Bayesian statistics is a powerful tool, but the prior matters. Before you reach for the conjugate prior, consider whether it actually represents your beliefs.

A couple of exercises:

How does our prior distribution change after a single coin toss (either heads or tails)?
How does our prior distribution change after ten heads and one tails?

Surprising reciprocity

Thu, 02 Jun 2016 20:00:00 +0000

$X$ and $Y$ are two correlated random variables with zero mean and equal variance. If the best way to predict $Y$ based on $X$ is $y = a x$, what is the best way to predict $X$ based on $Y$?

It is not $x = y / a$!

Let’s find the $a$ that minimizes the mean squared error $E[(Y-aX)^2]$:

\[E[(Y-aX)^2] = E[Y^2-2aXY+a^2X^2]=(1+a^2)\mathrm{Var}(X)-2a\mathrm{Cov}(X,Y);\]

\[\frac{\partial}{\partial a}E[(Y-aX)^2] = 2a\mathrm{Var}(X)-2\mathrm{Cov}(X,Y);\]

\[a=\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)}=\mathrm{Corr}(X,Y).\]

Notice that the answer, the (Pearson) correlation coefficient, is symmetric w.r.t. $X$ and $Y$. Thus it will be the same whether we want to predict $Y$ based on $X$ or $X$ based on $Y$!

How to make sense of this? It may help to consider a couple of special cases.

First, suppose that $X$ and $Y$ are perfectly correlated and you’re trying to predict $Y$ based on $X$. Since $X$ is such a good predictor, just use its value as it is ($a=1$).

Now, suppose that $X$ and $Y$ are uncorrelated. Knowing the value of $X$ doesn’t tell you anything about the value of $Y$ (as far as linear relationships go). The best predictor you have for $Y$ is its mean, $0$.

Finally, suppose that $X$ and $Y$ are somewhat correlated. The correlation coefficient is the degree to which we should trust the value of $X$ when predicting $Y$ versus sticking to $0$ as a conservative estimate.

This is the key idea — to think about $a$ in $y=ax$ not as a degree of proportionality, but as a degree of “trust”.

Added on 2020-08-21:

There was a related interesting discussion on twitter:

Which line do you think does a better job of predicting Y given X?
(Hint: This is one of those days where very basic statistics confuses the hell out of me.)
(1/n)

— Jay Hennig (@jehosafet) August 19, 2020

The answer is…black! Its rmse is 4 times lower than the red line. I keep staring at that plot and it still blows my mind. (2/n)
— Jay Hennig (@jehosafet) August 19, 2020

Meanwhile, the red line is better than the black line at predicting X given Y.
One explanation of the discrepancy is, predicting Y given X assumes that there is no measurement noise in X, while predicting X given Y assumes there is no measurement noise in Y.
(3/n)

— Jay Hennig (@jehosafet) August 19, 2020

The weird thing is, you are just as good at predicting X given Y as you are at predicting Y given X. So why does the red line look so much better than the black line?
(4/n)
— Jay Hennig (@jehosafet) August 19, 2020

By the way, both lines have zero mean and unit variance, so the slope of each line is equal to the pearson's correlation (ρ) between X and Y:
black: Y = ρX
red: X = ρY
(5/n)
— Jay Hennig (@jehosafet) August 19, 2020

When asked to draw a regression line, my experience is people draw something closer to the PCA line. https://t.co/978t8lcKrn
— Dean Eckles (@deaneckles) February 9, 2019

Basic HTTP auth with Scotty

Thu, 14 Apr 2016 20:00:00 +0000

Not so long ago, I needed to write a web app to automate the recording of our Haskell podcast, Bananas and Lenses.

To build it, I chose a lightweight Haskell web framework called Scotty. There is another lightweight Haskell web framework called Spock. Both start with the letter S and are characters from Star Trek, and I have little hope ever being able to tell which is which by name. I can say though that I enjoyed working with the one I happened to pick.

So, anyway, I needed to ensure that only my co-hosts and I could access the app. In such a simple scenario, basic HTTP auth is enough. I did a quick google search for “scotty basic auth”, but all I found was this gist in which the headers are extracted by hand. Ugh.

Indeed, at the time of writing, Scotty itself does not seem to provide any shortcuts for basic auth. And yet the solution is simple and beautiful; you just need to step back to see it. Scotty is based on WAI, the Haskell web application interface, and doesn’t attempt to hide that fact. On the contrary, it conveniently exposes the function

middleware :: Middleware -> ScottyM ()

which “registers” a WAI wrapper that runs on every request. And sure enough, WAI (wai-extra) provides an HttpAuth module.

To put everything together, here’s a minimal password-protected Scotty application (works with Stackage lts-5.1).

{-# LANGUAGE OverloadedStrings #-}
import Web.Scotty
import Network.Wai.Middleware.HttpAuth
import Data.SecureMem -- for constant-time comparison
import Lucid -- for HTML generation

password :: SecureMem
password = secureMemFromByteString "An7aLasi" -- https://xkcd.com/221/

main :: IO ()
main = scotty 8000 $ do
  middleware $ basicAuth (\u p -> return $ u == "user" && secureMemFromByteString p == password)
    "Bananas and lenses recording"

  get "/" . html . renderText $ do
    doctype_
    html_ $ do
      head_ $ do
        title_ "Bananas and lenses recording"

      body_ $ h1_ "Hello world!"

Two security-related points:

Data.SecureMem is used to perform constant-time comparison to avoid a timing attack.
Ideally, the whole thing should be run over https (as the password is submitted in clear), but this is outside of the scope of this article.

Descending sort in Haskell

Sat, 02 Apr 2016 20:00:00 +0000

When confronted with a problem of sorting a list in descending order in Haskell, it is tempting to reach for a “lazy” solution reverse . sort.

An obvious issue with this is efficiency. While sorting in descending order should in theory be exactly as efficient as sorting in ascending order, the above solution requires an extra list traversal after the sorting itself is done.

This argument can be dismissed on the grounds that reverse’s run time, $\Theta(n)$, is, in general, less than sort’s run time, $O(n \log n)$, so it’s not a big deal. Additionally, one could argue that, unlike more complex solutions, this one is “obviously correct”.

As the rest of this article explains, neither of these claims holds universally.

Proper solutions

Here are the two ways to sort a list in descending order that I am aware of. Both require the more general sortBy function

sortBy :: (a -> a -> Ordering) -> [a] -> [a]

The first argument to sortBy is the comparison function. For each pair of arguments it returns a value of type

data Ordering = LT | EQ | GT

which describes the ordering of those arguments.

The “standard” ordering is given by the compare function from the Ord typeclass. Thus, sort is nothing more than

sort = sortBy compare

The first solution to the descending sort problem exploits the fact that, to get the opposite ordering, we can simply swap around the two arguments to compare:

sortDesc = sortBy (flip compare)

The second solution relies on the comparing function from Data.Ord, which gives a particular way to compare two values: map them to other values which are then compared using the standard Ord ordering.

comparing :: Ord a => (b -> a) -> b -> b -> Ordering

This trick is often used in mathematics: to maximize a function $x\mapsto f(x)$, it suffices to minimize the function $x \mapsto -f(x)$. In Haskell, we could write

sortDesc = sortBy (comparing negate)

and it would work most of the time. However,

> sortDesc [1,minBound::Int]
[-9223372036854775808,1]

Besides, negation only works on numbers; what if you want to sort a list of pairs of numbers?

Fortunately, Data.Ord defines a Down newtype which does exactly what we want: it reverses the ordering between values that it’s applied to.

> 1 < 2
True
> Down 1 < Down 2
False

Thus, the second way to sort the list in descending order is

sortDesc = sortBy (comparing Down)

sortOn

Christopher King points out that the last example may be simplified with the help of the sortOn function introduced in base 4.8 (GHC 7.10):

sortOn :: Ord b => (a -> b) -> [a] -> [a]

Thus, we can write

sortDesc = sortOn Down

Very elegant, but let’s look at how sortOn is implemented:

sortOn f =
  map snd .
  sortBy (comparing fst) .
  map (\x -> let y = f x in y `seq` (y, x))

This is somewhat more complicated than sortBy (comparing Down), and it does the extra work of first allocating $n$ cons cells and $n$ tuples, then allocating another $n$ cons cells for the final result.

Thus, we might expect that sortOn performs worse than sortBy. Let’s check our intuition:

import Criterion
import Criterion.Main
import Data.List
import Data.Ord

list :: [Int]
list = [1..10000]

main = defaultMain
  [ bench "sort"   $ nf sort (reverse list)
  , bench "sortBy" $ nf (sortBy (comparing Down)) list
  , bench "sortOn" $ nf (sortOn Down) list
  ]

(Why reverse?)

benchmarking sort
time                 134.9 μs   (134.3 μs .. 135.4 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 134.8 μs   (134.2 μs .. 135.7 μs)
std dev              2.677 μs   (1.762 μs .. 3.956 μs)
variance introduced by outliers: 14% (moderately inflated)

benchmarking sortBy
time                 131.0 μs   (130.6 μs .. 131.4 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 131.1 μs   (130.8 μs .. 131.4 μs)
std dev              965.1 ns   (766.5 ns .. 1.252 μs)

benchmarking sortOn
time                 940.5 μs   (928.6 μs .. 958.1 μs)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 950.6 μs   (940.9 μs .. 961.1 μs)
std dev              34.88 μs   (30.06 μs .. 44.19 μs)
variance introduced by outliers: 27% (moderately inflated)

As we see, sortOn is 7 times slower than sortBy in this example. I also included sort in this comparison to show that sortBy (comparing Down) has no runtime overhead.

There is a good reason why sortOn is implemented in that way. To quote the documentation:

sortOn f is equivalent to sortBy (comparing f), but has the performance advantage of only evaluating f once for each element in the input list. This is called the decorate-sort-undecorate paradigm, or Schwartzian transform.

Indeed, if f performed any non-trivial amount of work, it would be wise to cache its results—and that’s what sortOn does.

But Down is a newtype constructor—it performs literally no work at all—so the caching effort is wasted.

sortWith

Yuriy Syrovetskiy points out that there is also sortWith defined in GHC.Exts, which has the same type as sortOn but does no caching. So if you want to abbreviate sortBy (comparing Down), you can say sortWith Down—but you need to import GHC.Exts first.

Wonder why sortWith lives in GHC.Exts and not in Data.List or Data.Ord? It was originally added to aid writing SQL-like queries in Haskell, although I haven’t seen it used a single time in my career.

Asymptotics

Thanks to Haskell’s laziness in general and the careful implementation of sort in particular, sort can run in linear time when only a fixed number of first elements is requested.

So this function will return the 10 largest elements in $\Theta(n)$ time:

take 10 . sortBy (comparing Down)

While our “lazy” solution

take 10 . reverse . sort

(which, ironically, turns out not to be lazy enough—in the technical sense of the word “lazy”) will run in $O(n \log n)$ time. This is because it requests the last 10 elements of the sorted list, and in the process of doing so needs to traverse the whole sorted list.

This may appear paradoxical if considered outside of the context of lazy evaluation. Normally, if two linear steps are performed sequentially, the result is still linear. Here we see that adding a linear step upgrades the overall complexity to $O(n \log n)$.

Semantics

As I mentioned in the beginning, the simplicity of reverse . sort may be deceptive. The semantics of reverse . sort and sortBy (comparing Down) differ in a subtle way, and you probably want the semantics of sortBy (comparing Down).

This is because sort and sortBy are stable sorting functions. They preserve the relative ordering of “equal” elements within the list. Often this doesn’t matter because you cannot tell equal elements apart anyway.

It starts to matter when you use comparing to sort objects by a certain feature. Here we sort the list of pairs by their first elements in descending order:

> sortBy (comparing (Down . fst)) [(1,'a'),(2,'b'),(2,'c')]
[(2,'b'),(2,'c'),(1,'a')]

According to our criterion, the elements (2,'b') and (2,'c') are considered equal, but we can see that their ordering has been preserved.

The revese-based solution, on the other hand, reverses the order of equal elements, too:

> (reverse . sortBy (comparing fst)) [(1,'a'),(2,'b'),(2,'c')]
[(2,'c'),(2,'b'),(1,'a')]

Sort with care!

A note on sorted lists

The original version of this article said that sort runs in $\Theta(n \log n)$ time. @obadzz points out that this is not true: sort is implemented in such a way that it will run linearly when the list is already almost sorted in any direction. Thus I have replaced $\Theta(n \log n)$ with $O(n \log n)$ when talking about sort’s complexity.

Why reverse in the benchmark?

Jens Petersen asks why in the benchmark above I’m comparing

nf (sortBy (comparing Down)) list

nf sort (reverse list)

I include sort here as a baseline, to demonstrate that sortBy (comparing Down) does not have any significant overhead comparing to a simple sort. But why have reverse there? Is that a typo?

First of all, it’s important to understand what

nf sort (reverse list)

does. It may look as if it’s measuring the time it takes to reverse and then sort a list—and why would anyone want to do such a thing?

But that’s not what’s happening. In criterion, the nf combinator takes two separate values: the function (here, sort) and its argument (here, (reverse list)). Both the function and the argument are precomputed, and what’s measured repeatedly is the time it takes to evaluate the application of the function to its argument.

The reason I measure the time it takes to sort the reversed list is to have more of an apples-to-apples comparison. If I compared

nf (sortBy (comparing Down)) list

nf sort (reverse list)

… I would be comparing the same algorithm but on the arguments on which it takes two different code paths: in one case it ends up reversing the list, in the other keeping it intact. (Counterintuitively, it looks like it’s faster for sort/sortBy to reverse the list than to keep it as-is.)

Therefore, by reversing the argument, I make sure that in all three cases (sort, sortBy, and sortOn) the input list “looks” the same as seen by the comparison function, and in all three cases the sorting algorithm needs to reverse the list.

rank vs. order in R

Sat, 19 Mar 2016 20:00:00 +0000

A lot of people (myself included) get confused when they are first confronted with rank and order functions in R. Not only do the descriptions of these functions sound related (they both have to do with how a vector’s elements are arranged when the vector is sorted), but their return values may seem identical at first.

Here’s how my first encounter with these two functions went. The easiest thing is to see how they work on already sorted vectors:

> rank(1:3)
[1] 1 2 3
> order(1:3)
[1] 1 2 3

Fair enough. Now let’s try a reversed sorted vector.

> rank(rev(1:3))
[1] 3 2 1
> order(rev(1:3))
[1] 3 2 1

Uhm, ok. I guess I should try to shuffle the elements.

> rank(c(1,3,2))
[1] 1 3 2
> order(c(1,3,2))
[1] 1 3 2

Perhaps 3 elements is too few.

> rank(c(1,3,2,4))
[1] 1 3 2 4
> order(c(1,3,2,4))
[1] 1 3 2 4

Or maybe I shouldn’t use small consequtive numbers?

> rank(c(10,30,20,40))
[1] 1 3 2 4
> order(c(10,30,20,40))
[1] 1 3 2 4

So, where is the difference, and why is it so hard to find?

Definitions

Let’s say we have a sequence of numbers $a_1, a_2, \ldots, a_n$. For simplicity, assume that all numbers are distinct.

Sorting this sequence yields a permutation $a_{s_1},a_{s_2},\ldots,a_{s_n}$, where $s_1$ is the index of the smallest $a_i$, $s_2$ is the index of the second smallest one, and so on, up to $s_n$, the index of the greatest $a_i$.

The sequence $s_1,s_2,\ldots,s_n$ is what the order function returns. If a is a vector, a[order(a)] is the same vector but sorted in the ascending order.

Now, the rank of an element is its position in the sorted vector. The rank function returns a vector $t_1,\ldots,t_n$, where $t_i$ is the position of $a_i$ within $a_{s_1},\ldots,a_{s_n}$.

It is hard to tell in general where $a_i$ will occur among $a_{s_1},\ldots,a_{s_n}$; but we know exactly where $a_{s_k}$ occurs: on the $k$th position! Thus, $t_{s_k}=k$.

Inverse and involutive permutations

Considered as functions (permutations), $s$ and $t$ are inverse ($s\circ t = t\circ s = id$). Or, expressed in R:

> all((rank(a))[order(a)] == 1:length(a))
[1] TRUE
> all((order(a))[rank(a)] == 1:length(a))
[1] TRUE

In the beginning, we saw several examples of $s$ and $t$ being the same, i.e. $s\circ s = id$. Such functions are called involutions.

That our examples led to involutive permutations was a coincidence, but not an unlikely one. Indeed, for $n=2$, all two permutations are involutions: $1,2$ and $2,1$. In general, permutations $1,2,\ldots,n$ and $n,n-1,\ldots,1$ will be involutions for any $n$; for $n=2$ it just happens so that there are no others.

For $n=3$, we have a total of $3!=6$ permutations. Two of them, the identical permutation and its opposite, are involutions as discussed above. Out of the remaining 4, half are involutions ($1,3,2$ and $2,1,3$) and the other half are not ($2,3,1$ and $3,1,2$). So, for $n=3$, the odds are 2 to 1 that order and rank will yield the same result.

Any permutation consists of one or more cycles. The non-involutive $2,3,1$ and $3,1,2$ are cycles of their own, while $1,3,2$ consists of two cycles: $(1)$ of size 1 and $(2\;3)$ of size 2. The sum of cycle sizes, of course, must equal $n$.

It is not hard to see that a permutation is involutive if and only if all its cycles are of sizes 1 or 2. This explains why involutions are so common for $n=3$; there’s not much room for longer cycles. For larger $n$, however, the situation changes, and getting at least one cycle longer than 2 becomes inevitable as $n$ grows.

We can easily compute the odds that rank and order coincide for a random permutation of size $n$. If $a$ is itself a permutation (e.g. if its generated by sample(n) in R), then $t=a$. All we need to do is to figure out how many involutions there are among $n!$ permutations of size $n$. That number is given by

\[ I(n)=1+\sum_{k=0}^{\lfloor(n-1)/2\rfloor} \frac{1}{(k+1)!} \prod_{i=0}^k \binom{n-2i}{2} \]

(hint: $k+1$ is the number of cycles of size 2).

I <- function(nn) {
  sapply(nn, function(n) {
    1 + sum(sapply(0:floor((n-1)/2),
      function(k) {
        prod(sapply(0:k, function(i) {
          choose(n-2*i,2)
        })) / factorial(k+1)
      }))
  })
}

n <- 1:10
plot(I(n)/factorial(n) ~ n,type='b')
grid(col=rgb(0.3,0.3,0.7))

Reducing boilerplate in finally tagless style

Wed, 03 Feb 2016 20:00:00 +0000

Introduction

Typed Tagless, a.k.a tagless-final or finally tagless, is an approach to embedding DSLs and modeling data in general, advocated by Oleg Kiselyov. Instead of defining a set of algebraic data types to describe data or terms, thus focusing on how data is constructed, the approach focuses on data consumption, defining a canonical eliminator for every constructor that we would otherwise define.

For instance, instead of defining lists as

data List a = Nil | Cons a (List a)

we would define a class

class List rep a where
  nil :: rep
  cons :: a -> rep -> rep

which of course corresponds to the Böhm-Berarducci (or Church) encoding of the above algebraic type.

Oleg has written extensively on the merits of this approach. In this article, I want to discuss a certain aspect of writing transformations in the finally tagless style.

The use case: language-integrated query

Oleg, together with Kenichi Suzuki and Yukiyoshi Kameyama, have published a paper Finally, Safely-Extensible and Efficient Language-Integrated Query. In this paper, they employ the finally tagless approach to embed, optimize, and interpret SQL queries in OCaml.

Here are some excerpts from their OCaml code:

(* Base Symantics *)
module type Symantics_base = sig
  ...
  (* lambda abstract *)
  val lam     : ('a repr -> 'b repr) -> ('a -> 'b) repr
  (* application *)
  val app     : ('a -> 'b) repr -> 'a repr -> 'b repr
  ...
end

(* Symantics with list operations *)
module type SymanticsL = sig
  include Symantics

  (* comprehension *)
  val foreach : (unit -> 'a list repr) ->
                ('a repr ->  'b list repr) -> 'b list repr
  (* condition *)
  val where   :  bool repr -> (unit -> 'a list repr) -> 'a list repr
  (* yield singleton list *)
  val yield   : 'a repr ->  'a list repr
  (* empty list *)
  val nil     :  unit -> 'a list repr
  (* not empty *)
  val exists  :  'a list repr ->  bool repr
  (* union list *)
  val (@%)    : 'a list repr -> 'a list repr -> 'a list repr

  (* the table constructor which take a table name and table contents *)
  val table : (string * 'a list) -> 'a list repr
end

(‘Symantics’ is not a typo; it’s a portmanteau of ‘syntax’ and ‘semantics’.)

Transformations

A SQL trasnformation (such as transforming a subquery to a join) is represented by an ML functor, i.e. a function mapping one SymanticsL to another, which interprets the term slightly differently than the original one. I say slightly, because normally a transformation touches only a few relevant methods. The others are transformed mechanically following the Reflection-Reification pattern (RR). Informally speaking, we leave the irrelevant methods unchanged, applying the minimal transformation that makes them typecheck.

The question is, how to avoid mentioning irrelevant methods when defining a transformation?

This question is not idle. The language-integrated query code contains about 40 methods and 13 transformations. Pause for a second and imagine the amount of boilerplate that would have to be written if we needed to define every single method for every transformation. As we will see below, ML modules make this a non-issue. In Haskell, however, it is an issue, exhibited in Oleg’s own Haskell example (although easy to miss for a class that only contains 3 methods).

In OCaml, the RR is defined as a transformation of the whole module:

module OL(X:Trans)
         (F:SymanticsL with type 'a repr = 'a X.from)  = struct
  include O(X)(F)
  open X

  let foreach src body =
    fwd (F.foreach (fun () -> bwd (src ())) (fun x -> bwd (body (fwd x))))
  let where test body  =
    fwd (F.where (bwd test) (fun () -> bwd (body ())))
  let yield e    = fmap F.yield e
  let nil ()     = fwd (F.nil ())
  let exists e   = fmap F.exists e
  let (@%) e1 e2 = fmap2 F.(@%) e1 e2

  let table (name,data) =
    fwd @@ F.table (name, data)
end

When they define a transformation, they first transform the module in this mechanical fashion, and then override the few relevant methods:

module AbsBeta_pass(F:SymanticsL) = struct
  module X0 = struct
    type 'a from = 'a F.repr
    type 'a term = Unknown : 'a from -> 'a term
             | Lam     : ('a term -> 'b term) -> ('a -> 'b) term
    let fwd x = Unknown x                              (* generic reflection *)
    let rec bwd : type a. a term -> a from = function  (* reification *)
      | Unknown e -> e
      | Lam f     -> F.lam (fun x -> bwd (f (fwd x)))
  end
  open X0
  module X = Trans_def(X0)
  open X
  (* optimization *)
  module IDelta = struct
    let lam f = Lam f
    let app e1 e2 =
      match e1 with
      | Lam f -> f e2
      | _ -> fmap2 F.app e1 e2
  end
end

(* Combine the concrete optimization with the default optimizer *)
module AbsBeta(F:SymanticsL) = struct
  module M = AbsBeta_pass(F)
  include OL(M.X)(F)        (* the default optimizer *)
  include M.IDelta          (* overriding `lam` and `app` *)
end

How can we do this in Haskell?

Explicit dictionaries

An explicit dictionariy (a data type containing methods as its fields) seems like a great fit for Symantics. The RR transformation would be a simple function mapping one record to another. To define a transformation, we would override the relevant methods via record update.

However, explicit dictionaries are not that well suited for the finally tagless style. In OCaml, you can include one module into another (notice include Symantics in the OCaml code above). This “unpacks” the contents of one module into another, so that when you open the second module, the contents of the first module is available, too.

This is important for the finally tagless style. One of its strength is extensibility, which is achieved through such inclusion. Consequently, deep inclusion chains are common. With Haskell’s data types, unpacking such chains manually at every use site will quickly become unwieldy.

Type classes

Type classes are better suited for inclusion. If we declare

class Symantics1 rep => Symantics2 rep where { ... }

and impose a Symantics2 rep constraint on a function definition, the methods of Symantics1 become available without any additional effort.

But then we don’t have good support for RR. Type class instances are not first class citizens; we can’t declare a function that transforms one instance into another. Nor can we create one instance from another by overriding a few methods… Or can we?

We can achieve our goal by using default method signatures.

We define the RR transformation simultaneously with the class itself:

class Symantics rep where
  lam :: (rep a -> rep b) -> rep (a -> b)
  default lam :: RR t rep => (t rep a -> t rep b) -> t rep (a -> b)
  lam f = fwd $ lam $ bwd . f . fwd

  app :: rep (a -> b) -> rep a -> rep b
  default app :: RR t rep => t rep (a -> b) -> t rep a -> t rep b
  app f x = fwd $ bwd f `app` bwd x

  foreach :: rep [a] -> (rep a -> rep [b]) -> rep [b]
  default foreach :: RR t rep => t rep [a] -> (t rep a -> t rep [b]) -> t rep [b]
  foreach a b = fwd $ foreach (bwd a) (bwd . b . fwd)

  ...

The implementation of RR is straightforward:

class RR t rep where
  fwd :: rep a -> t rep a
  bwd :: t rep a -> rep a

Now let’s define the AbsBeta pass in Haskell.

data AbsBeta rep a where
  Unknown :: rep a -> AbsBeta rep a
  Lam :: (AbsBeta rep a -> AbsBeta rep b) -> AbsBeta rep (a -> b)

instance Symantics rep => RR AbsBeta rep where
  fwd = Unknown
  bwd = \case
    Unknown t -> t
    Lam f -> lam (bwd . f . fwd)

instance Symantics rep => Symantics (AbsBeta rep) where
  lam = Lam
  app f x =
    case f of
      Unknown f' -> fwd $ app f' (bwd x)
      Lam b -> b x

All the methods not mentioned in the last instance get their default implementations based on RR, which is exactly what we wanted.

Associated types

Apart from methods, ML/OCaml modules can also define types. This is used in the Language-integrated query paper and code in the following way:

(* Base Symantics *)
module type Symantics_base = sig
  type 'a repr  (* representation type *)
  val observe : (unit -> 'a repr) -> 'a obs
  ...

In Haskell, we can replicate that with an associated type:

class SymanticsObs rep where
  type Obs rep :: * -> *

  observe :: rep a -> Obs rep a
  default observe :: RR t rep => t rep a -> Obs rep a
  observe = observe . bwd

The default definition for observe saves us from redefining it for derived representations, but what about Obs itself? We would like to write, in the spirit of default method signatures,

class SymanticsObs rep where
  type Obs rep :: * -> *
  type Obs (t rep) = rep

However, GHC would not let us to. Since recently, GHC does support default type declarations, but they need to be of the general form type Obs rep = ....

Nevertheless, we can create a type family that will extract the rep from t rep for us:

type family Peel (rep :: * -> *) :: (* -> *) where
  Peel (t rep) = rep

class SymanticsObs rep where
  type Obs rep :: * -> *
  type Obs rep = Obs (Peel rep)

  observe :: rep a -> Obs rep a
  default observe :: RR t rep => t rep a -> Obs rep a
  observe = observe . bwd

Now we can say

instance (Symantics rep, SymanticsObs rep) => SymanticsObs (AbsBeta rep)

without having to define either type Obs or observe explicitly.

Conclusion

Extensions such as default method signatures, default associated types, and type families can significantly reduce the boilerplate when defining transformations in the finally tagless style.

Update. Although I missed it on the first reading of the paper, /u/rpglover64 on reddit points out that the authors themselves acknowledge the boilerplate problem which this article addresses:

Haskell typeclasses made the encoding lightweight compared to OCaml modules. On the other hand, in OCaml we relied on the include mechanism to program optimizations by reusing the code for the identity transformation and overriding a couple of definitions. Haskell does not support that sort of code reuse among type classes. Therefore, programming tagless-final transformation in Haskell has quite a bit of boilerplate.

Fixing Permission denied (publickey). after an SSH upgrade

Mon, 21 Dec 2015 20:00:00 +0000

This weekend I upgraded my laptop from Fedora 22 to 23. Today, when I tried to push to github, I suddenly got

% git push     
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

To debug this, I ran (according to these instructions)

% ssh -vT git@github.com
OpenSSH_7.1p1, OpenSSL 1.0.2e-fips 3 Dec 2015
debug1: Reading configuration data /home/feuerbach/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to github.com [192.30.252.129] port 22.
debug1: Connection established.
debug1: identity file /home/feuerbach/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/feuerbach/.ssh/id_rsa-cert type -1
debug1: identity file /home/feuerbach/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /home/feuerbach/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/feuerbach/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/feuerbach/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/feuerbach/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/feuerbach/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.1
debug1: Remote protocol version 2.0, remote software version libssh-0.7.0
debug1: no match: libssh-0.7.0
debug1: Authenticating to github.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> none
debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> none
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8
debug1: Host 'github.com' is known and matches the RSA host key.
debug1: Found key in /home/feuerbach/.ssh/known_hosts:19
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/feuerbach/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Skipping ssh-dss key /home/feuerbach/.ssh/id_dsa for not in PubkeyAcceptedKeyTypes
debug1: Trying private key: /home/feuerbach/.ssh/id_ecdsa
debug1: Trying private key: /home/feuerbach/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

The important line is this one

debug1: Skipping ssh-dss key /home/feuerbach/.ssh/id_dsa for not in PubkeyAcceptedKeyTypes

It turns out that Fedora 23 ships with OpenSSH 7.1p1 which has disabled DSS (aka DSA) keys by default.

A short term solution is to add

Host *
PubkeyAcceptedKeyTypes=+ssh-dss

to ~/.ssh/config. A long-term solution is to replace the DSS keys with, say, RSA keys.

Torsors, midpoints, and homogeneous coordinates

Wed, 16 Dec 2015 20:00:00 +0000

Gustavo Goretkin writes:

Dear Roman,

I’ve thought about torsors for a while but only came across your page on it today.

Regarding the distinction between Vectors and Points, I think it’s useful to have this distinction in programming languages. Graphics people sometimes do this when they use normalized homogeneous coordinates. A point $P(x,y,z)$ is represented as $[x,y,z,1]$ and a vector $V(x,y,z)$ is represented as $[x,y,z,0]$. If you add a $V+V$, you get a $V$, if you add $P+V$, you get a $P$. If you do $P-P$, you get a $V$. Finally, If you add $P+P$, you get a homogeneous coordinate with last coordinate 2.

Now, I don’t know how to attach a geometric meaning to the point $P_1+P_2$, however, I think it is convenient to allow the average of $P_1$ and $P_2$ to be the point in between $P_1$ and $P_2$—halfway along the geodesic connecting the two points. Which is what happens if you normalize the homogeneous point above, because you divide all coordinates by two.

If I were using a programming language that did not allow me to add two points, I know what I’d do. I’d choose an arbitrary point $c$ and perform

\[\frac{(P_1-c) + (P_2-c)}2 + c.\]

This feels like a dilemma, though, because this produces the identical result as if you had allowed $\frac{P_1+P_2}2$. Perhaps my dilemma is purely rooted in pragmatics, but I wanted to know if you had any thoughts.

Is this even legal?

So we have two ways to compute the midpoint of $P_1P_2$. One is legal under the points-as-a-torsor interpretation and does not involve point addition:

\[\frac{(P_1-c) + (P_2-c)}2 + c,\]

where $c$ is arbitrary. A simpler (and still legal) formula is obtained by letting $c=P_1$:

\[\frac{P_2-P_1}2 + P_1.\]

On the other hand, we have a formula that “just works” if we allow point addition:

\[\frac{P_1+P_2}2.\]

Should the existence of this latter formula cast a shadow of doubt on the first two and the concept of a midpoint? Certainly not!

Why does the illegal formula work? Any torsor can be transformed into a proper linear space by choosing an arbtrary element of the torsor, $c$, and designating it as zero. Then we can compute any linear combination $\alpha P_1+\beta P_2$ just as we computed the midpoint above, $\alpha(P_1-c) + \beta(P_2 - c) + c.$

The problem with this approach, in general, is that the result will vary with the choice of $c$. Indeed,

\[ (\alpha(P_1-c_1) + \beta(P_2 - c_1) + c_1) - \\ (\alpha(P_1-c_2) + \beta(P_2 - c_2) + c_2) = \\ (\alpha+\beta-1)(c_2-c_1) \]

Thus, the result will be independent of $c$ only in the case $\alpha+\beta=1$, that is, when the linear combination is affine. Luckily, computing the midpoint happens to be such a combination, with $\alpha=\beta=1/2$.

Interface vs. implementation

If we wish to design a safer language that does not let us accidentally add points, we should not allow expressions like $\alpha P_1 + \beta P_2$, unless we can statically verify that $\alpha + \beta = 1$. But since, as we’ve established, an affine combination of two (and, by extension, $n$) points does make geometric sense, we would want to introduce a function like the following:

\[ \mathrm{affine}(\alpha_1,P_1;\alpha_2,P_2;\ldots;\alpha_n,P_n) = \frac{\alpha_1 P_1+\alpha_2 P_2+\ldots+\alpha_n P_n}{\alpha_1 +\alpha_2 +\ldots+\alpha_n} \label{affine} \]

The division by $\sum \alpha_i$ ensures that the combination is indeed affine and thus legal.

Internally, the function may be computed by any of the formulae shown above, even if some of them may be not allowed in the surface language. Usually, the straightforward definition above will be the fastest. A smarter algorithm could be employed to achieve better numeric stability. And in rare cases such as dealing with bounded numbers, the formula $\frac{P_2-P_1}2 + P_1$ is required to avoid overflow.

Homogeneous coordinates

It may seem as if our $\mathrm{affine}$ function is equivalent to addition in homogeneous coordinates; but there is one important distinction. Homogeneous coordinates are stateful; the sum $\sum\alpha_i$ is simply stored along the vector $\sum\alpha_iP_i$, and the division is implicit. Thus two geometrically identical points may behave differently when added to a third point:

\[ [6,6,6,1] + [0,0,0,1] = [6,6,6,2] \sim (3,3,3) \\ [12,12,12,2] + [0,0,0,1] = [12,12,12,3] \sim (4,4,4) \]

Homogeneous coordinates are a very powerful concept, but as a programming model they are error-prone. A safer model could be devised if the language can statically distinguish (say, through types) between points, vectors, and possibly denormalized points.

Testing FFT with R

Sat, 05 Dec 2015 20:00:00 +0000

In the previous article, we have built a simple FFT algorithm in Haskell. Now it’s time to test it, for correctness and for (asymptotic) efficiency. If you expect the conclusion to be “it’s good” on both accounts, read on; it may get a bit more interesting than that.

This is also a good occasion to introduce and play with inline-r, a Haskell package that allows to run R code from within Haskell.

Testing correctness

In R, FFT is available straight out of the box, without a need to import a single package.

We use tasty and quickcheck to generate random lists of complex numbers, run R’s FFT and our implementation on these random inputs and compare the results.

{-# LANGUAGE QuasiQuotes, ScopedTypeVariables, DataKinds, PartialTypeSignatures, GADTs #-}
import Test.Tasty
import Test.Tasty.QuickCheck
import Test.QuickCheck.Monadic
import Data.Complex

-- inline-r imports
import qualified Foreign.R as R (cast)
import Foreign.R.Type as R -- SComplex
import H.Prelude (runRegion, withEmbeddedR, defaultConfig)
import Language.R.QQ (r)
import Language.R.HExp as HExp (HExp(Complex),hexp)
import Data.Vector.SEXP as R (toList)

-- the module we wrote in the previous article
import FFT

-- Call R's FFT
r_fft :: [Complex Double] -> IO [Complex Double]
r_fft nums = runRegion $ do
  r_result1 <- [r|fft(nums_hs)|]
  let r_result2 = R.cast R.SComplex r_result1
      HExp.Complex r_result3 = hexp r_result2
      r_result4 = R.toList r_result3
  return r_result4

main =
  withEmbeddedR defaultConfig
  . defaultMain
  . testProperty "Haskell vs. R"
  $ \(nums :: [Complex Double]) -> monadicIO $ do

    r_result <- run $ r_fft nums

    let haskell_result = fst $ fft nums

    assert $
      Prelude.length r_result == Prelude.length haskell_result &&
      all ((<1e-8) . magnitude) (zipWith (-) r_result haskell_result)

The result?

Haskell vs. R: OK (0.28s)
  +++ OK, passed 100 tests.

All 1 tests passed (0.28s)

That’s reassuring; at least we are computing the right thing. Now let’s see if we’ve managed to stay within our $O(n \log n)$ bound.

Testing complexity

Luckily, our implementation already records the number of arithmetic operations it takes to compute the answer.

> snd . fft $ replicate 13 1
312

And we can follow this number as $n$ grows:

> map (\n -> snd . fft $ replicate n 1) [1..20]
[0,4,12,16,40,36,84,48,144,100,220,96,312,196,420,128,544,324,684,240]

But these numbers don’t tell us much. How do we know if this is $\Theta(n \log n)$ or $\Theta(n^2)$? For this, we will again turn to R.

R’s lm function approximates one value as a linear combination of other values. In this case, we’ll try to find a linear combination of $n$, $n \log n$, and $n^2$ that approximates the number of operations it takes our FFT implementation to complete on an input of size $n$. If it turns out that the coefficient of $n^2$ in this combination is a non-negligible positive number, then the complexity of our algorithm is probably not $O(n \log n)$.

{-# LANGUAGE QuasiQuotes #-}
import H.Prelude (runRegion, withEmbeddedR, defaultConfig)
import qualified H.Prelude as H
import Language.R.QQ (r)

import FFT

fft_ops :: Int -> Int
fft_ops n = snd . fft $ replicate n 1

analyze
  :: [Int] -- sequence of n's
  -> IO ()
analyze ns = runRegion $ do
  let ops = map fft_ops ns

      ns_dbl, ops_dbl :: [Double]
      ns_dbl = map fromIntegral ns
      ops_dbl = map fromIntegral ops

  H.print =<< [r| ops <- ops_dbl_hs; n <- ns_dbl_hs; lm(ops ~ I(n^2) + I(n*log(n)) + n)|]

main = withEmbeddedR defaultConfig $ do
  analyze [ 2^k   | k <- [1..10]]
  analyze [ 2*k-1 | k <- [1..30]]

First, we try $n$s which are powers of 2. Then the number of points for evaluation halves (while remaining even!) at each level of our divide-and-conquer algorithm.

Call:
lm(formula = ops ~ I(n^2) + I(n * log(n)) + n)

Coefficients:
  (Intercept)         I(n^2)  I(n * log(n))              n
    3.036e-13      1.840e-17      2.885e+00      3.399e-14

The number of steps is rather well approximated by $2.885\, n \log n$. That’s a win! (By the way, can you tell or guess where the number 2.885 comes from?)

But remember, the number of points only shrinks when it’s even. What the number is odd from the very beginning? Then it’ll never reduce at all!

Call:
lm(formula = ops ~ I(n^2) + I(n * log(n)) + n)

Coefficients:
  (Intercept)         I(n^2)  I(n * log(n))              n
    2.657e-12      2.000e+00      1.834e-13     -2.000e+00

Our fears are confirmed; this time the $n \log n$ coefficient is negligible, and the number of operations appears to be $2(n^2-n)$.

For an arbitrary number $n$, the efficiency of the algorithm depends on the largest power of 2 that divides $n$. If $n = 2^m q$, where $m$ is odd, then $n$ will halve during the first $m$ steps, and then stabilize at $q$.

Concluding remarks on inline-r

The way inline-r works is very cool. It takes advantage of some unique features of R; for example, that the type of R’s AST and the type of its runtime values is the same type, just like in Lisp.

So the quasi-quote [r|...|] works by calling a normal R function that parses the expression and returns its AST as an R value. Then inline-r traverses the AST, replaces metavariables like foo_hs by values dynamically constructed from foo, and passes the expression-as-a-value back to R for evaluation.

To learn more, refer to the paper Project H: Programming R in Haskell. Also, if you understand Russian, listen to our podcast where we discuss inline-r with Alexander Vershilov, one of its authors.

On the other hand, the code using inline-r may get somewhat bulky, as in the r_fft function. It’s hard for me to say yet whether this complexity is justified. The module organization is questionable; in our first program, accessing fairly basic functionality required importing 6 different modules. Finally, the documentation lacks in content and organization.

Yet this is a very impressive young project, and I would love to see its continued development.

Simple FFT in Haskell

Fri, 04 Dec 2015 20:00:00 +0000

The article develops a simple implementation of the fast Fourier transform in Haskell.

Raw performance of the algorithm is explicitly not a goal here; for instance, I use things like nub, Writer, and lists for simplicity. On the other hand, I do pay attention to the algorithmic complexity in terms of the number of arithmetic operations performed; the analysis thereof will be done in a subsequent article.

Background

Discrete Fourier transform turns $n$ complex numbers $a_0,a_1,\ldots,a_{n-1}$ into $n$ complex numbers

\[f_k = \sum_{l=0}^{n-1} e^{- 2 \pi i k l / n} a_l.\]

An alternative way to think about $f_k$ is as the values of the polynomial

\[P(x)=\sum_{l=0}^{n-1} a_l x^l\]

at $n$ points $w^0,w^1,\ldots,w^{n-1}$, where $w=e^{-2 \pi i / n}$ is a certain $n$th primitive root of unity.

The naive calculation requires $\Theta(n^2)$ operations; our goal is to reduce that number to $\Theta(n \log n)$.

An excellent explanation of the algorithm (which inspired this article in the first place) is given by Daniel Gusfield in his video lectures; he calls it “the most important algorithm that most computer scientists have never studied”. You only need to watch the first two lectures (and maybe the beginning of the third one) to understand the algorithm and this article.

Roots of unity

Roots of unity could in principle be represented in the Cartesian form by the Complex a type. However, that would make it very hard to compare them for equality, which we are going to do to achieve a subquadratic complexity.

So here’s a small module just for representing these special complex numbers in the polar form, taking advantage of the fact that their absolute values are always 1 and their phases are rational multiples of $\pi$.

module RootOfUnity
  ( U -- abstract
  , mkU
  , toComplex
  , u_pow
  , u_sqr
  ) where

import Data.Complex

-- | U q corresponds to the complex number exp(2 i pi q)
newtype U = U Rational
  deriving (Show, Eq, Ord)

-- | Convert a U number to the equivalent complex number
toComplex :: Floating a => U -> Complex a
toComplex (U q) = mkPolar 1 (2 * pi * realToFrac q)

-- | Smart constructor for U numbers; automatically performs normalization
mkU :: Rational -> U
mkU q = U (q - realToFrac (floor q))

-- | Raise a U number to a power
u_pow :: U -> Integer -> U
u_pow (U q) p = mkU (fromIntegral p*q)

-- | Square a U number
u_sqr :: U -> U
u_sqr x = u_pow x 2

Fast Fourier transform

{-# LANGUAGE ScopedTypeVariables #-}
module FFT (fft) where

import Data.Complex
import Data.Ratio
import Data.Monoid
import qualified Data.Map as Map
import Data.List
import Data.Bifunctor
import Control.Monad.Trans.Writer
import RootOfUnity

So we want to evaluate the polynomial $P(x)=\sum_{l=0}^{n-1}a_lx^l$ at points $w^k$. The trick is to represent $P(x)$ as $A_e(x^2) + x A_o(x^2)$, where $A_e(x)=a_0+a_2 x + \ldots$ and $A_o(x)=a_1+a_3 x + \ldots$ are polynomials constructed out of the even-numbered and odd-numbered coefficients of $P$, respectively.

When $x$ is a root of unity, so is $x^2$; this allows us to apply the algorithm recursively to evaluate $A_e$ and $A_o$ for the squared numbers.

But the real boon comes when $n$ is even; then there will be half as many of these squared numbers, because $w^k$ and $w^{k+n/2}$, when squared, both give the same number $w^{2k}$. This is when the divide and conquer strategy really pays off.

We will represent a polynomial $\sum_{l=0}^{n-1}a_lx^l$ in Haskell as a list of coefficients [a_0,a_1,...], starting with $a_0$.

To be able to split a polynomial into the even and odd parts, let’s define a corresponding list function

split :: [a] -> ([a], [a])
split = foldr f ([], [])
  where
    f a (r1, r2) = (a : r2, r1)

(I think I learned the idea of this elegant implementation from Dominic Steinitz.)

Now, the core of the algorithm: a function that evaluates a polynomial at a given list of points on the unit circle. It tracks the number of performed arithmetic operations through a Writer monad over the Sum monoid.

evalFourier
  :: forall a . RealFloat a
  => [Complex a] -- ^ polynomial coefficients, starting from a_0
  -> [U] -- ^ points at which to evaluate the polynomial
  -> Writer (Sum Int) [Complex a]

If the polynomial is a constant, there’s not much to calculate. This is our base case.

evalFourier []  pts = return $ 0 <$ pts
evalFourier [c] pts = return $ c <$ pts

Otherwise, use the recursive algorithm outlined above.

evalFourier coeffs pts = do
  let
    squares = nub $ u_sqr <$> pts -- values of x^2
    (even_coeffs, odd_coeffs) = split coeffs
  even_values <- evalFourier even_coeffs squares
  odd_values <- evalFourier odd_coeffs squares

  let
    -- a mapping from x^2 to (A_e(x^2), A_o(x^2))
    square_map =
      Map.fromList
      . zip squares
      $ zip even_values odd_values

    -- evaluate the polynomial at a single point
    eval1 :: U -> Writer (Sum Int) (Complex a)
    eval1 x = do
      let (ye,yo) = (square_map Map.! u_sqr x)
          r = ye + toComplex x * yo
      tell $ Sum 2 -- this took two arithmetic operations
      return r

  mapM eval1 pts

The actual FFT function is a simple wrapper around evalFourier which substitutes the specific points and performs some simple conversions. It returns the result of the DFT and the number of operations performed.

fft :: RealFloat a => [Complex a] -> ([Complex a], Int)
fft coeffs =
  second getSum
  . runWriter 
  . evalFourier coeffs 
  . map (u_pow w)
  $ [0..n-1]
  where
    n = genericLength coeffs
    w = mkU (-1 % n)

Static linking with ghc

Mon, 26 Oct 2015 20:00:00 +0000

Recently I needed to build a Haskell program that would run on my DigitalOcean box. The problem was that my laptop’s Linux distro (Fedora 22) was different from my server’s distro (Debian jessie), and they had different versions of shared libraries.

I could build my app directly on the server, but I decided to go with static linking instead. I didn’t find a lot of information about static linking with ghc on the internet, hence this article.

First, let’s clarify something. There are two kinds of libraries any Haskell program links against: Haskell libraries and non-Haskell (most often, C) libraries. Haskell libraries are linked statically by default; we don’t need to worry about them. ghc’s -static and -dynamic flag affect that kind of linking.

On the other hand, non-Haskell libraries are linked dynamically by default. To change that, we need to pass the following options to ghc:

-optl-static -optl-pthread

If you are using stack (as I did), the whole command becomes

stack build --ghc-options='-optl-static -optl-pthread' --force-dirty

--force-dirty may be needed because stack may not recognize the options change as a sufficient reason to re-run ghc; this may get fixed in future versions of stack.

The command may fail in case you don’t have some of the static libraries installed. In my case, the dynamic version of the executable had these dynamic dependencies (as reported by ldd):

linux-vdso.so.1 (0x00007ffcb20c2000)
librt.so.1 => /lib64/librt.so.1 (0x00007fa435dc6000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fa435bc3000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fa4359be000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fa43574e000)
libgmp.so.10 => /lib64/libgmp.so.10 (0x00007fa4354d6000)
libm.so.6 => /lib64/libm.so.6 (0x00007fa4351cd000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa434fb6000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa434bf6000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa4349d9000)
/lib64/ld-linux-x86-64.so.2 (0x000055571e53e000)

To satisfy them statically, I had to install only three Fedora packages:

pcre-static.x86_64
gmp-static.x86_64
glibc-static.x86_64

MonadFix example: compiling regular expressions

Wed, 02 Sep 2015 20:00:00 +0000

{-# LANGUAGE RecursiveDo, BangPatterns #-}
import Control.Applicative
import Data.Function (fix)
import Data.IntMap as IntMap
import Control.Monad.Fix (mfix)
import Control.Monad.Trans.State
import Control.Monad.Trans.Class (lift)
import Text.Read (readMaybe)

MonadFix is an odd beast; many Haskell programmers will never use it in their careers. It is indeed very rarely that one needs MonadFix; and for that reason, non-contrived cases where MonadFix is needed are quite interesting to consider.

In this article, I’ll introduce MonadFix and show how it can be handy for compiling the Kleene closure (also known as star or repetition) of regular expressions.

What is MonadFix?

If you hear about MonadFix for the first time, you might think that it is needed to define recursive monadic actions, just like ordinary fix is used to define recursive functions. That would be a mistake. In fact, fix is just as applicable to monadic actions as it is to functions:

guessNumber m = fix $ \repeat -> do
  putStrLn "Enter a guess"
  n <- readMaybe <$> getLine
  if n == Just m
    then putStrLn "You guessed it!"
    else do
      putStrLn "You guessed wrong; try again"
      repeat

So, what is mfix for? First, recall that in Haskell, one can create recursive definitions not just for functions (which makes sense in other, non-lazy languages) or monadic actions, but for ordinary data structures as well. This is known as cyclic (or circular, or corecursive) definitions; and the technique itself is sometimes referred to as tying the knot.

The classic example of a cyclic definition is the (lazy, infinite) list of Fibonacci numbers:

fib = 0 : 1 : zipWith (+) fib (tail fib)

Cyclic definitions are themselves rare in day-to-day Haskell programming; but occasionally, the right hand side will be not a pure value, but a monadic computation that needs to be run in order to obtain the value.

Consider this (contrived) example, where we start the sequence with an arbitrary number entered by the user:

fibIO1 = do
  putStrLn "Enter the start number"
  start <- read <$> getLine
  return $ start : 1 : zipWith (+) fibIO1 (tail fibIO1)

This doesn’t typecheck because fibIO is not a list; it’s an IO action that produces a list.

But if we try to run the computation, it doesn’t make much sense either:

fibIO2 = do
  putStrLn "Enter the start number"
  start <- read <$> getLine
  fib <- fibIO2
  return $ start : 1 : zipWith (+) fib (tail fib)

This version of fibIO will ask you to enter the start number ad infinitum and never get to evaluating anything.

Of course, the simplest thing to do would be to move IO out of the recursive equation; that’s why I said the example was contrived. But MonadFix gives another solution:

fibIO3 = mfix $ \fib -> do
  putStrLn "Enter the start number"
  start <- read <$> getLine
  return $ start : 1 : zipWith (+) fib (tail fib)

Or, using the do-rec syntax:

fibIO4 = do
  rec
    fib <- do
      putStrLn "Enter the start number"
      start <- read <$> getLine
      return $ start : 1 : zipWith (+) fib (tail fib)
  return fib

Compiling regular expressions

As promised, I am going to show you an example usage of MonadFix that solved a problem other than “how could I use MonadFix?”. This came up in my work on regex-applicative.

For a simplified presentation, let’s consider this type of regular expressions:

data RE
  = Sym Char  -- symbol
  | Seq RE RE -- sequence
  | Alt RE RE -- alternative
  | Rep RE    -- repetition

Our goal is to compile a regular expression into a corresponding NFA. The states will be represented by integer numbers. State 0 corresponds to successful completion; and each Sym inside a regex will have a unique positive state in which we are expecting the corresponding character.

type NFAState = Int

The NFA will be represented by a map

type NFA = IntMap (Char, [NFAState])

where each state is mapped to the characters expected at that state and the list of states where we go in case we get the expected character.

To compile a regular expression, we’ll take as an argument the list of states to proceed to when the regular expression as a whole succeeds (otherwise we’d have to compile each subexpression separately and then glue NFAs together). This is essentially the continuation-passing style; only instead of functions, our continuations are NFA states.

During the compilation, we’ll use a stack of two State monads: one to assign sequential state numbers to Syms; the other to keep track of the currently constructred NFA.

-- Returns the list of start states and the transition table
compile :: RE -> ([NFAState], NFA)
compile re = runState (evalStateT (go re [0]) 0) IntMap.empty

-- go accepts exit states, returns entry states
go :: RE -> [NFAState] -> StateT NFAState (State NFA) [NFAState]
go re exitStates =
  case re of
    Sym c -> do
      !freshState <- gets (+1); put freshState
      lift $ modify' (IntMap.insert freshState (c, exitStates))
      return [freshState]
    Alt r1 r2 -> (++) <$> go r1 exitStates <*> go r2 exitStates
    Seq r1 r2 -> go r1 =<< go r2 exitStates

This was easy so far: alternatives share their exit states and their entry states are combined; and consequtive subexpressions are chained. But how do we compile Rep? The exit states of the repeated subexpression should become its own entry states; but we don’t know the entry states until we compile it!

And this is precisely where MonadFix (or recursive do) comes in:

    Rep r -> do
      rec
        let allEntryStates = ownEntryStates ++ exitStates
        ownEntryStates <- go r allEntryStates
      return allEntryStates

Why does this circular definition work? If we unwrap the State types, we’ll see that the go function actually computes a triple of three non-strict fields:

The last used state number
The list of entry states
The NFA map

The elements of the triple may depend on each other as long as there are no actual loops during evaluation. One can check that the fields can be indeed evaluated linearly in the order in which they are listed above:

The used state numbers at each step depend only on the regular expression itself, so it can be computed wihtout knowing the other two fields.
The list of entry states relies only on the state number information; it doesn’t need to know anything about the NFA transitions.
The NFA table needs to know the entry and exit states; but that is fine, we can go ahead and compute that information without creating any reverse data dependencies.

Better YAML parsing

Sun, 26 Jul 2015 20:00:00 +0000

If you need to parse a YAML file in Haskell today, you will probably reach for Michael Snoyman’s yaml package.

That parser works in two stages.

During the first stage, it parses YAML into a generic representation, such as an array of dictionaries of strings. For this, the yaml package uses the libyaml C library written by Kirill Simonov.

During the second stage, the generic representation is converted into the application-specific Haskell type. For instance, an abstract dictionary may be mapped to a record type.

This idea of two-stage parsing is borrowed from the aeson package, which parses JSON in a similar way. And because JSON’s and YAML’s data models are similar, the yaml package borrows from Aeson not only the above idea but also the generic representation and the machinery to convert it to Haskell types.

Thanks to this approach, if you have a FromJSON instance for a type, you can deserialize this type not only from JSON but also from the more human readable and writable YAML.

But there is a downside, too. Because Aeson’s primary goal is performance, it doesn’t try to provide good error messages or even validate the input beyond what’s necessary. This is not a problem for JSON because it is typically generated by programs.

But YAML is often written by humans, so it is important to detect possible mistakes and report them clearly.

Example

Consider a Haskell type representing a shopping cart item:

{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson (FromJSON(..), withObject, withText, (.:), (.:?), (.!=))
import Data.Yaml (decodeEither)
import Data.Text (Text)
import Control.Applicative

data Item = Item
  Text -- title
  Int -- quantity
  deriving Show

In YAML, an Item may be written as:

title: Shampoo
quantity: 100

In our application, most of the time the quantity will be 1, so we will allow two alternative simplified forms. In the first form, the quantity field is omitted and defaulted to 1:

title: Shampoo

In the second form, the object will be flattened to a bare string:

Shampoo

Here’s a reasonably idiomatic way to write an Aeson parser for this format:

defaultQuantity :: Int
defaultQuantity = 1

instance FromJSON Item where
  parseJSON v = parseObject v <|> parseString v
    where
      parseObject = withObject "object" $ \o ->
        Item <$>
          o .: "title" <*>
          o .:? "quantity" .!= defaultQuantity
        
      parseString = withText "string" $ \t ->
        return $ Item t defaultQuantity

With this example, I can now demonstrate the two weak spots of Aeson parsing: insufficient input validation and confusing error messages.

Validation

The following YAML parses successfully. But does the resulting Item match your expectations?

decodeEither "{title: Shampoo, quanity: 2}" :: Either String Item

Right (Item "Shampoo" 1)

If you look closer, you’ll notice that the word quantity is misspelled. But the parser doesn’t have any problem with that. Such a typo may go unnoticed for a long time and quitely affect how your application works.

Error reporting

Let’s say I am a returning user who vaguely remembers the YAML format for Items. I might have written something like

decodeEither "{name: Shampoo, quantity: 2}" :: Either String Item

Left "when expecting a string, encountered Object instead"

“That’s weird. I could swear this app accepted some form of an object where you could specify the quantity. But apparently I’m wrong, it only accepts simple strings.”

How to fix it

Check for unrecognized fields

To address the first problem, we need to know the set of acceptable keys. This set is impossible to extract from a FromJSON parser because it is buried inside an opaque function.

Let’s change parseJSON to have type FieldParser a, where FieldParser is an applicative functor that we’ll define shortly. The values of FieldParser can be constructed with combinators:

field
  :: Text -- ^ field name
  -> Parser a -- ^ value parser
  -> FieldParser a

optField
  :: Text -- ^ field name
  -> Parser a -- ^ value parser
  -> FieldParser (Maybe a)

The combinators are analogous to the ones I described in JSON validation combinators.

How can we implement the FieldParser type? One (“initial”) way is to use a free applicative functor and later interpret it in two ways: as a FromJSON-like parser and as a set of valid keys.

But there’s another (“final”) way which is to compose the applicative functor from components, one per required semantics. The semantics of FromJSON is given by ReaderT Object (Either ParseError). The semantics of a set of valid keys is given by Constant (HashMap Text ()). We take the product of these semantics to get the implementation of FieldParser:

newtype FieldParser a = FieldParser
  (Product
    (ReaderT Object (Either ParseError))
    (Constant (HashMap Text ())) a)

Here I used HashMap Text () instead of HashSet Text to be able to subtract this set from the object (represented as HashMap Text Value) later.

Another benefit of this approach is that it’s no longer necessary to give a name to the object (often called o) as in the Aeson-based parser. I’ve always found that awkward and unnecessary.

Improve error messages

Aeson’s approach to error messages is straightforward: it tries every alternative in turn and, if none succeeds, it returns the last error message.

There are two approaches to get a more sophisticated error reporting:

Collect errors from all alternatives and somehow merge them. Each error would carry its level of “matching”. An alternative that matched the object but failed at key lookup matches better than the one that expected a string instead of an object. Thus the error from the first alternative would prevail. If there are multiple errors on the same level, we should try to merge them. For instance, if we expect an object or a string but got an array, then the error message should mention both object and string as valid options.
Limited backtracking. This is what Parsec does. In our example, when it was determined that the object was “at least somewhat” matched by the first alternative, the second one would have been abandoned. This approach is rather restrictive: if you have two alternatives each expecting an object, the second one will never fire. The benefit of this approach is its efficiency (sometimes real, sometimes imaginary), since we never explore more than one alternative deeply.

It turns out, when parsing Values, we can remove some of the backtracking without imposing any restrictions. This is because we can “factor out” common parser prefixes. If we have two parsers that expect an object, this is equivalent to having a single parser expecting an object. To see this, let’s represent a parser as a record with a field per JSON “type”:

data Parser a = Parser
  { parseString :: Maybe (Text -> Either ParseError a)
  , parseArray  :: Maybe (Vector Value -> Either ParseError a)
  , parseObject :: Maybe (HashMap Text Value -> Either ParseError a)
  ...
  }

Writing a function Parser a -> Parser a -> Parser a which merges individual fields is then a simple exercise.

Why is every field wrapped in Maybe? How’s Nothing different from Just $ const $ Left "..."? This is so that we can see which JSON types are valid and give a better error message. If we tried to parse a JSON number as an Item, the error message would say that it expected an object or a string, because only those fields of the parser would be Just values.

The Parser type above can be mechanically derived from the Value datatype itself. In the actual implementation, I use generics-sop with great success to reduce the boilerplate. To give you an idea, here’s the real definition of the Parser type:

newtype ParserComponent a fs = ParserComponent (Maybe (NP I fs -> Either ParseError a))
newtype Parser a = Parser (NP (ParserComponent a) (Code Value))

We can then apply a Parser to a Value using this function.

Example revisited

Here is the same Item type and a combinator-based YAML parser:

{-# LANGUAGE OverloadedStrings #-}
import Data.Text (Text)
import Data.Maybe
import Data.Monoid
import Data.Yaml.Combinators

data Item = Item
  Text -- title
  Int -- quantity
  deriving Show

itemParser :: Parser Item
itemParser
  =  (flip Item 1 <$> string)
  <> (object $ Item
      <$> field "title" string
      <*> (fromMaybe 1 <$> optField "quantity" integer))

Let’s see now what errors it produces.

Validation

The YAML with a typo in the key name no longer parses:

either putStrLn print $ parse itemParser "{title: Shampoo, quanity: 2}"

Unexpected 

quanity: 2

as part of

quanity: 2
title: Shampoo

Error reporting

Since we supplied an object, the parser explains what’s wrong with that object without telling us it’d rather receive a string.

either putStrLn print $ parse itemParser "{name: Shampoo, quantity: 2}"

Expected field "title" as part of

quantity: 2
name: Shampoo

Implementation

I originally implemented these combinators as an internal module while working for Signal Vine in 2015. They kindly agreed to release it under the MIT instance, and I finally did so in 2017.

You can find the code packaged under the name yaml-combinators on hackage and github.

How Haskell handles signals

Mon, 06 Jul 2015 20:00:00 +0000

How is it possible to write signal handlers in GHC Haskell? After all, the set of system calls allowed inside signal handlers is rather limited. In particular, it is very hard to do memory allocation safely inside a signal handler; one would have to modify global data (and thus not be reentrant), call one of the banned syscalls (brk, sbrk, or mmap), or both.

On the other hand, we know that almost any Haskell code requires memory allocation. So what’s the trick?

The trick is that a Haskell handler is not installed as a true signal handler. Instead, a signal is handled by a carefully crafted RTS function generic_handler (rts/posix/Signals.c). All that function does (assuming the threaded RTS) is write the signal number and the siginfo_t structure describing the signal to a special pipe (called the control pipe, see GHC.Event.Control).

The other end of this pipe is being watched by the timer manager thread (GHC.Event.TimerManager). When awaken by a signal message from the control pipe, it looks up the handler corresponding to the signal number and, in case it’s an action, runs it in a new Haskell thread.

The signal handlers are stored in a global array, signal_handlers (GHC.Conc.Signal). When you install a signal action in Haskell, you put a stable pointer to the action’s code into the array cell corresponding to the signal number, so that the timer thread could look it up later when an actual signal is delivered.

How to force a list

Thu, 28 May 2015 20:00:00 +0000

Let’s say you need to force (evaluate) a lazy Haskell list.

A long time ago, this was a common way to fight lazy I/O: you read a String and then force it. These days you can have normal I/O with strict Text or ByteString instead.

Anyway, let’s say you do need to force a list. This came up in a pull request for lexer-applicative. Another scenario is if you want to evaluate a lazy Text or ByteString without copying the chunks. Or, you know, for any other reason.

First of all, how exactly do you want to force it? There are two primary ways: force the spine or force the elements too. (You can’t force the elements without forcing the spine.)

Forcing the spine means forcing all cons cells without touching the elements. One way to do that is to evaluate the length of the list, but that feels ad-hoc because it computes the result that is not needed. Here’s an elegant way to walk the spine:

forceSpine :: [a] -> ()
forceSpine = foldr (const id) ()

(Obviously, you need to force the resulting () value, by calling evaluate or seq-ing it to something else, for any evaluation to take place.)

const id, also known as flip const, returns its second argument while ignoring the first. So the evaluation goes like this:

forceSpine [x1, x2]
= foldr (const id) () [x1, x2]
= (const id) x1 $ foldr (const id) () [x2]
= foldr (const id) () [x2]
= (const id) x2 $ foldr (const id) () []
= foldr (const id) () []
= ()

See how forceSpine “unpacks” the list (thus forcing the spine) and throws all elements away.

I mentioned that you may also want to force the elements of the list, too. Most of the time you want to deep-force them, and so you should just rnf the whole list. Even when the elements are atomic (like Char or Int), evaluating them to weak head normal form is still equivalent to rnf.

But occasionally you do want to shallow-force the elements. In that case, simply replace const id with seq in the definition of forceSpine to obtain forceElements:

forceElements :: [a] -> ()
forceElements = foldr seq ()

Again, looking at the evaluation chain helps to understand what’s going on:

forceElements [x1, x2]
= foldr seq () [x1, x2]
= seq x1 $ foldr seq () [x2]
= foldr seq () [x2]
= seq x2 $ foldr seq () []
= foldr seq () []
= ()

Same as before, only elements get forced before being thrown away.

And here’s a table that may help you understand better the difference between seq, forceSpine, forceElements and rnf:

list	`seq` ()	`forceSpine`	`forceElements`	`rnf`
`[Just True]`	`()`	`()`	`()`	`()`
`[Just undefined]`	`()`	`()`	`()`	`undefined`
`[undefined]`	`()`	`()`	`undefined`	`undefined`
`True : undefined`	`()`	`undefined`	`undefined`	`undefined`
`undefined`	`undefined`	`undefined`	`undefined`	`undefined`

Since forceSpine and forceElements are based on foldr, they can be trivially generalized to any Foldable container, with the caveat that you should understand how the container and its Foldable instance work. For example, forceSpine is useless for Data.Map.Map, since it is already spine-strict, and forceElements for a tuple will only force its second element.

Announcing lambda prover

Wed, 27 May 2015 20:00:00 +0000

Over the last few days I wrote prover, a program that can reduce terms and prove equality in the untyped lambda calculus.

Motivation

Such a tool ought to exist, but I couldn’t find anything like that.

Occasionally I want to prove stuff, such as Monad or Applicative laws for various types. Very seldom such proofs require induction; mostly they involve simple reductions and equational reasoning.

Sometimes I specifically want to do it in the untyped lambda calculus just to prove a point. Other times, I’m interested in a typed theory (say, System F as an approximation for Haskell). Fortunately, the subject reduction property guarantees us that the results proven in ULC will hold in System F just as well.

Algebraic datatypes are more tricky. From the practical perspective, I could add data declarations and case expressions to prover, and desugar them via the Church encoding, just like I did with Maybe by hand. But there’s also the theoretical side of proving that the results about Church-encoded types translate back into the original language. Intuitively, that should hold, but I’d appreciate links to proper proofs if anyone is aware of them.

Finally, part of the motivation was to experiment with some pieces of Haskell tech that I don’t use in my day-to-day work and evaluate them. This part was certainly successful; I may share some specific impressions later. Until then, feel free to read the code; at this point it’s not too convoluted and yet I’m sure you’ll find some interesting bits there.

Demo

Here is an example invocation that establishes the right identity monad law for the reader monad:

% prover -i examples/reader.lam --equal 'bind a return' a
bind a return
  {- inline bind -}
= (λb c d. c (b d) d) a return
  {- inline return -}
= (λb c d. c (b d) d) a (λb c. b)
  {- β-reduce -}
= (λb c. b (a c) c) (λb c. b)
  {- β-reduce -}
= λb. (λc d. c) (a b) b
  {- β-reduce -}
= λb. (λc. a b) b
  {- β-reduce -}
= λb. a b
  {- η-reduce -}
= a

The file examples/reader.lam (included in the repository) contains the definitions of return and bind:

return = \x r. x ;
bind = \a k r. k (a r) r ;

You can also ask prover to reduce an expression:

% prover -i examples/arith.lam --reduce 'mult two three'            
λa b. a (a (a (a (a (a b)))))

Files are optional; you can give an entire term on the command line:

% prover --reduce '(\x. y x) (\z . z)'
y (λa. a)

prover uses De Bruijn indices, so bound variable names are not preserved.

One thing to note is that right now --reduce reports a fixed point of the reduction pipeline and not necessarily a normal form:

% prover --reduce '(\x. x x) (\x. x x)'
(λa. a a) (λa. a a)

If prover can’t find a reduced form, it will say so:

% prover --reduce '(\x. x x) (\x. x x x)'
No reduced form found

prover has a couple of knobs for more complicated cases. There is the number of iterations configured with --fuel; and for the --equal mode there is --size-limit, which instructs the tool to ignore large terms. E.g. this invocation completes immediately:

% prover -i examples/pair.lam -i examples/bool.lam -i examples/arith.lam --reduce 'pred one' 
λa b. b

But in order to get a nice proof for the same reduction, you’ll need to find the right limits and wait for about 7 seconds. You will also be surprised how non-trivial the proof is.

% prover -i examples/pair.lam -i examples/bool.lam -i examples/arith.lam \
  --equal 'pred one' zero \
  --fuel 50 --size-limit 40
pred one
  {- inline pred -}
= (λa b c. snd (a (λd. pair true (fst d b (λe. e) (snd d))) (pair false c))) one
[...]
= λa b. pair true ((λc. c (λd e. e) b) (λc d. c) a (λc. c) ((λc. c (λd e. e) b) (λc d. d))) (λc d. d)
[...]
= λa b. b

zero
  {- inline zero -}
= λa b. b

This is because --equal has to consider all reduction paths to find the minimal one, and there are too many different ways to reduce this term.

Finally, its majesty factorial:

% prover -i examples/pair.lam -i examples/bool.lam -i examples/arith.lam -i examples/fixpoint.lam \
  --fuel 20 \
  --reduce 'fact three' 
λa b. a (a (a (a (a (a b)))))

(I didn’t manage to compute fact four, though.)

Smarter validation

Sat, 02 May 2015 20:00:00 +0000

Today we’ll explore different ways of handling and reporting errors in Haskell. We shall start with the well-known Either monad, proceed to a somewhat less common Validation applicative, and then improve its efficiency and user experience.

The article contains several exercises that will hopefully help you better understand the issues that are being addressed here.

See also: Lazy validation.

Running example

{-# LANGUAGE GeneralizedNewtypeDeriving, KindSignatures, DataKinds,
             ScopedTypeVariables, RankNTypes, DeriveFunctor #-}
import Text.Printf
import Text.Read
import Control.Monad
import Control.Applicative
import Control.Applicative.Lift (Lift)
import Control.Arrow (left)
import Data.Functor.Constant (Constant)
import Data.Monoid
import Data.Traversable (sequenceA)
import Data.List (intercalate, genericTake, genericLength)
import Data.Proxy
import System.Exit
import System.IO
import GHC.TypeLits

Our running example will consist of reading a list of integer numbers from a file, one number per line, and printing their sum.

Here’s the simplest way to do this in Haskell:

printSum1 :: FilePath -> IO ()
printSum1 path = print . sum . map read . lines =<< readFile path

This code works as expected for a well-formed file; however, if a line in the file can’t be parsed as a number, we’ll get unhelpful

Prelude.read: no parse

Either monad

Let’s rewrite our function to be aware of possible errors.

parseNum
  :: Int -- line number (for error reporting)
  -> String -- line contents
  -> Either String Integer
     -- either parsed number or error message
parseNum ln str =
  case readMaybe str of
    Just num -> Right num
    Nothing -> Left $
      printf "Bad number on line %d: %s" ln str

-- Print a message and exit
die :: String -> IO ()
die msg = do
  hPutStrLn stderr msg
  exitFailure

printSum2 :: FilePath -> IO ()
printSum2 path =
  either die print .
  liftM sum .
  sequence . zipWith parseNum [1..] .
  lines =<< readFile path

Now, upon reading a line that is not a number, we’d see something like

Bad number on line 2: foo

This is a rather standard usage of the Either monad, so I won’t get into details here. I’ll just note that there are two ways in which this version is different from the first one:

We call readMaybe instead of read and, upon detecting an error, construct a helpful error message. For this reason, we keep track of the line number.
Instead of throwing a runtime exception right away (using the error function), we return a pure Either value, and then combine these Eithers together using the Moand Either isntance.

The two changes are independent; there’s no reason why we couldn’t use error and get the same helpful error message. The exceptions emulated by the Either monad have the same semantics here as the runtime exceptions. The benefit of the pure formulation is that the semantics of runtime exceptions is built-in; but the semantics of the pure data is programmable, and we will take advantage of this fact below.

Validation applicative

You get a thousand-line file with numbers from your accountant. He asks you to sum them up because his enterprise software crashes mysteriously when trying to read it.

You accept the challenge, knowing that your Haskell program won’t let you down. The program tells you

Bad number on line 378: 12o0

— I see! Someone put o instead of zero. Let me fix it.

You locate the line 378 in your editor and replace 12o0 with 1200. Then you save the file, exit the editor, and re-run the program.

Bad number on line 380: 11i3

— Come on! There’s another similar mistake just two lines below. Except now 1 got replaced by i. If you told me about both errors from the beginning, I could fix them faster!

Indeed, there’s no reason why our program couldn’t try to parse every line in the file and tell us about all the mistakes at once.

Except now we can’t use the standard Monad and Applicative instances of Either. We need the Validation applicative.

The Validation applicative combines two Either values in such a way that, if they are both Left, their left values are combined with a monoidal operation. (In fact, even a Semigroup would suffice.) This allows us to collect errors from different lines.

newtype Validation e a = Validation { getValidation :: Either e a }
  deriving Functor

instance Monoid e => Applicative (Validation e) where
  pure = Validation . Right
  Validation a <*> Validation b = Validation $
    case a of
      Right va -> fmap va b
      Left ea -> either (Left . mappend ea) (const $ Left ea) b

The following example demonstrates the difference between the standard Applicative instance and the Validation one:

> let e1 = Left "error1"; e2 = Left " error2"
> e1 *> e2
Left "error1"
> getValidation $ Validation e1 *> Validation e2
Left "error1 error2"

A clever implementation of the same applicative functor exists inside the transformers package. Ross Paterson observes that this functor can be constructed as

type Errors e = Lift (Constant e)

(see Control.Applicative.Lift).

Anyway, let’s use this to improve our summing program.

printSum3 :: FilePath -> IO ()
printSum3 path =
  either (die . intercalate "\n") print .
  liftM sum .
  getValidation . sequenceA .
  map (Validation . left (\e -> [e])) .
  zipWith parseNum [1..] .
  lines =<< readFile path

Now a single invocation of the program shows all the errors it can find:

Bad number on line 378: 12o0
Bad number on line 380: 11i3

Exercise. Could we use Writer [String] to collect error messages?

Exercise. When appending lists, there is a danger of incurring quadratic complexity. Does that happen in the above function? Could it happen in a different function that uses the Validation applicative based on the list monoid?

Smarter Validation applicative

Next day your accountant sends you another thousand-line file to sum up. This time your terminal gets flooded by error messages:

Bad number on line 1: 27297.
Bad number on line 2: 11986.
Bad number on line 3: 18938.
Bad number on line 4: 22820.
...

You already see the problem: every number ends with a dot. This is trivial to diagnose and fix, and there is absolutely no need to print a thousand error messages.

In fact, there are two different reasons to limit the number of reported errors:

User experience: it is unlikely that the user will pay attention to more than, say, 10 messages at once. If we try to display too many errors on a web page, it may get slow and ugly.
Efficiency: if we agree it’s only worth printing the first 10 errors, then, once we gather 10 errors, there is no point processing the data further.

Turns out, each of the two goals outlined above will need its own mechanism.

Bounded lists

We first develop a list-like datatype which stores only the first n elements and discards anything else that may get appended. This primarily addresses our first goal, user experience, although it will be handy for achieving the second goal too.

Although for validation purposes we may settle with the limit of 10, it’s nice to make this a generic, reusable type with a flexible limit. So we’ll make the limit a part of the type, taking advantage of the type-level number literals.

Exercise. Think of the alternatives to storing the limit in the type. What are their pros and cons?

On the value level, we will base the new type on difference lists, to avoid the quadratic complexity issue that I allude to above.

data BoundedList (n :: Nat) a =
  BoundedList
    !Integer -- current length of the list
    (Endo [a])

Exercise. Why is it important to cache the current length instead of computing it from the difference list?

Once we’ve figured out the main ideas (encoding the limit in the type, using difference lists, caching the current length), the actual implementation is straightforward.

singleton :: KnownNat n => a -> BoundedList n a
singleton a = fromList [a]

toList :: BoundedList n a -> [a]
toList (BoundedList _ (Endo f)) = f []

fromList :: forall a n . KnownNat n => [a] -> BoundedList n a
fromList lst = BoundedList (min len limit) (Endo (genericTake limit lst ++))
  where
    limit = natVal (Proxy :: Proxy n)
    len = genericLength lst

instance KnownNat n => Monoid (BoundedList n a) where
  mempty = BoundedList 0 mempty
  mappend b1@(BoundedList l1 f1) (BoundedList l2 f2)
    | l1 >= limit = b1
    | l1 + l2 <= limit = BoundedList (l1 + l2) (f1 <> f2)
    | otherwise = BoundedList limit (f1 <> Endo (genericTake (limit - l1)) <> f2)
    where
      limit = natVal (Proxy :: Proxy n)

full :: forall a n . KnownNat n => BoundedList n a -> Bool
full (BoundedList l _) = l >= natVal (Proxy :: Proxy n)

null :: BoundedList n a -> Bool
null (BoundedList l _) = l <= 0

SmartValidation

Now we will build the smart validation applicative which stops doing work when it doesn’t make sense to collect errors further anymore. This is a balance between the Either applicative, which can only store a single error, and Validation, which collects all of them.

Implementing such an applicative functor is not as trivial as it may appear at first. In fact, before reading the code below, I recommend doing the following

Exercise. Try implementing a type and an applicative instance for it which adheres to the above specification.

Did you try it? Did you succeed? This is not a rhetorical question, I am actually interested, so let me know. Is your implementation the same as mine, or is it simpler, or more complicated?

Alright, here’s my implementation.

newtype SmartValidation (n :: Nat) e a = SmartValidation
  { getSmartValidation :: forall r .
      Either (BoundedList n e) (a -> r) -> Either (BoundedList n e) r }
  deriving Functor

instance KnownNat n => Applicative (SmartValidation n e) where
  pure x = SmartValidation $ \k -> k <*> Right x
  SmartValidation a <*> SmartValidation b = SmartValidation $ \k ->
    let k' = fmap (.) k in
    case a k' of
      Left errs | full errs -> Left errs
      r -> b r

And here are some functions to construct and analyze SmartValidation values.

-- Convert SmartValidation to Either
fatal :: SmartValidation n e a -> Either [e] a
fatal = left toList . ($ Right id) . getSmartValidation

-- Convert Either to SmartValidation
nonFatal :: KnownNat n => Either e a -> SmartValidation n e a
nonFatal a = SmartValidation $ (\k -> k <+> left singleton a)

-- like <*>, but mappends the errors
(<+>)
  :: Monoid e
  => Either e (a -> b)
  -> Either e a
  -> Either e b
a <+> b = case (a,b) of
  (Right va, Right vb) -> Right $ va vb
  (Left e,   Right _)  -> Left e
  (Right _,  Left e)   -> Left e
  (Left e1,  Left e2)  -> Left $ e1 <> e2

Exercise. Work out what fmap (.) k does in the definition of <*>.

Exercise. In the definition of <*>, should we check whether k is full before evaluating a k'?

Exercise. We developed two mechanisms — BoundedList and SmartValidation, which seem to do about the same thing on different levels. Would any one of these two mechanisms suffice to achieve both our goals, user experience and efficiency, when there are many errors being reported?

Exercise. If the SmartValidation applicative was based on ordinary lists instead of difference lists, would we be less or more likely to run into the quadratic complexity problem compared to simple Validation?

Conclusion

Although the Validation applicative is known among Haskellers, the need to limit the number of errors it produces is rarely (if ever) discussed. Implementing an applicative functor that limits the number of errors and avoids doing extra work is somewhat tricky. Thus, I am happy to share my solution and curious about how other people have dealt with this problem.

Safe concurrent MySQL access in Haskell

Fri, 17 Apr 2015 20:00:00 +0000

Update (2016-10-26). The mysql package has got a new maintainer, Paul Rouse. He merged my changes that address Issue 1 described below in mysql-0.1.2. The other issues are still relevant, though.

mysql, Bryan O’Sullivan’s low-level Haskell bindings to the libmysqlclient C library, powers a few popular high-level MySQL libraries, including mysql-simple, persistent-mysql, snaplet-mysql-simple, and groundhog-mysql.

Most users do not suspect that using mysql as it stands concurrently is unsafe.

This article describes the issues and their solutions.

Issue 1: unsafe foreign calls

As of version 0.1.1.8, mysql marks many of its ffi imports as unsafe. This is a common trick to make these calls go faster. In our case, the problem with unsafe calls is that they block a capability (that is, an OS thread that can execute Haskell code). This is bad for two reasons:

Fewer threads executing Haskell code may result in less multicore utilization and degraded overall performance.
If all capabilities get blocked executing related MySQL statements, they may deadlock.

Here’s a demonstration of such a deadlock:

{-# LANGUAGE OverloadedStrings #-}
import Database.MySQL.Simple
import Control.Concurrent
import Control.Concurrent.STM
import Control.Applicative
import Control.Monad
import Control.Exception

main = do
  tv <- atomically $ newTVar 0
  withConn $ \conn -> do
    mapM_ (execute_ conn)
      [ "drop table if exists test"
      , "create table test (x int)"
      , "insert into test values (0)"
      ]
    
  forM_ [1..2] $ \n -> forkIO $ withConn $ \conn -> (do
    execute_ conn "begin"
    putStrLn $ show n ++ " updating"
    execute_ conn "update test set x = 42"
    putStrLn $ show n ++ " waiting"
    threadDelay (10^6)
    execute_ conn "commit"
    putStrLn $ show n ++ " committed"
    ) `finally`
    (atomically $ modifyTVar tv (+1))

  atomically $ check =<< (>=2) <$> readTVar tv
  where
    withConn = bracket (connect defaultConnectInfo) close

If you run this with stock mysql-0.1.1.8, one capability (i.e. without +RTS -Nx), and either threaded or non-threaded runtime, you’ll see:

1 updating
1 waiting
2 updating
1 committed
test: ConnectionError {
  errFunction = "query",
  errNumber = 1205,
  errMessage = "Lock wait timeout exceeded; try restarting transaction"}

Here’s what’s going on:

Both threads are trying to update the same row inside their transactions;
MySQL lets the first update pass but blocks the second one until the first update committed (or rolled back);
The first transaction never gets a chance to commit, because it has no OS threads (capabilities) to execute on. The only capability is blocked waiting for the second UPDATE to finish.

The solution is to patch mysql to mark its ffi calls as safe (and use the threaded runtime). Here’s what would happen:

To compensate for the blocked OS thread executing the second UPDATE, the GHC runtime moves the capability to another thread (either fresh or drawn from a pool);
The first transaction finishes on this unblocked capability;
MySQL then allows the second UPDATE to go through, and the second transaction finishes as well.

Issue 2: uninitialized thread-local state in libmysqlclient

To quote the docs:

When you call mysql_init(), MySQL creates a thread-specific variable for the thread that is used by the debug library (among other things). If you call a MySQL function before the thread has called mysql_init(), the thread does not have the necessary thread-specific variables in place and you are likely to end up with a core dump sooner or later.

Here’s the definition of the thread-local state data structure, taken from mariadb-10.0.17:

struct st_my_thread_var
{
  int thr_errno;
  mysql_cond_t suspend;
  mysql_mutex_t mutex;
  mysql_mutex_t * volatile current_mutex;
  mysql_cond_t * volatile current_cond;
  pthread_t pthread_self;
  my_thread_id id;
  int volatile abort;
  my_bool init;
  struct st_my_thread_var *next,**prev;
  void *keycache_link;
  uint  lock_type; /* used by conditional release the queue */
  void  *stack_ends_here;
  safe_mutex_t *mutex_in_use;
#ifndef DBUG_OFF
  void *dbug;
  char name[THREAD_NAME_SIZE+1];
#endif
};

This data structure is used by both server and client code, although it seems like most of these fields are used by the server, not client (with the exception of the dbug thing), which would explain why Haskellers have gotten away with not playing by the rules so far. However:

I am not an expert, and I spent just about 20 minutes grepping the codebase. Am I sure that there’s no code path in the client that accesses this? No.
Am I going to ignore the above warning and bet the stability of my production system on MySQL/MariaDB devs never making use of this thread-local state? Hell no!

What should we do to obey the rules?

First, make threads which work with MySQL bound, i.e. launch them with forkOS instead of forkIO. Otherwise, even if an OS thread is initialized, the Haskell thread may be later scheduled on a different, uninitialized OS thread.

If you create a connection in a thread, use it, and dispose of it, then using a bound thread should be enough. This is because mysql’s connect calls mysql_init, which in turn calls mysql_thread_init.

However, if you are using a thread pool or otherwise sharing a connection between threads, then connect may occur on a different OS thread than a subsequent use. Under this scenario, every thread needs to call mysql_thread_init prior to other MySQL calls.

Issue 3: non-thread-safe calls

The mysql_library_init function needs to be called prior to any other MySQL calls. It only needs to be called once per process, although it is harmless to call it more than once.

It is called implicitly by mysql_init (which is in turn called by connect). However, this function is documented as not thread-safe. If you connect from two threads simultaneously, bad or unexpected things can happen.

Also, if you are calling mysql_thread_init as described above, it should be called after mysql_library_init.

This is why it is a good idea to call mysql_library_init in the very beginning, before you spawn any threads.

Using a connection concurrently

This is not specific to the Haskell bindings, just something to be aware of:

You should not use the same MySQL connection simultaneously from different threads.

First, the docs explicitly warn you about that:

Multiple threads cannot send a query to the MySQL server at the same time on the same connection

(there are some details on this in case you are interested)

Second, the MySQL wire protocol is not designed to multiplex several communication «threads» onto the same TCP connection (unlike, say, AMQP), and trying to do so will probably confuse both the server and the client.

Example

Here is, to the best of my knowledge, a correct example of concurrently accessing a MySQL database. The example accepts request at http://localhost/key and looks up that key in a MySQL table.

It needs to be compiled against my fork of mysql, which has the following changes compared to 0.1.1.8:

Unsafe calls are marked as safe (the patch is due to Matthias Hörmann);
mysql_library_init and mysql_thread_init are exposed under the names initLibrary and initThread.

(How to use a fork that is not on hackage? For example, through a stackage snapshot.)

{-# LANGUAGE OverloadedStrings, RankNTypes #-}
import Network.Wai
import qualified Network.Wai.Handler.Warp as Warp
import Network.HTTP.Types
import qualified Database.MySQL.Base as MySQL
import Database.MySQL.Simple
import Control.Exception (bracket)
import Control.Monad (void)
import Control.Concurrent (forkOS)
import qualified Data.Text.Lazy.Encoding as LT
import Data.Pool (createPool, destroyAllResources, withResource)
import Data.Monoid (mempty)
import GHC.IO (unsafeUnmask)

main = do
  MySQL.initLibrary
  bracket mkPool destroyAllResources $ \pool ->
    Warp.runSettings (Warp.setPort 8000 . Warp.setFork forkOSWithUnmask $ Warp.defaultSettings) $
      \req resp -> do
        MySQL.initThread
        withResource pool $ \conn ->
          case pathInfo req of
            [key] -> do
              rs <- query conn "SELECT `desc` FROM `test` WHERE `key` = ?"
                (Only key)
              case rs of
                Only result : _ -> resp $
                  responseLBS
                    ok200
                    [(hContentEncoding, "text/plain")]
                    (LT.encodeUtf8 result)
                _ -> resp e404
            _ -> resp e404

  where
    mkPool = createPool (connect defaultConnectInfo) close 1 60 10
    e404 = responseLBS notFound404 [] mempty
    forkOSWithUnmask :: ((forall a . IO a -> IO a) -> IO ()) -> IO ()
    forkOSWithUnmask io = void $ forkOS (io unsafeUnmask)

The forkWithUnmask business is only an artifact of the way warp spawns threads; normally a simple forkOS would do. On the other hand, this example shows that in the real world you sometimes need to make an extra effort to have bound threads. Even warp got this feature only recently.

Note that this isn’t the most efficient implementation, since it essentially uses OS threads instead of lightweight Haskell threads to serve requests.

On destructors

The *_init functions allocate memory, so there are complementary functions, mysql_thread_end and mysql_library_end, which free that library.

However, you probably do not want to call them. Here’s why.

Most multithreaded Haskell programs have a small numbers of OS threads managed by the GHC runtime. These threads are also long-lived. Trying to free the resources associated with those threads won’t give much, and not doing so won’t do any harm.

Furthermore, suppose that you still want to free the resources. When should you do so?

Naively calling mysql_thread_end after serving a request would be wrong. It is only the lightweight Haskell thread that is finishing. The OS thread executing the Haskell thread may be executing other Haskell threads at the same time. If you suddenly destroy MySQL’s thread-local state, the effect on other Haskell threads would be the same as if you didn’t call mysql_thread_init in the first place.

And calling mysql_library_end without mysql_thread_end makes MySQL upset when it sees that not all threads have ended.

References

Examples of monads in a dynamic language

Sun, 22 Feb 2015 20:00:00 +0000

In Monads in dynamic languages, I explained what the definition of a monad in a dynamic language should be and concluded that there’s nothing precluding them from existing. But I didn’t give an example either.

So, in case you are still wondering whether non-trivial monads are possible in a dynamic language, here you go. I’ll implement a couple of simple monads — Reader and Maybe — with proofs.

And all that will take place in the ultimate dynamic language — the (extensional) untyped lambda calculus.

The definitions of the Reader and Maybe monads are not anything special; they are the same definitions as you would write, say, in Haskell, except Maybe is Church-encoded.

What I find fascinating about this is that despite the untyped language, which allows more things to go wrong than a typed one, the monad laws still hold. You can still write monadic code and reason about it in the untyped lambda calculus in the same way as you would do in a typed language.

Reader

return x = λr.x
a >>= k  = λr.k(ar)r

Left identity

return x >>= k
  { inline return }
  = λr.x >>= k
  { inline >>= }
  = λr.k((λr.x)r)r
  { β-reduce }
  = λr.kxr
  { η-reduce }
  = kx

Right identity

a >>= return
  { inline return }
  = a >>= λx.λr.x
  { inline >>= }
  = λr.(λx.λr.x)(ar)r
  { β-reduce }
  = λr.ar
  { η-reduce }
  = a

Associativity

a >>= f >>= g
  { inline 1st >>= }
  = λr.f(ar)r >>= g
  { inline 2nd >>= }
  = λr.g((λr.f(ar)r)r)r
  { β-reduce }
  = λr.g(f(ar)r)r

a >>= (λx. f x >>= g)
  { inline 2nd >>= }
  = a >>= λx.λr.g((fx)r)r
  { inline 1st >>= }
  = λr.(λx.λr.g(fxr)r)(ar)r
  { β-reduce }
  = λr.g(f(ar)r)r

Maybe

return x = λj.λn.jx
a >>= k  = λj.λn.a(λx.kxjn)n

Left identity

return x >>= k
  { inline return }
  = λj.λn.jx >>= k
  { inline >>= }
  = λj.λn.(λj.λn.jx)(λx.kxjn)n
  { β-reduce }
  = λj.λn.kxjn
  { η-reduce }
  = kx

Right identity

a >>= return
  { inline return }
  = a >>= λx.λj.λn.jx
  { inline >>= }
  = λj.λn.a(λx.(λx.λj.λn.jx)xjn)n
  { β-reduce }
  = λj.λn.a(λx.jx)n
  { η-reduce }
  = λj.λn.ajn
  { η-reduce }
  = a

Associativity

a >>= f >>= g
  { inline 1st >>= }
  = (λj.λn.a(λx.fxjn)n) >>= g
  { inline 2nd >>= }
  = (λj.λn.(λj.λn.a(λx.fxjn)n)(λx.gxjn)n)
  { β-reduce }
  = λj.λn.a(λx.fx(λx.gxjn)n)n

a >>= (λx. f x >>= g)
  { inline 2nd >>= }
  = a >>= (λx.λj.λn.fx(λx.gxjn)n)
  { inline 1st >>= }
  = λj.λn.a(λx.(λx.λj.λn.fx(λx.gxjn)n)xjn)n
  { β-reduce }
  = λj.λn.a(λx.fx(λx.gxjn)n)n

Foldable, Traversable, and parametricity

Thu, 12 Feb 2015 20:00:00 +0000

The Foldable-Traversable proposal (aka FTP) has spawned a lot of debate in the Haskell community.

Here I want to analyze the specific concern which Ben Moseley raised in his post, FTP dangers.

Ben’s general point is that more polymorphic (specifically, ad-hoc polymorphic, i.e. using type classes) functions are less readable and reliable than their monomorphic counterparts.

On the other hand, Tony Morris and Chris Allen argue on twitter that polymorphic functions are more readable due to parametricity.

Is that true, however? Are the ad-hoc generalized functions more parametric than the monomorphic versions?

@shebang @dibblego is (a -> b) -> [a] -> [b] more parametric than Functor f => (a -> b) -> f a -> f b ?
— Je Suis Petit Gâteau (@bitemyapp) February 12, 2015

Technically, the Functor-based type is more parametric. A function with type (a -> b) -> [a] -> [b] is something like map, except it may drop or duplicate some elements. On the other hand, Functor f => (a -> b) -> f a -> f b has to be fmap.

But this is a trick question! The first thing we see in the code is the function’s name, not its type. What carries more information, map or fmap? (Assuming both come from the current Prelude.) Certainly map. When fmap is instantiated at the list type, it is nothing more than map. When we see fmap, we know that it may or may not be map. When we see map, we know it is map and nothing else.

The paradox is that there are more functions with map’s type than fmap’s, but there are more functions with fmap’s name than map’s. Even though fmap is more parametric, that doesn’t win us much.

Nevertheless, is there a benefit in using more parametric functions in your code? No. If it were true, we’d all be pumping our code with «parametricity» by writing id 3 instead of 3. You can’t get more parametric than id.

Merely using parametric functions doesn’t make code better. Parametricity may pay off when we’re defining polymorphic parametric functions in our code instead of their monomorphic instantiations, since parametric types are more constrained and we’re more likely to get a compile error should we do anything stupid.

(It won’t necessarily pay off; the type variables and constraints do impose a tax on types’ readability.)

But if you have an existing, monomorphic piece of code that works with lists, simply replacing Data.List functions with Data.Foldable ones inside it, ceteris paribus, will not make your code any safer or more readable.

Dealing with broken Haskell packages

Mon, 09 Feb 2015 20:00:00 +0000

As we approach the release of GHC 7.10, there is a new wave of Haskell packages that require trivial fixes to build with the new versions of the compiler and standard libraries, but whose authors/maintainers are not around to apply the fixes. This is especially annoying when there is a pull request on GitHub, and all the maintainer would have to do is to press the green Merge button, and upload the new version on hackage.

If you are a responsible maintainer and don’t want this to happen to your packages in the future, you should appoint backup maintainers for your packages.

But what if you are a user of a package that remains broken on hackage, even though a fix is available? Here I review several ways to deal with this problem, including the new and promising Stackage snapshots.

Building the package locally

If all you care about is to get something installed locally (be it the broken package itself, or something that directly or indirectly depends on it), you can install the fixed version locally.

Non-sandboxed way

Check out the repository or branch with the fix, and cabal-install it:

% git clone -b ghc710 https://github.com/markus1189/feed.git
% cabal install ./feed

(I prepend ./ to make sure cabal understands that I mean the directory, and not the package name on hackage.)

Sandboxed way

If you’re installing in the sandbox, then you can use add-source (although the non-sandboxed version will work in this case, too):

% git clone -b ghc710 https://github.com/markus1189/feed.git
% cabal sandbox add-source feed
% cabal install whatever-you-needed

If the package whatever-you-needed has feed among its transitive dependencies, cabal will automatically install it from the add-source’d directory.

Limitations

This approach doesn’t work well if:

You are a maintainer of a package that depends on the broken library. It’s hard to ask your users to check out and build the fixed version by hand.
You work on an in-house application that your coworkers should be able to build, for the same reason.

Forking the package

You cannot upload the fixed version of a package on hackage bypassing the maintainer. However, you can upload it under a new name. This works well if you are a maintainer of a package that directly depends on the broken package, because you can easily make your package depend on your fork.

Examples of this are tasty depending on regex-tdfa-rc (a fork of regex-tdfa) and tasty-golden depending on temporary-rc (a fork of temporary).

Limitations

This doesn’t work well if there’s a chain of dependencies leading from your package to the broken one. You have to either persuade the other maintainer(s) to depend on your fork or fork the entire chain.
If the broken package becomes actively developed again, you need to either move back to using it or backport the bugfixes from it to your fork. (I only fork packages when I find this scenario unlikely.)
Other packages that depend on the broken package won’t automatically get fixed.
Some people get upset when you fork packages.

Stackage snapshots

Instead of uploading the fixed version to hackage (which you can’t), you can upload it to Stackage instead, by creating a custom snapshot.

The procedure is described in Experimental package releases via Stackage Server. You create four files:

The fixed tarball (produced by cabal sdist). You probably want to bump the package’s version, so that it doesn’t conflict with the version already on hackage.
Two text files: desc and slug. The first one contains a human-readable description of the snapshot; the second contains an id that will become part of the snapshot’s URL.
A text file with the packages to be copied directly from hackage. For the purpose of this article, you probably want to leave this file empty. (I don’t know if it’s ok not to include it at all.)

Then you pack these four files into a tarball (that’s right, it’ll be a tarball with a tarball inside) and upload to stackage (after registering, if you haven’t registered before).

The outcome will be a custom hackage-like repository which will contain the single version of a single package — the one you’ve uploaded. (Of course, you can include multiple packages or versions if you like.)

The Stackage website will give you the remote-repo line that you can put into your cabal.config along with the hackage or stackage repos that are already there.

In contrast to building packages locally, you can easily tell your users or coworkers to add that repo as well.

Limitations

If the new hackage release of the broken package will get the same version number as your stackage version, there will be a conflict. (I actually don’t know what happens in that case; my guess is that cabal will silently pick one of the two available versions.)
If the package you maintain (which depends on the broken package) is a small one, or is deep down the dependency chain, it may be hard to tell your users to add the repository. If, on the other hand, you maintain a major web framework or other such thing, it would probably work.

Taking over a package

There’s a procedure for taking over a package described on the wiki. You’ll need to contact the current maintainer; wait an indefinite amount of time (there’s no consensus about it; estimates vary from 2 weeks to 6-12 months); ask again on the mailing list and wait again; finally ask Hackage admins to grant you the rights.

Limitations

Since this procedure takes a long time, it’s almost never sufficient by itself, and you’ll need to resort to one of the other strategies until you’re given the upload rights.
It’s not clear how long you actually need to wait.
I find it odd that you need to jump through all these hoops in order to do a service to the community.

Spiral similarity solves an IMO problem

Sat, 31 Jan 2015 20:00:00 +0000

While recovering from a knee surgery, I entertained myself by solving a geometry problem from the last International Mathematical Olympiad. My solution, shown below, is an example of using plane transformations (spiral similarity, in this case) to prove geometric statements.

Problem

(IMO-2014, P4)

Points $P$ and $Q$ lie on side $BC$ of acute-angled triangle $ABC$ such that $\angle PAB=\angle BCA$ and $\angle CAQ = \angle ABC$. Points $M$ and $N$ lie on lines $AP$ and $AQ$, respectively, such that $P$ is the midpoint of $AM$ and $Q$ is the midpoint of $AN$. Prove that the lines $BM$ and $CN$ intersect on the circumcircle of triangle $ABC$.

Solution

Let $\angle BAC = \alpha$.

\[\angle APB = \pi - \angle PAB - \angle PBA = \pi - \angle ACB - \angle CBA = \alpha\]

Let $B_1$ and $C_1$ be such points that $B$ and $C$ are midpoints of $AB_1$ and $AC_1$, respectively.

Consider a spiral similarity $h$ such that $h(B_1)=A$ and $h(B)=C$ (it necessarily exists).

Now we shall prove that $h(M)=N$, i.e. that $h$ transforms the green $\triangle B_1BM$ into the magenta $\triangle ACN$ .

Being a spiral similarity, $h$ rotates all lines by the same angle. It maps $B_1B$ to $AC$, therefore that angle equals $\angle(B_1B, AC)=\pi-\alpha$. (We need to be careful to measure all rotations in the same direction; on my drawing it is clockwise.)

$h(A)=C_1$, since $h$ preserves length ratios. So $h(AM)$ (where $AM$ denotes the line, not the segment) is a line that passes through $h(A)=C_1$. It also needs to be parallel to $BC$, because $\angle (AM,BC)=\pi-\alpha$ is the rotation angle of $h$. $C_1B_1$ is the unique such line ($C_1B_1 \parallel BC$ by the midline theorem).

Since $h(AM)=C_1B_1$ and $h(MB_1)=NA$, \[h(M)=h(AM\cap MB_1)=h(AM)\cap h(MB_1)=C_1B_1\cap NA=N.\]

Now that we know that $h(BM)=CN$, we can deduce that $\angle BZC=\angle(BM,CN)=\pi-\alpha$ (the rotation angle). And because $\angle BAC+\angle BZC=\pi$, $Z$ lies on the circumcircle of $ABC$.

Recognizing lists

Thu, 29 Jan 2015 20:00:00 +0000

import Data.Monoid
import Data.List (isSuffixOf)
import Control.Applicative
import Control.Applicative.Permutation as Perm
import Control.Monad.Trans.State

You are given the following Haskell data type:

data T = A Int | B String

Your task is to write a function

recognize :: [T] -> Bool

that would accept a list iff it contains exactly three elements, of which:

one is an A with a value between 3 and 7
one is an A with an even value
one is a B with a value ending in .hs

The A elements mentioned in conditions (1) and (2) have to be distinct, so [A 4, A 17, B "Prelude.hs"] should not be accepted, even though A 4 satisfies both (1) and (2).

This problem often arises in blackbox testing. You do something to a system; you wait for some time while gathering all events that the system generates. Then you want to check that the system generated all the events it had to generate, and no extra ones. However, the order of events may be non-deterministic. If the system concurrently makes a web request, updates a database and sends a message to a message bus, the order in which these events will be observed by your test suite is arbitrary.

A similar problem arises in input validation.

Interface

We want to solve this problem in a general and scalable way. What would an interface to such solution look like? There are two principal options.

One is a simple function

recognize :: [t -> Bool] -> [t] -> Bool

that takes individual predicates and combines them into a recognizer.

Another is to split the process into the build and use phases by the means of an intermediate data type:

data Parser t
instance Monoid Parser

toParser :: (t -> Bool) -> Parser t
runParser :: Parser t -> [t] -> Bool

In this approach, you convert individual recognizers to monoidal values, combine those values and extract the result.

Semantically, the two are equivalent: if you have one of these interfaces available, you can implement the other one. In a way, this is a consequence of lists being the free monoid, but writing this down explicitly is a good exercise nevertheless.

The advantages of a build-use interface are:

When the cost of the build phase is non-trivial, it can be amortized over multiple uses and/or incremental builds. (This won’t be the case here.)
The use of a specialized type is generally good for the type system hygiene.

We’ll start with the build-use interface because I find it more instructive.

The Parser

To build the monoidal parser, we’ll use the action-permutations library. You may remember it from the article on JSON parsing.

newtype Parser t = Parser { getParser :: Perms (StateT [t] Maybe) () }

StateT [t] Maybe a is the familiar type of parser combinators, [t] -> Maybe (a, [t]), that return the left-over input. We wrap it in Perms to achieve the order independence, and then newtype it to convert its Applicative instance to a Monoid one:

instance Monoid (Parser t) where
  mempty = Parser (pure ())
  mappend (Parser a) (Parser b) = Parser $ a *> b

The build and use functions are straightforward:

toParser :: (t -> Bool) -> Parser t
toParser isX = Parser $ Perm.atom $ StateT $ \ts ->
  case ts of
    (t:ts') | isX t -> Just ((), ts')
    _ -> Nothing

runParser :: Parser t -> [t] -> Bool
runParser p ts =
  case (flip execStateT ts . runPerms . getParser) p of
    Just [] -> True
    _ -> False

Usage

First, let’s express our conditions in Haskell code.

isA1, isA2, isB :: T -> Bool

isA1 x
  | A n <- x, 3 <= n, n <= 7 = True
  | otherwise = False

isA2 x
  | A n <- x, even n = True
  | otherwise = False

isB x
  | B s <- x, ".hs" `isSuffixOf` s = True
  | otherwise = False

Combine them together:

parser :: Parser T
parser = (mconcat . map toParser) [isA1, isA2, isB]

(we could replace mconcat . map toParser with foldMap toParser from Data.Foldable)

recognize :: [T] -> Bool
recognize = runParser parser

Now try it out:

ex1 = recognize [A 4, A 17, B "Prelude.hs"]
-- => False
ex2 = recognize [A 4, A 18, B "Prelude.hs"]
-- => True
ex3 = recognize [B "Prelude.hs", A 18, A 4]
-- => True
ex4 = recognize [B "Prelude.hs", A 18, A 4, A 1]
-- => False

Lexical analysis with parser combinators

Fri, 02 Jan 2015 20:00:00 +0000

When writing a programming language parser in Haskell, the usual dilemma is whether to use lexer/parser generators (Alex+Happy), or make a single parser (using a parser combinator library like Parsec) without an explicit tokenizer.

In this article, I’ll explain why you may want to use a separate lexer built with applicative parser combinators, and how you might go about writing one.

Many of the ideas described in this article have been since implemented in lexer-applicative.

Alex

Alex, the Haskell lexer generator, is a nice tool, but it has a weakness. With Alex, you cannot observe the internal structure of a token.

The most common instance of this problem is parsing string interpolation, think "Hello, ${name}!". Such a string is a single token, yet it has some internal structure to it. To analyze this with Alex, you’ll probably need to have two different lexers, and run one inside the other.

Another instance is date literals (useful for many DSLs), such as 2015-01-02 (or d"2015-01-02" to avoid confusion with infix subtraction.) You can recognize such a literal with an Alex regular expression $d+\-$d+\-$d+. However, you won’t be able to get hold of the individual $d+ pieces — Alex regexps lack capture groups. So you’ll need to use a separate date parsing function to parse the same string second time.

Parsec

On the other hand, we have parser combinators that solve this problem nicely. You define parsers for numbers; then you glue them with dashes, and you get a parser for dates.

However, using Parsec (or similar) without a separate lexer is not the best choice:

Dealing with whitespace and comments is awkward in the parser; you need to wrap everything in a token combinator. (If you decide to do that, at least use a free applicative functor to ensure that you don’t forget to consume that whitespace).
Separating parsing into lexical and syntax analysis is just a good engineering practice. It reduces the overall complexity through «divide and conquer». The two phases usually have well-defined responsibilities and a simple interface — why not take advantage of that?
If a language needs the maximal munch rule, it’s hard to impossible to encode that with Parsec or similar libraries.
Tokens enable the syntax analyzer to report better errors. This is because you can tell which token you didn’t expect. In a Char-based Parsec parser, you can only tell which character (or an arbitrary number of characters) you didn’t expect, because you don’t know what constitutes a token.
Potential performance considerations. If a parser has to try several syntax tree alternatives, it reparses low-level lexical tokens anew every time. From this perspective, the lexical analyzer provides a natural «caching layer».

regex-applicative

My regex-applicative library provides applicative parser combinators with regexp semantics. We can write a simple lexer on top of it that would give us the benefits of both Alex and Parsec.

{-# LANGUAGE OverloadedStrings, OverloadedLists #-}

import qualified Data.Text as T
import qualified Data.HashSet as HS
import Text.Regex.Applicative
import Text.Regex.Applicative.Common (decimal)
import Data.Char
import Data.Time

For example, here’s a parser for dates. (For simplicity, this one doesn’t check dates for validity.)

pDate :: RE Char Day
pDate = fromGregorian <$> decimal <* "-" <*> decimal <* "-" <*> decimal

And here’s a parser for templates — strings that may include interpolated variables and escaping.

pTemplate :: RE Char [Either T.Text T.Text] -- Left = text, Right = variable
pTemplate = "\"" *> many piece <* "\""
  where
    -- piece of text or interpolated variable
    piece = 
      (Left . T.pack <$> some ch) <|>
      (Right <$> var)

    -- interpolated variable
    var = "${" *> pVar <* "}"

    -- normal character, plain or escaped
    ch =
      psym (\c -> not $ c `HS.member` escapable) <|>
      ("\\" *> psym (\c -> c `HS.member` escapable))

    -- set of escapable characters
    escapable = ['"', '\\', '$']

    pVar = T.pack <$> some (psym isAlpha)

Individual parsers are merged into a single pToken parser:

data Token
   = Template [Either T.Text T.Text]
   | Date Day
-- | ...

pToken :: RE Char Token
pToken =
   (Template <$> pTemplate) <|>
   (Date <$> pDate)
-- <|> ...

Finally, a simple tokenizing function might look something like this:

tokens :: String -> [Token]
tokens "" = []
tokens s =
  let re = (Just <$> pToken) <|> (Nothing <$ some (psym isSpace)) in
  case findLongestPrefix re s of
    Just (Just tok, rest) -> tok : tokens rest
    Just (Nothing, rest) -> tokens rest -- whitespace
    Nothing -> error "lexical error"

The resulting token list can be further parsed with Parsec or Happy.

This simple function can be extended to do a lot more:

Track position in the input string, use it for error reporting, include it into tokens
Consume input incrementally
Get feedback from the syntax analyzer, and, depending on the syntactic context, perform different lexical analyses. This is useful for more complex features of programming languages, such as the off-side rule in Haskell or arbitrarily-nested command substitution and «here documents» in POSIX Shell. (This is another thing Alex cannot do, by the way.)

Denotational design does not work

Wed, 31 Dec 2014 20:00:00 +0000

Conal Elliott in his paper Denotational design with type class morphisms, as well as in the recent podcast, advocates denotational design as a principle of software design. Unfortunately, in my experience it never works for realistically complex problems.

On simplicity

First, I’ll formulate Conal’s approach as I understand it. For any given entity of your model, you should come up with a simple mathematical object — the denotation — that faithfully represents that entity.

The implementation of the type may vary, presumably to maximize its runtime efficiency, but it should not expose any more information than the chosen denotation has. That is considered an «abstraction leak». Conal specifically talks about that in the podcast (31m50s, for example).

Here I need to stress an important but subtle point in Conal’s principle: simplicity. You only follow Conal’s principle if you find a simple denotation, not just any denotation.

This point is important because without it any design is denotational design, trivially. Universal algebra tells us that for any set of operations and any (non-contradictory) laws about them there exists a model that satisfies these laws. For any Haskell module, we can interpret its types in the set theory (or a more complex domain if needed), and call that our denotation.

But that’s not what Conal is after. His approach is interesting exactly because he argues that it is possible to find simple denotations. This subtle point makes Conal’s approach simultaneously attractive and unrealistic. I’ll demonstrate this with two examples from my own work experience.

DSLs

At Barclays I worked on FPF, an embedded domain-specific language for describing financial instruments. In his paper, Conal shows how a denotation for such a DSL can quickly grow in complexity when requirements change. When variables and errors are introduced, the denotation changes from a to Env -> Result a. Still, this is a very simple DSL that only supports evaluation.

In reality, the main reason people make DSLs instead of using general-purpose languages is the ability to analyze DSL programs. One important feature of FPF is that it could pretty-print a program into a nice PDF. That poses an obvious problem — not every two semantically equivalent programs (under the interpretation semantics) result in equally nice PDFs. Inlining is a semantically sound transformation, but when our users get PDFs with all the definitions inlined, they get angry.

Sure, we could say that now our denotation becomes the domain product (Env -> Result a, String), where String is the pretty printer output. But in reality we have a dozen different analyses, and most of them are not expressible in terms of each other, or any single simple model. They also do not satisfy many laws. For instance, one day a user (quant or trader) could come and tell us that the barrier classifier should classify two mathematically equivalent expressions as different barriers because those expressions follow certain conventions. And even though the quant is mathematically inclined, denotations and type class morphism would be the last thing he wants to hear about in response to his feature request.

So, in practice, the best denotation for the DSL expressions was the AST itself. Which, according to my interpretation of Conal’s principles, is not an example of a denotational design, but a failure to apply one.

Machines

At my current job (Signal Vine), I work on a platform for scripted interaction with students via text messages. For every student enrolled in a messaging campaign, we send a message, receive a reply, process it, and the cycle repeats.

This is very similar to FRP; perhaps not the FRP Conal prefers (in the podcast he stresses the importance of continuous functions as opposed to events), but the kind of discrete FRP that Justin Le models with Mealy machines.

So it would seem that I should model a student as

newtype Student = Student (InboundSMS -> (OutboundSMS, Student))

That would be an exemplary case of denotational design. But that would be far from our customers’ needs. Every student has a set of profile variables that are filled when the student responds to a text, and our customers (counselors who work with that student) want to see those variables. They also want to see which messages were sent, what the student’s replies were, and even what messages will be sent to the student in the future. These requirements defeat the attempt to model a student in a simple, abstract way. Instead, I need to store all the information I have about the student because sooner or later I’ll need to expose that information to the user.

Conclusions

Denotational design is a very neat idea, but I believe that it only works in simple cases and when requirements are static. In real-world commercial programming, it breaks for two main reasons:

Users often want maximum insight into what’s going on, and you need to break the abstraction to deliver that information.
Requirements change, and an innocent change in requirements may lead to a drastic change and increase in complexity of the denotation.

It is certainly useful to think about denotations of your entities in specific, simple contexts (like the evaluation semantics for a DSL); such thought experiments may help you better understand the problem or even find a flaw in your implementation.

But when you are implementing the actual type, your best bet is to create an algebraic data type with all the information you have, so that when you need to extend the interface (or «leak the abstraction»), it won’t cause you too much pain.

Taking advantage of type synonyms in monad-control

Fri, 26 Dec 2014 20:00:00 +0000

Bas van Dijk has recently released monad-control-1.0.0.0, the main purpose of which is to replace associated data types with associated type synonyms. The change caused minor breakages here and there, so people might wonder whether and why it was worth it. Let me show you a simple example that demonstrates the difference.

Let’s say we are writing a web application. wai defines an application as

type Application =
  Request ->
  (Response -> IO ResponseReceived) ->
  IO ResponseReceived

Our web app will need a database connection, which we’ll pass using the ReaderT transformer:

type ApplicationM m =
  Request ->
  (Response -> m ResponseReceived) ->
  m ResponseReceived

myApp :: ApplicationM (ReaderT DbConnection IO)

However, warp can only run an Application, not ApplicationM:

run :: Port -> Application -> IO ()

Can we build runM :: Port -> ApplicationM m -> m () on top of the simple run function? Solving the problems like this one is exactly the purpose of monad-control.

Here’s how such a function might look like:

runM
  :: (MonadBaseControl IO m)
  => Port -> ApplicationM m -> m ()
runM port app = do
  liftBaseWith $ \runInIO ->
    run port $ \req ack -> runInIO $ app req (liftBase . ack)

What’s going on here? liftBaseWith, like liftM or liftBase, allows to run a primitive monadic action in a complex monad stack. The difference is that it also gives us a function, here named runInIO, which lets to “lower” complex actions to primitive ones. Here we use runInIO to translate the return value of our app, m (), into a basic IO () value that the run function can digest.

All is well, except…

Could not deduce (StM m ResponseReceived ~ ResponseReceived)
Expected type: IO ResponseReceived
  Actual type: IO (StM m ResponseReceived)
Relevant bindings include
  runInIO :: RunInBase m IO
In the expression: runInIO $ app req (liftBase . ack)

The type of runInIO is forall a . m a -> IO (StM m a) (a.k.a. RunInBase m IO), while we would like a simple forall a . m a -> IO a. The purpose of StM is to encompass any “output effects”, such as state or error.

In our case, we don’t have any “output effects” (nor would we be allowed to), so StM (ReaderT DbConnection IO) ResponseReceived is really isomorphic to ResponseReceived.

In monad-control 0.x, StM used to be an associated data family, and its constructors for the standard monad transformers were hidden. Even though we knew that the above two types were isomorphic, we still couldn’t resolve the error nicely.

Not anymore! Since in monad-control 1.0 StM is an associated type synonym, StM (ReaderT DbConnection IO) ResponseReceived and ResponseReceived are not just hypothetically isomorphic; they are literally the same type. After we add the corresponding equality constraint to runM

runM
  :: (MonadBaseControl IO m, StM m ResponseReceived ~ ResponseReceived)
  => Port -> ApplicationM m -> m ()

our app compiles!

This example is not just an isolated case. The general problem with monad-control is that it is all too easy to discard the output effects as Edward Yang shows.

Monads for which StM m a ~ a provide a “safe subset” of monad-control. Let’s call them “stateless” monads.

Previously, it was hard to tell apart stateless and stateful monads because the output effects or absence thereof hid behind the opaque StM data family.

Now not only is it transparent when the output effects are absent, but we can actually encode that fact right in the type system! As an example, Mitsutoshi Aoe and I are experimenting with a safe lifted async module.

One may wonder if this subset is too boring, since it only includes monads that are isomorphic to a reader transformer over the base monad. While that is technically true, there are a lot of things you can do with a reader. The ZoomT and CustomWriterT transformers that I described in another article, as well as the Proxied transformer they’re based upon, are stateless and thus safe to use with monad-control.

Another common example of a stateless monad transformer is a ReaderT holding a reference (such as IORef, TVar, or MVar). This is commonly used to maintain a shared state in multithreaded server applications, so it is fortunate that we can fork threads and catch exceptions inside such monads.

Extensible effects: abstracting from the transformer

Sat, 06 Dec 2014 20:00:00 +0000

In Type-based lift, we saw a way to lift monadic actions automatically to the right layer of a multilayer monad transformer stack, based only on the types involved.

Namely, we defined a closed type family

type family Find (t :: (* -> *) -> (* -> *)) (m :: * -> *) :: Nat where
  Find t (t m) = Zero
  Find t (p m) = Suc (Find t m)

that computes the type-level index of the layer t in the stack m. Such an index can then be used to construct an appropriate lifting function of type t n a -> m a.

This works well as a shortcut, so instead of writing lift, or lift . lift, or lift . lift . lift, we can write magicLift, and let it figure out how far to lift.

However, the lifting is expressed in terms of specific transformers, and not the effects they can handle. For example, a stateful computation may target the strict State monad or the lazy one, but not both, because they are implemented by distinct types.

Let’s fix this!

CanDo

To know which effects can be handled by each transformer, we’ll introduce a new type family, CanDo:

type family CanDo (m :: (* -> *)) (eff :: k) :: Bool

Now we need to modify the Find family to find the first (top-most) layer for which the CanDo predicate will return True. Since on the type level we don’t have lambdas and case statements, doing so is a bit cumbersome but still possible:

type family MapCanDo (eff :: k) (stack :: * -> *) :: [Bool] where
  MapCanDo eff (t m) = (CanDo (t m) eff) ': MapCanDo eff m
  MapCanDo eff m = '[ CanDo m eff ]

type family FindTrue (bs :: [Bool]) :: Nat where
  FindTrue (True ': t) = Zero
  FindTrue (False ': t) = Suc (FindTrue t)

type Find eff (m :: * -> *) = FindTrue (MapCanDo eff m)

Next, we need to introduce dummy types denoting effects, and relate them to transformers:

import qualified Control.Monad.Trans.State.Lazy as SL
import qualified Control.Monad.Trans.State.Strict as SS

data EffState (s :: *)
data EffWriter (w :: *)
data EffReader (e :: *)

type instance CanDo (SS.StateT s m) eff = StateCanDo s eff
type instance CanDo (SL.StateT s m) eff = StateCanDo s eff

type family StateCanDo s eff where
  StateCanDo s (EffState s) = True
  StateCanDo s (EffReader s) = True
  StateCanDo s (EffWriter s) = True
  StateCanDo s eff = False

As we see, the relationship between effects and transformers is many-to-many. A single effect can be implemented by multiple transformers, and a single transformer can implement multiple effects.

Should StateT implement EffReader?

It’s not only for demonstration that I made StateT implement the EffReader effect. One can view EffReader (and EffWriter) computations as a subclass of all stateful computations

Suppose that you have two computations with the same state that you want to execute sequentially, but one of them only needs read access to the state. Why not express that fact in its type?

Other cool tricks

Here are a couple of cool (and useful!) tricks that are possible with extensible effects. I will only describe what they do and not how they’re implemented; you can find all code in the repo.

Zooming

newtype ZoomT big small m a = ...

ZoomT handles EffState small effects by transforming them to EffState big effects. To enable this, we must supply a lens from big into small:

runZoom
  :: Lens big small 
  -> ZoomT big small m a
  -> m a

Compared to traditional zooming (as in lens), where we can only focus on a single state at a time, here we can apply many ZoomTs stacked on top of each other, thus handling multiple different EffState effects simultaneously.

Custom Writer handlers

The classic use case for a Writer monad is logging. Ironically, the way a writer monad does logging (accumulating log messages in memory) is wrong for almost all purposes, and possible right ways (flushing messages to disk or sending them over the network) are out of reach for it.

newtype CustomWriterT w m a = ...

evalWriterWith
  :: (w -> m ())
  -> CustomWriterT w m a
  -> m a

The idea here is that CustomWriterT handles EffWriter effect by calling the given handler for it — exactly what we wanted!

Rebalancing open source portfolio

Sat, 01 Nov 2014 20:00:00 +0000

Being an open source developer means being an investor. I invest my time in creating, contributing to, and maintaining projects. As an investor, I occasionally need to re-evaluate my portfolio and decide whether it is optimal.

The result of a recent such re-evaluation is that I’ve decided to stop my active involvement in the haskell-suite family of projects.

This decision wasn’t easy. Being a maintainer of haskell-src-exts was a unique experience. During the last two years, 24 people have contributed to the project, which is amazing, given its complexity. During ICFP’14, I met with 4 or 5 contributors and many users of haskell-src-exts. People would approach me in the hallway and ask when the new version would be released or a certain feature added.

Leaving haskell-names behind is also sad. It hasn’t seen as many uses as haskell-src-exts (which has always been puzzling — you can’t do much with a bare parser. But apparently folks prefer to roll their own, incomplete and buggy, name resolution.) Still, it’s used by fay, and I feel bad about letting the fay devs down.

So, why did I choose to stop investing in the haskell suite?

I have much less spare time these days, so I have to drop something.
I became involved with the haskell suite because of my interest in developer tools for Haskell (hasfix, ariadne). I am no longer working on those, and I’m not using any haskell-src-exts-powered tools (hlint, stylish-haskell) myself, which means I don’t get big personal returns on my investment.
The main competitor for the haskell suite is the GHC API. It is now clear to me that we are losing to them in most areas. There is very little hope for the type checker, let alone other parts of the compilation pipeline. The community did a great job in catching up with GHC on the parser side, but we still have less resources than GHC and are bound to lag behind, it seems. On the other side, there has been some work recently (by Alan Zimmerman, in particular) to improve GHC’s AST and catch up with HSE on that front. So, even if I decide to invest in compilers or dev tools in the future, I’m probably going to team up with GHC instead.

What’s going to happen to these projects I’m leaving behind? I’m less worried about haskell-src-exts. Peter Jonsson has been doing some great work on it, and I hope he’ll step in. There’s also Niklas Broberg and all the wonderful contributors.

I’m more worried about haskell-names, haskell-packages, ariadne. If you are willing to take over any of them, let me know (and you can count on my help). Otherwise, I’m going to provide very basic support for them, but nothing more.

Oh, and in case you missed the announcement, I’m dropping support for my test-framework packages, too.

On the other hand (and to end with something positive), I have no intention to stop maintaining tasty in the foreseeable future. Haskell needs at least one decent testing framework, after all! :-) By the way, here’s a cool use case for tasty that Lars Hupel shared for the upcoming edition of HCAR:

Technische Universität München uses Tasty to assess homework submitted for the introductory functional programming course, which is compulsory for second-year Bachelor’s students in computer science. Students can upload their source code to a web application, which compiles them and executes tests (mostly QuickCheck). Graphical reports are generated with ‘tasty-html’ and presented to the students. Grading happens automatically with a custom test reporter, which – roughly speaking – checks that a minimum number of significant tests have passed. This is a huge improvement over the past years, where tests were run by a homegrown framework which offered only textual output. Tasty however is nicely extensible and allows for easy customization to the instructors’ and students’ needs.

Dependent Haskell

Fri, 05 Sep 2014 20:00:00 +0000

Emulating dependent types (and, more generally, advanced type-level programming) has been a hot topic in the Haskell community recently. Some incredible work has been done in this direction: GADTs, open and closed type families, singletons, etc. The plan is to copy even more features to the type level, like type classes and GADTs, and simplify the promotion of value-level functions.

On the other hand, there’s a good deal of scepticism around this idea. «If you want to program like in Agda, why don’t you program in Agda?»

First, libraries. It took us many years to arrive at the state of hackage that is suitable for industrial usage — and we still don’t have satisfactory answers to many problems.

My guess is that it will take at least as long as that for the dependently typed community to arrive at this point — not only because of the smaller community, but also because they will look for even more type-safe solutions, which is naturally a harder problem.

Second, the compiler/runtime. How optimized is the generated code? Is there a profiler? How about concurrency and parallelism? Is there an FFI? How good is the garbage collector?

These questions are especially acute for Idris, which chose to have its own compiler. Code extraction into Haskell or OCaml may be a more viable alternative, but without having several real-world projects that are implemented this way it’s very hard to assess it properly.

No doubt, these (or other) languages will get there sooner or later. But it seems that practical dependently typed programming will become viable in Haskell much sooner than in Agda or Idris.

What you need to know about bracket

Wed, 30 Jul 2014 20:00:00 +0000

The bracket function in Haskell can be used to release resources reliably:

bracket
        :: IO a         -- ^ computation to run first ("acquire resource")
        -> (a -> IO b)  -- ^ computation to run last ("release resource")
        -> (a -> IO c)  -- ^ computation to run in-between
        -> IO c         -- returns the value from the in-between computation

There are a couple of things you should know when using bracket, lest you be surprised by the results.

First, in some cases the release action may not be called. Or it may be called but not given a chance to complete.

Second, the acquire and release action are run with async exception masked, which may not be what you want.

Release guarantee

It is useful to distinguish two different kinds of resources, let’s call them “internal” and “external”.

An example of an internal resource is an MVar that you take (acquire) and put back (release):

withMVar :: MVar a -> (a -> IO b) -> IO b
withMVar m io =
  bracket (takeMVar m) (putMVar m) io

An example of an external resource is a directory that you create (acquire) and remove (release):

withFoo :: IO a -> IO a
withFoo io =
  bracket_ (createDirectory "foo") (removeDirectoryRecursive "foo") io

The release of an internal resource is noticable only within your program, and it only matters as long as your program keeps running. If the program exits, it’s fine not to release an internal resource. An external resource, on the other hand, always needs to be released.

It is impossible to guarantee that external resources are always released. The power may go off, or your program may be terminated with SIGKILL, and there won’t be time to clean up.

Thus, we would like a guarantee that:

internal resources are released if the program keeps running, to avoid memory leaks and deadlocks
external resources are released, assuming the program keeps running or is terminated in a “civil” way

Now let’s see why these promises may be broken and what to do about it.

`bracket` in non-main threads

The main thread is the Haskell thread that executes your main function.

When the main thread finishes, the program finishes, too, without sending exceptions to other threads and giving them any chance to clean up.

Therefore, if your bracket is executed in some other thread, its release action may not run when the program terminates.

This limitation is especially acute in library code, where you don’t know whether you’re running in the main thread or have any way to tie to it.

To avoid this problem, you need to manage your threads. Keep a list of them (or weak pointers to them, see mkWeakThreadId), and give them a heads-up (in the form of an asynchronous exception) before you exit from main.

This problem only affects external resources.

asynchronous exceptions

Even if the release action is called, it may not complete if it is interrupted by an asynchronous exception.

A good example is the withTempDirectory function defined in the package temporary:

withTempDirectory targetDir template =
  Exception.bracket
    (liftIO (createTempDirectory targetDir template))
    (liftIO . ignoringIOErrors . removeDirectoryRecursive)

Bit Connor describes the issue in detail:

This function uses bracket which splits it up into three stages:

“acquire” (create the directory)

“in-between” (user action)

“release” (recursively delete the directory)

Consider the following scenario:

Stage 1 (“acquire”) completes successfully.

Stage 2 (“user action”) places many files inside the temporary directory and completes successfully.

Stage 3 begins: There are many files inside the temporary directory, and they are deleted one by one. But before they have all been deleted, an async exception occurs. Even though we are currently in a state of “masked” async exceptions (thanks to bracket), the individual file delete operations are “interruptible” and thus our mask will be pierced. The function will return before all of the temporary files have been deleted (and of course the temporary directory itself will also remain).

This is not good. “with-style” functions are expected to guarantee proper and complete clean up of their resources. And this is not just a theoretical issue: there is a significant likelihood that the problem can occur in practice, for example with a program that uses a temporary directory with many files and the user presses Ctrl-C.

This problem affects both internal and external resources if the release action includes any interruptible operations.

To prevent the interruption, wrap the release action in uninterruptibleMask.

signals

By default, only the SIGINT signal is turned into a Haskell exception by the GHC RTS.

If your process is terminated by any other signal, the release action won’t run.

To address this, turn signals into exceptions yourself:

import Control.Concurrent (mkWeakThreadId, myThreadId)
import Control.Exception (Exception(..), throwTo)
import Control.Monad (forM_)
import Data.Typeable (Typeable)
import System.Posix.Signals
import System.Mem.Weak (deRefWeak)

newtype SignalException = SignalException Signal
  deriving (Show, Typeable)
instance Exception SignalException

installSignalHandlers :: IO ()
installSignalHandlers = do
  main_thread_id <- myThreadId
  weak_tid <- mkWeakThreadId main_thread_id
  forM_ [ sigHUP, sigTERM, sigUSR1, sigUSR2, sigXCPU, sigXFSZ ] $ \sig ->
    installHandler sig (Catch $ send_exception weak_tid sig) Nothing
  where
    send_exception weak_tid sig = do
      m <- deRefWeak weak_tid
      case m of
        Nothing  -> return ()
        Just tid -> throwTo tid (toException $ SignalException sig)

main = do
  installSignalHandlers
  ...

For the rationale behind this specific collection of signals (which is far from exhaustive), see this pull request by Aleksey Khudyakov.

This problem only affects external resources. It was pointed out to me by Jon Boulle and Mandeep Gill.

References:

Inadvertent mask

You could try to write something like

main = do
  bracket (forkIO myComputation) killThread $ \_ -> do
    ...

The idea here is that if main exits, for whatever reason, you’ll send an exception to the thread you forked, and give it a chance to clean up.

First, this isn’t going to help much because main will exit right after killThread, probably right in the middle of myComputation’s cleanup process. Some kind of synchronisation should be introduced to address this properly. (The price is that your program may not exit promptly when you interrupt it with Ctrl-C.)

There’s another, more subtle issue with the code above. Let’s look at the definition of bracket:

bracket before after thing =
  mask $ \restore -> do
    a <- before
    r <- restore (thing a) `onException` after a
    _ <- after a
    return r

As you see, the before action is run in the masked state. Forked threads inherit the masked state of their parents, so myComputation and all threads spawned by it will unwittingly run in the masked state, unless they do forkIOWithUnmask.

In this simple case, you should just use withAsync from the async package. What about more complex ones?

If you do forking explicitly, then you can write bracket-like code yourself and restore the forked computation. Here’s an example of synchronised cleanup:

main = do
  cleanupFlag <- atomically $ newTVar False
  mask $ \restore -> do
    pid <- forkIO $ restore $ myComputation cleanupFlag
    restore restOfMain
      `finally` do
        killThread pid
        -- wait until myComputation finishes its cleanup
        -- and sets the flag to True
        atomically $ readTVar cleanupFlag >>= check

(You could use an MVar for synchronisation as well.)

And what if forking happens inside some library function that you need to call? In that case, you may want to restore that whole function from the beginning.

Type-based lift

Tue, 15 Jul 2014 20:00:00 +0000

In mtl, the ask method of the MonadReader class will automatically «lift» itself to the topmost ReaderT layer in the stack, which is very convenient, but only works as long as the topmost layer is the one you need. If you have multiple ReaderTs in the stack, you often have to insert manual lifts.

Previously I described why a smarter automatic lifting mechanism is needed to build truly reusable pieces of monadic code without too much boilerplate.

In this article I show two ways to achieve a type-based lift (that is, a lift which takes into account the r of ReaderT r), one relying on IncoherentInstances, the other — on closed type families.

Class-based approach and IncoherentInstances

In Two failed attempts at extensible effects, I wrote that simply removing the fundep from mtl wouldn’t work. This claim was recently disproved by Ben Foppa and his extensible-transformers library.

Why did I think that such an approach wouldn’t work?

{-# LANGUAGE MultiParamTypeClasses, FlexibleInstances, OverlappingInstances #-}
import Control.Monad.Trans.Reader hiding (ask)
import qualified Control.Monad.Trans.Reader as Trans
import Control.Monad.Trans.Class

class MonadReader r m where
  ask :: m r

instance Monad m => MonadReader r (ReaderT r m) where
  ask = Trans.ask

instance (Monad m, MonadReader r m, MonadTrans t) => MonadReader r (t m) where
  ask = lift ask

GHC, when asked to compile something that uses the above instances, will ask you in return to enable the IncoherentInstances extension. My experience with GHC told me that such a request is just a polite way for GHC to say «You’re doing something wrong!», so I immediately dismissed that approach. I had never seen a case where IncoherentInstances would be an acceptable solution to the problem. Well, this one seems to be exactly such a case!

Switching IncoherentInstances on here not only makes the type checker happy, but also makes the code work as expected, at least in the few tests that I tried.

Closed type classes

Intuitively, the reason why GHC needs so many ugly extensions to make the above code work is that we’re trying to simulate a closed type class with an open one.

Our type class is essentially a type-level if operator comparing two types, and its two instances correspond to the two branches of the if operator.

If only we had closed type classes, we could write

import Control.Monad.Trans.Reader hiding (ask)
import qualified Control.Monad.Trans.Reader as Trans
import Control.Monad.Trans.Class

class MonadReader r m where
  ask :: m r

  instance Monad m => MonadReader r (ReaderT r m) where
    ask = Trans.ask

  instance (Monad m, MonadReader r m, MonadTrans t) => MonadReader r (t m) where
    ask = lift ask

(where I put instance declarations inside the class declaration to show that the class is closed).

Alas, GHC 7.8 does not have closed type classes, and I have not even heard of them being developed. All we have is closed type families. Closed type families would let us compute, say, a type-level number showing how far we have to lift a monadic action to reach the right level. They, however, do not allow us to compute a value-level witness — the very lifting function!

Closed type families

Still, it is possible to achieve automatic lifting using closed type families alone. We developed this approach together with Andres Löh at ZuriHac’14.

The main idea is to split the problem into two.

First, we compute the amount of lifting required using a closed type family

-- Peano naturals, promoted to types by DataKinds
data Nat = Zero | Suc Nat

type family Find (t :: (* -> *) -> (* -> *)) (m :: * -> *) :: Nat where
  Find t (t m) = Zero
  Find t (p m) = Suc (Find t m)

Second, assuming we know how far to lift, we can compute the lifting function using an ordinary (open) MPTC:

class Monad m => MonadReaderN (n :: Nat) r m where
  askN :: Proxy n -> m r

instance Monad m => MonadReaderN Zero r (ReaderT r m) where
  askN _ = Trans.ask

instance (MonadTrans t, Monad (t m), MonadReaderN n r m, Monad m)
  => MonadReaderN (Suc n) r (t m)
  where
    askN _ = lift $ askN (Proxy :: Proxy n)

It is important to note that our instances of MonadReaderN are non-overlapping. The instance is uniquely determined by the n :: Nat type parameter.

Finally, we glue the two components together to get a nice ask function:

-- Nice constraint alias
type MonadReader r m = MonadReaderN (Find (ReaderT r) m) r m

ask :: forall m r . MonadReader r m => m r
ask = askN (Proxy :: Proxy (Find (ReaderT r) m))

Problem solved?

Not quite. Both solutions described here do abstract from the position of a monad transformer in the stack, but they do not abstract from the transformer itself. The MonadReader r constraint can only be satisfied with ReaderT r but not, say StateT r. Moreover, a MonadState constraint, defined as

type MonadState s m = MonadStateN (Find (Control.Monad.State.Lazy.StateT s) m) s m

can only be satisfied by the lazy, but not strict, StateT.

I address this issue in the subsequent article.

How to run SQL actions in persistent

Mon, 07 Jul 2014 20:00:00 +0000

When I started writing an application that used persistent to interact with a MySQL database, I decided to put the whole application inside one big SqlPersistM action, and run it once inside main. (To make it clear, this is not a Yesod application; I simply use persistent as a standalone library.)

However, as I learned more about persistent and how it worked, it became clear that this was the wrong way to use persistent. Here’s why.

Problems of one big `SqlPersistM` action

Finalizing transactions

persistent’s SQL layer treats an SqlPersistT action as a single transaction. Thus, until you run the action, the transaction is not committed. Obviously, this is an issue for any long-running server application.

You could work around this by calling transactionSave manually. Now you have a different but related problem…

Overlapping transactions

Normally a single SQL connection can participate in just one SQL transaction. (There are probably exceptions to this rule which I am not aware of, but this is how it happens unless you do something special.)

Thus, assuming your application is multithreaded, you’ll end up committing other threads’ transactions that are active at the same time.

(Besides, I am not sure whether executing multiple SQL statements over the same connection simultaneously is supported at all.)

Resource deallocation

persistent uses resourcet to ensure that resources (such as buffers that hold result sets) are released as soon as they are not needed.

resourcet works by handling these two scenarios:

No exception is thrown; resources are deallocated by an explicit release call.
An exception is thrown, preventing the release action from happening. However, once the exception escapes the enclosing ResourceT block, it triggers the exception handler inside runResourceT. The exception handler then performs deallocation.

When your application consists of one long-running SqlPersistM action, chances are you’re catching some exceptions inside the ResourceT block, by the means of monad-control. Doing that invalidates resourcet’s assumptions: an exception prevents the release action from happening, and yet it never makes it up to runResourceT, and so your long-running app leaks resources.

Do it right

It implies from the above considerations that the right way to use persistent with a SQL backend is:

Make SqlPersistT correspond to logical transactions in your application.
Make ResourceT computations as short-lived as possible. Ideally, don’t catch exceptions inside ResourceT; use finally instead.
Use a connection pool.

Disclaimer

I am not an expert in either persistent or SQL databases; I am in the process of figuring this out myself. Corrections (and confirmations) are welcome.

Two failed attempts at extensible effects

Sat, 14 Jun 2014 20:00:00 +0000

After I had published The problem with mtl, many people wondered what my proposed solution was. If you, like them, are impatient to find out, feel free to peek at the slides from my kievfprog talk, or directly at the code on github.

Still, I’ll continue this series at my own pace. Today we’ll look at two failed solutions to the problem described in the previous article.

Free monads

The approach based on free monads, proposed in the original Extensible Effects paper by Oleg Kiselyov, Amr Sabry, and Cameron Swords, and implemented in the corresponding package indeed addresses our problem. My plan for ZuriHac was to revise its API and measure and possibly improve its performance.

I started the hackathon by writing a benchmark to compare different free monad implementations, to decide which one to use internally in the library.

The competing free monad implementations were:

Free: the standard Pure a | Free (f (Free f a)) free monad
Codensity: forall b. (a -> f b) -> f b, as described in Asymptotic Improvement of Computations over Free Monads
Church: forall w. (a -> w) -> (f w -> w) -> w, as described in Free Monads for Less. This is the Church- (or Boehm-Berarducci-) encoded version of Free.
NoRemorse, as described in Reflection without Remorse (code taken from Atze van der Ploeg’s repository)

The benchmark compared the performance of State-like monads implemented on top of each free monad. I also added a plain State from the transformers package to this comparison.

What surprised me most was not the relative performance of different representations, but how the State monad implemented through free monads did relatively to the State from transformers.

The free monad based State consistently performed up to two orders of magnitude slower than the transformers’ version. And the free monad version was essentially Free State, even without the Typeable-based open unions (which certainly carry an overhead of their own).

Thus, it became clear that if an extensible effects library is to perform well, it has to be based on raw transformers, not free monads.

mtl

If, as I wrote in the previous article, the functional dependency is an issue in mtl, can we simply get rid of it?

Well, we could, but that by itself wouldn’t help much. You see, mtl’s classes work by having instances that lift, say, MonadState actions through ReaderT, WriterT, and other transformers known to mtl:

instance MonadState s m => MonadState s (ReaderT r m) where
    get = lift get
    put = lift . put
    state = lift . state

In order to make multiple MonadState instances work, we’d have to write a similar instance for lifting MonadState through StateT itself. But in that case GHC would become confused: it wouldn’t know whether to lift a given get through a StateT, or attempt to execute it right there. It just isn’t smart enough to make a decision based on the type of the StateT’s state.

My solution to this problem is exactly to teach GHC to make such a decision. We’ll see how it works in the next article.

The problem with mtl

Wed, 11 Jun 2014 20:00:00 +0000

This article starts a series in which I am going to publish my thoughts on and experience with different «extensible effects» approaches. This one in particular will explain the problem with the classic mtl approach that motivates us to explore extensible effects in the first place.

How transformer stacks are born

Often we start with a single monad — perhaps Reader or State. Then we realize it would be nice to add more to it — other ReaderTs or StateTs, probably an EitherT etc.

At that point writing the whole stack in a type signature becomes rather onerous. So we create a type alias for it, or even a newtype, to improve type error messages. At first it looks like a good idea — we have «the monad» for our application. It removes a lot of the cognitive overhead — all our internal APIs are structured around this monad. The more time we spend working on our application, the more useful functions we invent that are automatically compatible and composable; the more joy it becomes to write code.

At least this is how I used to structure my code. I learned this approach from xmonad, the first «serious» Haskell project I studied and took part in. It has the X monad, and all the functions work in and/or with this monad.

Concrete stacks are too rigid

This approach breaks, however, once we want to have multiple applications based on the same code. At work, for instance, I’d like to reuse a significant part of code between the real application, the simulator (kind of a REPL for our messaging campaigns) and tests. But those necessarily use different monad stacks! The simulator doesn’t deal with MySQL and RabbitMQ connections; the server doesn’t need to be able to travel back and forth in time, like our simulator does; and tests for a piece of functionality should ideally use the smallest stack that’s necessary for that functionality.

So we should abstract in some way from the monad stack.

mtl’s classes

One such abstraction comes directly from mtl, the monad transformers library.

If we simply write

{-# LANGUAGE NoMonomorphismRestriction #-}
import Control.Monad.State
import Control.Monad.Reader

foo = do
  x <- ask
  put $ fromEnum $ not x

without supplying any type signature, then the inferred type will be

foo :: (MonadReader Bool m, MonadState Int m) => m ()

This type signature essentially says that foo is a monadic computation which has two effects: reading a boolean value and reading/writing an integral value. These effects are handled by the familiar «handlers» runState and runReader.

We can combine any such computations together, and the type system will automaticaly figure out the total set of effects, in the form of class constraints. E.g. if we also have

bar :: (MonadState Int m, MonadWriter All m) => m ()

then

(do foo; bar) :: (MonadReader Bool m, MonadState Int m, MonadWriter All m) => m ()

So it looks like mtl can provide us with everything that the «extensible effects» approach promises. Or does it?

The limitation

Unfortunately, if we write something a little bit different, namely

{-# LANGUAGE NoMonomorphismRestriction #-}
import Control.Monad.State
import Control.Monad.Reader

foo = do
  x <- get
  put $ fromEnum $ not x

where we’ve changed ask to get, the compiler gets confused:

test.hs:6:3:
    No instance for (Monad m) arising from a do statement
    Possible fix:
      add (Monad m) to the context of the inferred type of foo :: m ()
    In a stmt of a 'do' block: x <- get
    In the expression:
      do { x <- get;
           put $ fromEnum $ not x }
    In an equation for ‘foo’:
        foo
          = do { x <- get;
                 put $ fromEnum $ not x }

test.hs:6:8:
    No instance for (MonadState Bool m) arising from a use of ‘get’
    In a stmt of a 'do' block: x <- get
    In the expression:
      do { x <- get;
           put $ fromEnum $ not x }
    In an equation for ‘foo’:
        foo
          = do { x <- get;
                 put $ fromEnum $ not x }

test.hs:7:3:
    No instance for (MonadState Int m) arising from a use of ‘put’
    In the expression: put
    In a stmt of a 'do' block: put $ fromEnum $ not x
    In the expression:
      do { x <- get;
           put $ fromEnum $ not x }

This is because mtl asserts, via a mechanism called functional dependency, that a monadic stack can have only once instance of MonadState. Because get and put in the above example operate with different types of state, that code is invalid.

Merging transformer layers

Since we can’t have multiple different MonadState constraints for our reusable monadic computation, we need to merge all StateT layers in order to be able to access them through the MonadState class:

data MyState = MyState
  { _sInt :: Int
  , _sBool :: Bool
  }

Then we could generate lenses and put them in a class to achieve modularity:

class Has f t where
  hasLens :: Lens t f

foo :: (MonadState s m, Has Int s, Has Bool s) => m ()

The drawbacks of this approach are:

It is boilerplate-heavy, requiring an instance per field and a record per stack. When you need to convert between these records, it can be quite annoying.
Since monad transformers don’t commute in general, you can’t always merge two StateT layers together. For instance, there’s no way to achieve the semantics of StateT s1 (MaybeT (StateT s2 Identity)) using only one layer of StateT.

Conclusion

mtl’s classes almost provide a valid «extensible effects» implementation, if not for the functional dependency that lets us have only single MonadState instance per stack.

In the subsequent article we’ll explore ways to address this limitation.

Avoid equational function definitions

Fri, 09 May 2014 20:00:00 +0000

One of the first things that Haskell beginners usually notice is that Haskell has this somewhat unusual but attractive way of defining functions case-by-case:

foldr f z []     = z 
foldr f z (x:xs) = f x (foldr f z xs)

It looks fun and math-y. The other way to do pattern matching, case expressions, is much less advertized, probably because case invokes associations with dirty old imperative programming. Here’s how the same function could be defined using case:

foldr f z l =
  case l of
    []   -> z 
    x:xs -> f x (foldr f z xs)

However, there are plenty of reasons to prefer case to multiple function definition clauses.

(If some of these look insignificant at first sight, think of a datatype with tens of constructors, which is quite common when working with abstract syntax trees.)

DRY. Notice how in the equational style the function name and argument names get repeated.
It makes it clear what the function decides upon. The equational style allows you to pattern match on different arguments in different clauses, or even on multiple arguments in the same clause:
```
f [] 0 = 0
f _  1 = 1
f _  _ = 2
```
It gives more power, but also makes it harder to see what’s going on. More importantly, even when this additional power is not used, it’s not obvious from the code itself until you eye-scan all the clauses.
It makes code easier to modify or refactor. Tasks like
- adding or removing a function argument
- introducing a local definition common for multiple cases
- preprocessing function arguments or post-processing the function result
are trivial with the case expression, and hard to impossible (without rewriting or introducing other top-level functions) with clauses.
When profiling, you often want to add an {-# SCC #-} pragma for a function. If the function is written using multiple cases, you need to attach this pragma to every clause separately. Moreover, even if you do so, they won’t account for the evaluation of arguments due to pattern matching in left-hand sides of the equations.
Once you start reading the Core or STG code, writing functions using case makes it much easier to follow the connection between the original source and its intermediate representation.

Perhaps the only reason to have multiple clauses is if you need that additional power of matching on several arguments at the same time, e.g.

Right a <*> Right b = Right (a b)
Left  a <*> Right _ = Left a
Right _ <*> Left b  = Left b
Left  a <*> Left b  = Left (a <> b)

You could do this with case by matching on tuples, but it isn’t as nice.

Other than this, I rarely ever define functions in the equational style in my code.

Lens is unidiomatic Haskell

Thu, 24 Apr 2014 20:00:00 +0000

Edward Kmett writes:

Ironically if I had to say anything from the traffic in my inbox and on #haskell about it, it is mostly the old guard who gets disgruntled by lens.

So let me try and explain why that is. I’ll go ahead and say this: lens is unidiomatic Haskell.

By which I mean that lens isn’t like any other library that we normally use. It doesn’t follow the conventions of Haskell APIs, on which I elaborate below.

Now let me clarify that this doesn’t necessarily mean that lens is a bad library. It’s an unusual library. It’s almost a separate language, with its own idioms, embedded in Haskell.

It is as unusual as, say, Yesod is. But unlike Yesod, which follows Haskell’s spirit, not letter (syntax), lens, I argue, follows Haskell’s letter, but not its spirit.

So here’s why I think lens violates the spirit of Haskell:

In Haskell, types provide a pretty good explanation of what a function does. Good luck deciphering lens types.

Here’s a random function signature I picked from lens:
```
below :: Traversable f => APrism' s a -> Prism' (f s) (f a)
```
Despite having some (fairly basic) understanding of what prisms are, this signature tells me nothing at all.

So you have to rely on documentation much more than on types. Yeah, just like in Ruby.
Usually, types in Haskell are rigid. This leads to a distinctive style of composing programs: look at the types and see what fits where. This is impossible with lens, which takes overloading to the level mainstream Haskell probably hasn’t seen before.

We have to learn the new language of the lens combinators and how to compose them, instead of enjoying our knowledge of how to compose Haskell functions. Formally, lens types are Haskell function types, but while with ordinary Haskell functions you immediately see from types whether they can be composed, with lens functions this is very hard in practice.
The size of the lens API is comparable to the size of what I’d call «core Haskell» (i.e. more or less the base library). It is also similar in spirit to base: it has a big number of trivial combinations of basic functions, in order to create a base vocabulary in which bigger programs are expressed.

Ordinary libraries, instead, give only basic functions/combinators, and rely on the vocabulary provided by base (or lens) to compose them together.

This is why I regard lens as a language in its own. And this demonstrates why learning lens is hard: surely learning a new language is harder than learning a new library.
Dependencies. A library as basic in its purpose as lens ideally should have almost no dependencies at all. Instead, other libraries should depend on it and implement its interface. (Or even do it without depending on it, as is possible with lens.)

A package implementing lenses that depends on a JSON parsing library and a gzip compression library sounds almost like a joke to me.

OTOH, it kind of makes sense if you think about lens as a language. It just ships with a “rich standard library”. Nice!
Backward composition of lenses. It’s a minor issue, and I wouldn’t mention it if it wasn’t a great demonstration of how lens goes against the conventions of Haskell.

Note that I’m not trying to make a judgment here (although my tone probably does give away my attitude towards lens). I’m simply explaining why people may dislike and resist it.

Nor am I trying to argue against any particular design decision of lens. I’m sure they all have valid rationale behind them.

I just hope that someone will write an idiomatic Haskell library as powerful as (or close to) lens, with perhaps a different set of compromises made. Otherwise, I’m afraid we all will have to learn this new language sooner or later.

Setting up Samsung Wireless Printer on Linux

Mon, 21 Apr 2014 20:00:00 +0000

Here’s a complete guide for setting up a wireless Samsung printer on Linux, where by “setting up” I mean making it connect to your wireless network.

It worked for me with Samsung ML-2165W on Debian GNU/Linux «jessie», but should work for other models and distributions, too.

Connecting Samsung printer to a wireless network

Create a new, temporary user. We’ll use it to launch Samsung’s wireless setup utility. This is optional, but it provides an additional layer of security (who knows what those utilities from Samsung do behind the scenes) and ensures that nothing will mess with your system.

We add the new user to the lp group, so that it can talk to the printer.
```
 user$ sudo useradd --create-home --shell /bin/bash --groups lp samsung
```
Allow the new user to use our display. (Samsung’s utility is graphical.)
```
 user$ xhost +local:samsung
```
Now, time to switch to our temporary user.
```
 user$ sudo -E su samsung
```
Download Samsung’s PSU (“Printer Settings Utility”) archive from their website. Unpack it and go to the wirelesssetup directory.
```
 samsung$ wget http://downloadcenter.samsung.com/content/DR/201110/20111019151150392/PSU_1.01.tar.gz
 samsung$ tar xzf PSU_1.01.tar.gz
 samsung$ cd cdroot/Linux/wirelesssetup
```
(If Samsung’s link doesn’t work, here is a backup of that archive.)
Check if there are any missing dynamic libraries:
```
 samsung$ ldd bin64/wirelesssetup  | grep 'not found'
```
(Note: this is for a 64-bit system. On a 32-bit system, replace bin64 with bin.)

In my case, the output was
```
 libnetsnmp.so.10 => not found
```
This particular library is included in the PSU archive, so we load it by
```
 samsung$ export LD_PRELOAD=$PWD/../psu/share/lib64/libnetsnmp.so.10.0.2
```
(Likewise, replace lib64 with lib on a 32-bit system.)

If there are more missing libraries, first see if your distribution ships them. The major versions must match! E.g. Debian jessie ships libnetsnmp.so.30.0.2, which has the major version number 30, so that won’t do.

If libstdc++.so.5 is missing, it may be provided by a compatibility package (e.g. compat-libstdc++-33 on Fedora 26).

If your distribution doesn’t have the right version, use a resource like http://rpm.pbone.net/ to find a package that has one. Unpack it (do not install!) and set LD_PRELOAD and/or LD_LIBRARY_PATH so that they are found.
Now connect the printer via a USB cable to the Linux machine and run
```
 samsung$ bin64/wirelesssetup /dev/usb/lp0
```
A graphical window should appear, where you’ll be able to choose your wireless network and enter the password to it.
After you made the printer connect to the wireless network, you can logout and remove the temporary user. Note that the command below will remove that user’s home directory.
```
 user$ sudo userdel --remove samsung
```

Configuring the printer

This PPD file worked well for me with this printer model.

JSON validation combinators

Sun, 20 Apr 2014 20:00:00 +0000

At Signal Vine we have a JSON-based language for describing text messaging campaigns. We may design a better surface syntax for this language in the future, but the current one gets the job done and there are certain existing systems that depend on it.

Anyway, the problem with this language is that it is too easy to make mistakes — including errors in JSON syntax, structural errors (plugging an array where an object is expected), or name errors (making a typo in a field name).

So the first thing I did was write a validator for our JSON format.

There are several projects of «JSON schemas» around, but there were many reasons against using them.

I don’t know about the quality of the tools that support such schemas (i.e. the quality of error messages they generate), and the expressivity of the schemas themselves (whether they’d let us to express the structure of our JSON structure and the constraints we’d like to enforce). So, though it may seem that using an existing solution is «free», it is not — I’d have to spend time learning and evaluating these existing solutions.
I remember that we went through this in our team at Barclays, and eventually decided to create a custom JSON schema language, although I was not involved in the evaluation process, so can’t share the details.
I was almost certain that no existing «generic JSON schema» solution can provide the power of a custom one. For instance, some of the JSON strings contain expressions in another in-house mini-language. Ideally, I’d like to parse those expressions while I am parsing the enclosing JSON structure, and give locations of possible errors as they appear in the JSON file.
I’d need a parser for the language anyway. Maintaining a schema separately from the parser would mean one more thing to keep in sync and worry about.

I couldn’t use an existing JSON parsing library either. Of course, aeson was out of question, being notorious for its poor error messages (since it’s based on attoparsec and optimized for speed). json, though, is based on parsec, so its error messages are better.

But there is a deeper reason why a JSON parsing library is inadequate for validation. All of the existing JSON libraries first parse into a generic JSON structure, and only then do they try to recognize the specific format and convert to a value of the target Haskell type.

Which means that during parsing, only JSON syntax errors will be detected, but not the other kinds of errors described above. Granted, they all can be detected sooner or later. But what differentiates sooner from later is that once we’re out of the parsing monad, we no longer have access to the position information (unless, of course, our JSON parsing library does extra work to store locations in the parsed JSON structure — which it typically doesn’t). And not having such position information severely impacts our ability to produce good error messages.

To summarize, in order to provide good diagnostics, it is important to parse exactly the language we expect (and not some superset thereof), and to perform all the checks in the parsing monad, where we have access to location information.

JSON parsing combinators

Even though I couldn’t re-use an existing JSON parser or schema, I still wanted my parser to be high-level, and ideally to resemble a JSON schema, just embedded in Haskell.

The rest of this article describes the JSON schema combinators I wrote for this purpose.

Strings

As I mentioned before, the json package uses parsec underneath, so I was able to reuse some basic definitions from there — most notably, p_string, which parses a JSON string. This is fortunate, because handling escape sequences is not straightforward, and I’d rather use a well-tested existing implementation.

string :: Parser String
string = {- copied from Text.JSON.Parsec -}

I introduced one other combinator, theString, which parses a given string:

theString :: String -> Parser ()
theString str = (<?> "\"" ++ str ++ "\"") $ try $ do
  str' <- string
  if str == str'
    then return ()
    else empty

Objects

Objects are an interesting case because we know what set of fields to expect, but not the order in which they come (it may be arbitrary). Such syntax is known as a «permutation phrase», and can be parsed as described in the classical paper Parsing Permutation Phrases by Arthur Baars, Andres Löh and Doaitse Swierstra.

There are surprisingly many implementations of permutation parsing on hackage, including one in parsec itself. Most of them suffer from one or both of the following issues:

they use custom combinators, which, despite being similar to Applicative and Alternative operators, have their quirks and require learning
they don’t support permutation phrases with separators, which is obviously required to parse JSON objects. (The technique to parse permutation phrases with separators was descibed in the original paper, too.)

On the other hand, the action-permutations library by Ross Paterson addresses both of these issues. It provides the familiar Applicative interface to combine permutation elements (or atoms, as it calls them), and includes the function runPermsSep to parse phrases with separators. The interface is also very generic, requiring the underlying functor to be just Alternative.

Below are the combinators for parsing JSON objects. field parses a single object field (or member, as it’s called in the JSON spec), using the supplied parser to parse the field’s value. optField is similar, except it returns Nothing if the field is absent (in which case field would produce an error message). Finally, theField is a shortcut to parse a field with the fixed contents. It is useful when there’s a tag-like field identifying the type/class of the object, for instance

data Item
  = Book
      String -- writer
  | Song
      String -- composer
      String -- singer

item =
  (try . object $
    Book <$
    theField "type" "book" <*>
    field "writer" string)
  <|>
  (try . object $
    Song <$
    theField "type" "song" <*>
    field "composer" string <*>
    field "singer" string)

(Note: examples in this article have nothing to do with the JSON schema we actually use at Signal Vine.)

One thing to pay attention to is how field parsers (field, theField and optField) have a different type from the ordinary parsers. This makes it much easier to reason about what actually gets permuted.

object :: Perms Parser a -> Parser a
object fields = (<?> "JSON object") $
  between (tok (char '{')) (tok (char '}')) $
    runPermsSep (tok (char ',')) fields

-- common function used by field and optField
field' 
  :: String -- key name
  -> Parser a -- value parser
  -> Parser a
field' key value = theString key *> tok (char ':') *> value

field
  :: String -- key name
  -> Parser a -- value parser
  -> Perms Parser a
field key value = atom $ field' key value

theField
  :: String -- key name
  -> String -- expected value
  -> Perms Parser ()
theField key value = () <$ field key (theString value)

optField
  :: String -- key name
  -> Parser a -- value parser
  -> Perms Parser (Maybe a)
optField key value = maybeAtom $ field' key value

Aside: correct separator parsing

There was only one issue I ran into with action-permutations, and it is interesting enough that I decided to describe it here in more detail.

Consider, for example, the expression runPermsSep sep (f <$> atom a <*> atom b <*> atom c)

It would expand to

(flip ($) <$> a <*> (
  (sep *> (flip ($) <$> b <*>
  (sep *> (flip ($) <$> c <*>
    pure (\xc xb xa -> f xc xb xa)))))
  <|>
  (sep *> (flip ($) <$> c <*>
  (sep *> (flip ($) <$> b <*>
    pure (\xc xb xa -> f xb xc xa)))))
))
<|>
(flip ($) <$> b <*> (
  ...
))
<|>
(flip ($) <$> c <*> (
  ...
))

See the problem? Suppose the actual order of the atoms in the input stream is a, c, b. At the beginning the parser is lucky to enter the right branch (the one starting from flip ($) <$> a <*> ...) on the first guess. After that, it has two alternatives: b-then-c, or c-then-b. First it enters the b-then-c branch (i.e. the wrong one) and fails. However, it fails after having consumed some input (namely, the separator) — which in libraries like parsec and trifecta means that the other branch (the right one) won’t be considered.

We cannot even work around this outside of the library by using try, because we can’t insert it in the right place. E.g. wrapping the separator in try won’t work. The right place to insert try would be around the whole alternative

  (sep *> (flip ($) <$> b <*>
  (sep *> (flip ($) <$> c <*>
    pure (\xc xb xa -> f xc xb xa)))))

but this piece is generated by the lirbary and, as a library user, we have no control over it.

The usage of try inside the library itself is unsatisfactory, too. Remember, the interface only assumes the Alternative instance, which has no notion of try. If we had to make it less generic by imposing a Parsing constraint, that would be really unfortunate.

Fortunately, once identified, this problem is not hard to fix properly — and no usage of try is required! All we need is to change runPermsSep so that it expands to the tree where separator parsing is factored out:

(flip ($) <$> a <*> sep *> (
  (flip ($) <$> b <*> sep *>
  (flip ($) <$> c <*>
    pure (\xc xb xa -> f xc xb xa)))))
  <|>
  (flip ($) <$> c <*> sep *>
  (flip ($) <$> b <*>
    pure (\xc xb xa -> f xb xc xa)))
))
<|>
(flip ($) <$> b <*> sep *> (
  ...
))
<|>
(flip ($) <$> c <*> sep *> (
  ...
))

Now, all alternatives start with atoms, so we have full control over whether they consume any input.

Mathematically, this demonstrates that <*> does not distribute over <|> for some backtracking parsers. Note that such distributive property is not required by the Alternative class.

Even for parser monads that allow backtracking by default (attoparsec, polyparse) and for which there’s no semantic difference between the two versions, this change improves efficiency by sharing separator parsing across branches.

My patch fixing the issue has been incorporated into the version 0.0.0.1 of action-permutations.

Arrays

Arrays should be easier to parse than objects in that the order of the elements is fixed. Still, we need to handle separators (commas) between array elements.

If we interpreted arrays as lists, then the schema combinator for arrays might look like

array
  :: Parser a -- parser for a signle element
  -> Parser [a] -- parser for the array

Implementation would be straightforward, too.

However, in our JSON schema we use arrays as tuples rather than lists. That is, we typically expect an array of a fixed number of heterogeneous elements. Thus we’d like to combine these tuple elements into a single parser using the applicative interface.

Let’s say we expect a 2-tuple of a string (a person’s name) and an object (that person’s address).

data Address -- = ...
data Person =
  Person
    String -- name
    Address

Written by hand, the parser may look like

address :: Parser Address
address = _

personAddress = 
  between (tok (char '[')) (tok (char ']')) $
    Person <$> string <* sep <*> address
  where sep = tok $ char ','

It makes sense to move brackets parsing to the array combinator:

array :: Parser a -> Parser a
array p = (<?> "JSON array") $
  between (tok (char '[')) (tok (char ']')) p

But what should we do with the commas? Manually interspersing elements with separators is error-prone and doesn’t correspond to my view of a high-level JSON schema description.

Inserting comma parsers automatically isn’t impossible — after all, it is done in the action-permutations package and we used it to parse object fields, which are comma-separated, too. But it cannot be as easy as adding a separator to every element since there’s one less separators than elements. We have somehow to detect the last element and not to expect a separator after it.

A nice and simple way to achieve this is with a free applicative functor. A free applicative functor will allow us to capture the whole applicative expression and postpone the decision on where to insert separator parsers until we can tell which element is the last one. In this case we’ll use Twan van Laarhoven’s free applicative, as implemented in the free package.

element :: Parser a -> Ap Parser a
element = liftAp

theArray :: Ap Parser a -> Parser a
theArray p = between (tok (char '[')) (tok (char ']')) $ go p
  where
    go :: Ap Parser a -> Parser a
    go (Ap p (Pure f)) = f <$> p
    go (Ap p1 pn) = flip id <$> p1 <* tok (char ',') <*> go pn

Like object fields, array elements have a special type which makes it clear which pieces exactly are comma-separated.

In fact, the applicative functor Perms is essentially the free applicative functor Ap plus branching.

Optional array elements

Now comes the twist. Some of the array elements may be optional — in the same way as positional function arguments in some languages may be optional. Since the elements are positional, if one of them is omitted, all subsequent ones have to be omitted, too — otherwise we won’t be able to tell which one was omitted, exactly.

For that reason, all optional elements should come after all the non-optional ones; if not, then we’ve made a mistake while designing (or describing) our schema. Ideally, our solution should catch such mistakes, too.

So, how can the above solution be adapted to handle optional arguments?

Attempt #1:

optElement :: Parser a -> Ap Parser (Maybe a)
optElement p = element $ optional p

Here optional is a combinator defined in Control.Applicative as

optional v = Just <$> v <|> pure Nothing

This won’t work at all, as it doesn’t give us any information about whether it’s an optional element or just a parser that happens to return a Maybe type.

Attempt #2:

Well, let’s just add a flag indicating whether the element was created using optElement, shall we?

data El a =
  El
    Bool -- is optional?
    (Parser a)

element :: Parser a -> Ap El a
element = liftAp . El False

optElement :: Parser a -> Ap El (Maybe a)
optElement = liftAp . El True . optional

Now we can check that optional arguments come after non-optional ones. If an element’s parse result is Nothing, we also know whether that element is an optional one, and whether we should stop trying to parse the subsequent elements.

Still, there are two related issues preventing this version from working:

How do we actually know when a parser returns Nothing? Once we lift a parser into the free applicative, its return type becomes existentially quantified, i.e. we should treat it as polymorphic and cannot assume it has form Maybe a (by pattern-matching on it), even if we can convince ourselves by looking at the Bool flag that it indeed must be of that form.
Similarly, once we’ve detected an absent optional element (assuming for a second that it is possible), we have to force all the remaining optional parsers to return Nothing without parsing anything. But again, we cannot convince the compiler that Nothing is an acceptable return value of those parsers.

Attempt #3:

So, we need certain run-time values (the optionality flag) to introduce type-level information (namely, that the parser’s return type has form Maybe a). That’s exactly what GADTs do!

data El a where
  El :: Parser a -> El a
  OptEl :: Parser a -> El (Maybe a)

El’s two constructors are a more powerful version of our old Bool flag. They let us see whether the element is optional, and if so, guarantee that its parser’s return type is Maybeish.

And here’s the code for the parsing functions:

element :: Parser a -> Ap El a
element = liftAp . El

optElement :: Parser a -> Ap El (Maybe a)
optElement = liftAp . OptEl

theArray :: Ap El a -> Parser a
theArray p = between (tok (char '[')) (tok (char ']')) $
  go True False False p
  where
    go :: Bool -> Bool -> Bool -> Ap El a -> Parser a
    go _ _ _ (Pure x) = pure x
    go isFirst optionalOccurred optionalOmitted (Ap el1 eln) =
      let
        eltSequenceError :: a
        eltSequenceError =
          error "theArray: a non-optional element after an optional one"

        !_check =
          case el1 of
            El {} | optionalOccurred -> eltSequenceError
            _ -> ()
      in
        if optionalOmitted
          then
            case el1 of
              El {} -> eltSequenceError
              OptEl {} -> go False True True eln <*> pure Nothing
          else do
            let
              sep = if isFirst then pure () else () <$ tok (char ',')
            case el1 of
              El p1 ->
                flip id <$ sep <*> p1 <*> go False False False eln
              OptEl p1 -> do
                r1 <- optional $ sep *> p1
                go False True (isNothing r1) eln <*> pure r1

theArray is a state machine with three pieces of state: isFirst, optionalOccurred and optionalOmitted. isFirst and optionalOmitted are used to guide actual parsing, while optionalOccurred is needed to check the proper arrangement of optional vs. non-optional arguments.

Conclusions

Although the standard approach to JSON parsing is to parse into a generic JSON representation first, the article shows that an alternative approach — parsing the expected structure directly — is also viable and can be employed to improve error reporting.

Of course, the tricks and ideas described here are not specific to JSON. Understanding how they work and how to use them may become handy in a variety of parsing situations.

Find out the type of an expression/function with typed holes

Thu, 13 Mar 2014 20:00:00 +0000

An often asked Haskell question is how to find out a type of a locally defined function or expression.

The classic solutions are to specify the wrong type and read the error message, or get help from a tool like hdevtools.

Here’s a new one: use the typed holes feature of GHC 7.8+.

On the surface, typed holes solve a somewhat different problem: find out the desired type of the yet unwritten code, while we want to find out the actual type of the written code.

But typed holes are very easy to adapt for our needs. The asTypeOf :: a -> a -> a function forces the types of its two arguments to unify. So we can force the type of a hole to be the same as the type of an existing expression — and voilà!

(Note that asTypeOf is exported from Prelude, so typically you don’t have to import anything to bring it in scope. Nor do you have to enable any extensions for typed holes to work.)

In my case I had code like this

  ...
  let leave = branch testName False
  ...

and I wanted to see what the type of leave is. So I appended `asTypeOf` _:

  ...
  let leave = branch testName False `asTypeOf` _
  ...

and ghci told me:

*Test.Tasty.Runners.Html> :r
[2 of 2] Compiling Test.Tasty.Runners.Html ( Test/Tasty/Runners/Html.hs, interpreted )

Test/Tasty/Runners/Html.hs:86:58:
    Found hole ‘_’
      with type: Maybe (String, H.AttributeValue)
                 -> H.AttributeValue
                 -> H.AttributeValue
                 -> H.AttributeValue
                 -> H.Markup
    Relevant bindings include
      leave :: Maybe (String, H.AttributeValue)
               -> H.AttributeValue
               -> H.AttributeValue
               -> H.AttributeValue
               -> H.Markup
        (bound at Test/Tasty/Runners/Html.hs:86:17)
      mkSummary :: H.Html -> Summary
        (bound at Test/Tasty/Runners/Html.hs:88:17)
      status :: Tasty.Status (bound at Test/Tasty/Runners/Html.hs:82:13)
      i :: IntMap.Key (bound at Test/Tasty/Runners/Html.hs:79:11)
      testName :: String (bound at Test/Tasty/Runners/Html.hs:78:19)
      runTest :: t
                 -> String
                 -> t1
                 -> Traversal (Functor.Compose (t2 IO) (Const Summary))
        (bound at Test/Tasty/Runners/Html.hs:78:9)
      runner :: Tasty.OptionSet
                -> Tasty.TestTree
                -> m (IntMap.IntMap (STM.TVar Tasty.Status) -> IO Bool)
        (bound at Test/Tasty/Runners/Html.hs:72:3)
      (Some bindings suppressed; use -fmax-relevant-binds=N or -fno-max-relevant-binds)
    In the second argument of ‘asTypeOf’, namely ‘_’
    In the expression: branch testName False `asTypeOf` _
    In an equation for ‘leave’:
        leave = branch testName False `asTypeOf` _
Failed, modules loaded: Paths_tasty_html.
*Paths_tasty_html>

As you see, I got not just the type of the hole itself, but also the types of some other relevant definitions — very handy!

Another thing you can do is put _ = _ inside a let or where bindings group and play with -fmax-relevant-binds=N and -fno-max-relevant-binds.

Happy, Alex, and GHC 7.8

Sat, 08 Mar 2014 20:00:00 +0000

As we approach the 7.8 release of GHC, more and more people are running into problems with packages that use Alex and/or Happy for parsing.

The errors look like

templates/GenericTemplate.hs:104:22:
    Couldn't match expected type ‛Bool’
                with actual type ‛Happy_GHC_Exts.Int#’
    In the expression:
      (n Happy_GHC_Exts.<# (0# :: Happy_GHC_Exts.Int#))
    In a stmt of a pattern guard for
                   a case alternative:
      (n Happy_GHC_Exts.<# (0# :: Happy_GHC_Exts.Int#))
    In a case alternative:
        n | (n Happy_GHC_Exts.<# (0# :: Happy_GHC_Exts.Int#))
          -> (happyReduceArr Happy_Data_Array.! rule) i tk st
          where
              rule
                = (Happy_GHC_Exts.I#
                     ((Happy_GHC_Exts.negateInt#
                         ((n Happy_GHC_Exts.+# (1# :: Happy_GHC_Exts.Int#))))))

for Happy and

    Pattern bindings containing unlifted types should use an outermost bang pattern:
      ((I# (ord_c))) = ord c
    In the expression:
      let
        (base) = alexIndexInt32OffAddr alex_base s
        ((I# (ord_c))) = ord c
        (offset) = (base +# ord_c)
        ....
      in
        case new_s of {
          -1# -> (new_acc, input)
          _ -> alex_scan_tkn
                 user orig_input (len +# 1#) new_input new_s new_acc }
    In a case alternative:
        Just (c, new_input)
          -> let
               (base) = alexIndexInt32OffAddr alex_base s
               ((I# (ord_c))) = ord c
               ....
             in
               case new_s of {
                 -1# -> (new_acc, input)
                 _ -> alex_scan_tkn
                        user orig_input (len +# 1#) new_input new_s new_acc }
    In the second argument of ‘seq’, namely
      ‘case alexGetChar input of {
         Nothing -> (new_acc, input)
         Just (c, new_input)
           -> let
                (base) = ...
                ....
              in
                case new_s of {
                  -1# -> ...
                  _ -> alex_scan_tkn
                         user orig_input (len +# 1#) new_input new_s new_acc } }’

for Alex. (These are not all the error messages that are produced by GHC, but hopefully enough that this article is googlable.)

First I give instructions on how to fix these problems, and then explain why they arise in the first place.

TL;DR: how do I fix the package?

As a maintainer

Install the latest versions of alex and happy. GHC 7.8 support was added in alex-3.1.0 and happy-1.19.0, but later versions contain additional bugfixes.

Double-check that cabal picks the latest versions of these tools: in your package’s source tree run

cabal configure -v | grep -e alex -e happy

The output should look like

Using alex version 3.1.3 found on system at: /home/feuerbach/bin/alex
Using happy version 1.19.3 found on system at: /home/feuerbach/bin/happy

Bump the package’s version (the fourth component is enough: e.g. 1.2.3 -> 1.2.3.1), build the package and upload:
```
cabal build
cabal sdist
cabal upload dist/$pkg-$version.tar.gz
```

That’s it; no actual source code modification to your package is necessary. If you’re curious as to why this works, read on.

As a user

First of all, check that you have latest alex and happy installed. That by itself can resolve your problem.

If it doesn’t, notify the package maintainer(s) about this problem and send them a link to this article. Only they are in a position to fix the problem properly.

Until the maintainer(s) react, you can fix the problem locally as follows:

Get into the source tree:
```
cabal get $pkg
cd $pkg-$version
```
(If cabal says it doesn’t know about the get command, you have to update it with
```
cabal install cabal-install
```
The command was called unpack before, but since you are using GHC 7.8 now, the older versions of cabal will get you in trouble anyway.)
Now that you’re in the source tree, run
```
cabal clean
```
You may think «but I’ve just downloaded a fresh copy of the package’s source — surely it is clean!» Not really; read on for the details.
Finally,
```
cabal install
```
should run without any alex- or happy-related errors.

What’s going on here?

Code produced by old Happy and Alex no longer builds

Because Alex and Happy strive to produce the most efficient code, they make use of unboxed types and primitives. And those are affected by certain changes in GHC 7.8:

PrimOps for comparing unboxed values now return Int# instead of Bool
Lazy unlifted bindings are an error now. For background on this one, see #2806, #3278 and #8022.

Happy and Alex were then updated to generate code that builds with the new GHC. So, it seems, just updating happy/alex should do the trick. Not so fast!

cabal includes generated code in the source distribution

When cabal creates a source distribution for uploading to hackage (cabal sdist), it includes the files generated by alex and happy in the tarball. So even when you have the new alex and happy installed, cabal install will not see the need to regenerate .hs files from .x and .y sources, and will run into the errors described above.

That’s why maintainers have to re-upload their tarballs generated with new alex and happy; and until they do, users have to run cabal get and cabal clean.

The rationale behind this cabal behavior is not to force users install alex or happy. Alas, it doesn’t work so well in practice:

Dec 09 19:37:21 dcoutts refold: there's a few problems with our shipped pre-processed sources system
Dec 09 19:37:41 dcoutts it doesn't interact well with using  build-tools: happy
Dec 09 19:38:03 dcoutts if there are shipped sources then obviously we do not need happy
Dec 09 19:38:14 dcoutts the shipped sources currently go in dist
Dec 09 19:38:18 dcoutts that then fails if you clean
Dec 09 19:38:33 dcoutts it only allows one instance of shipped sources
Dec 09 19:38:52 dcoutts e.g. for happy & alex, they can produce ghc-specific output or generic output
Dec 09 19:39:20 dcoutts this is less of a problem these days since in practice there are
                        not other compilers
Dec 09 19:40:19 dcoutts and then this new problem, if we do ship sources, we don't know what version
                        of the pre-processor generated them, so we cannot easily hack around version
                        incompatibilities
Dec 09 19:40:21 refold  yes, using dist is hack
Dec 09 19:40:30 dcoutts the plan was to use a different dir
Dec 09 19:40:31 refold  also fails with a different --builddir
Dec 09 19:40:34 dcoutts right

cabal sdist is not “pure”

We usually think of cabal sdist as a pure function taking in the source tree and producing the tarball. It’s not that simple.

Above I wrote that cabal sdist includes alex and happy-generated sources in the tarball. However, as Mikhail Glushenkov explains, it doesn’t actually generate them. It only includes them if they are already present as an artifact of a previous cabal build.

When I uploaded haskell-src-exts-1.16.0, I wasn’t aware of this and apparently ran cabal clean before cabal sdist. As a consequence, the tarball doesn’t have the dist/ subdirectory with generated files as you can easily check.

In order to install that particular version of haskell-src-exts, a user needs to have happy installed (and if her happy is old, she’ll get the exact same error described above). When I learned about it, I made a point release, 1.16.0.1, which does include the happy output.

So this is another thing that maintainers need to be aware of and watch out for.

cabal sandbox tips

Wed, 05 Mar 2014 20:00:00 +0000

In case you missed it, starting from version 1.18 cabal-install has awesome sandboxing capabilities. Here I share a couple of tricks that will make your sandboxing experience even better.

Location-independent sandboxes

By default, cabal uses sandbox only in the directory where cabal.sandbox.config is present. This is inconvenient when sharing a sandbox among multiple projects, and in general makes it somewhat fragile.

With cabal 1.19 (i.e. cabal HEAD as of now) you can set the CABAL_SANDBOX_CONFIG environment variable to the path to your cabal.sandbox.config, and the corresponding sandbox will be used regardless of your current directory.

I’ve defined convenience functions for myself such as

tasty() {
  export CABAL_SANDBOX_CONFIG=$HOME/prog/tasty/sandbox/cabal.sandbox.config
  sandbox_name=tasty
}

for every sandbox I commonly use.

Notice how I also set the sandbox_name variable to the human-readable name of the sandbox. It can be displayed in the prompt as follows:

setopt prompt_subst # force prompt re-evaluation
PROMPT='${sandbox_name+[sandbox: $sandbox_name] }%~ %% '

(sandbox name in the prompt idea is due to /u/cameleon)

Sandbox-aware ghc

Sandboxes only affect cabal, but not ghc or ghci when those are invoked directly. At some point in the future we’ll be able to write

% cabal exec ghc ...

For now I’ve defined the following sandbox-aware wrappers for ghc and ghci:

Clone the repo somewhere

% git clone https://gist.github.com/9365969.git ghc_sandbox

and include in your .bashrc or .zshrc

. ~/path/to/ghc_sandbox/ghc_sandbox.sh

(Why am I wrapping ghci instead of using cabal repl? cabal repl has some side-effects, like re-installing packages, that are not always desirable. And ghci is much faster to start, too.)

tasty-0.8 and other news

Mon, 03 Mar 2014 20:00:00 +0000

I’m glad to announce the 0.8 release of tasty, a modern Haskell testing framework.

Among the important user-visible changes are:

New running modes --hide-successes and --quiet
Short flags for some existing options (-p for --pattern, -j for --num-threads)
Timeout support
Possibility to pass options via environment variables
Fix of a resources-related bug

For details, see the CHANGELOG and README.

tasty now has a mailing list and an IRC channel #tasty at FreeNode. The IRC channel is logged at ircbrowse.net (thanks to Chris Done).

Volunteers

I’d like to thank people who kindly responded to my requests for help with tasty-related packages:

Danny Navarro and Chris Catalfo are working on tasty-html;
Tyler Huffman is going to improve the HSpec test provider.

Dependencies

I recently started to pay more attention to (transitive) dependencies of my packages. More transitive dependencies (esp. those that I do not control) means greater probability that something will break, not to mention the compile times.

As Vincent Hanquez put it,

operation-dependency-streamline. roll your own copies of code and types instead of depending on different packages.
— Vincent Hanquez (@vincenthz) February 27, 2014

For comparison, here are dependency graphs for tasty-0.7 and tasty-0.8, produced by John Millikin’s new cabal-graphdeps tool:

The gains were achieved by:

Dropping the dependency on either. First I just copied the code over to tasty, but then realized that using exceptions in that case was an even better solution.
Refusing to depend on reducers. Instead, I just copied the desired pieces.
Using unbounded-delays for timeouts instead of data-timeout that I considered initially. This one actually shows the danger of fat dependencies — one of data-timeout’s dependencies fails to build with GHC 7.4 due to an alleged compiler bug affecting some piece of code that is completely irrelevant for my purposes.

My Haskell will

Sat, 08 Feb 2014 20:00:00 +0000

I hate it when maintainers become unreachable. At the same time, I’m not immune to that myself (if nothing else, I could be hit by a bus tomorrow).

So I contacted a few people with a request to become backup maintainers (BM) for some of my more popular Haskell packages, and they kindly agreed.

Specifically:

Oliver Charles is now BM for all my testing-related packages: tasty and its add-ons, smallcheck, obsolete test-framework add-ons, and ansi-terminal (a dependency of tasty)
Adam Bergmark is now BM for the haskell-suite projects: haskell-names, haskell-packages, hse-cpp, and traverse-with-class (a dependency of haskell-names)
Sjoerd Visscher is co-BM for traverse-with-class
Oleksandr Manzyuk is now BM for ariadne and bert (a dependency of ariadne)

Being a backup maintainer comes with very light responsibilities:

should I become unreachable (temporarily or permanently), and a change has to be made to a package to keep it usable, the BM is supposed to review, apply, and release that change.
if I am unreachable for a long time or permanently, and there’s a person/people who want to take over maintenance/development of all or some of the packages, and the BM has no objections to them doing so, the BM is supposed to give them the necessary privileges. (Of course, that person may be the BM him/herself!)

The BM for a package is added as a maintainer of that package on hackage and as a collaborator for the package’s github repository.

To make it clear, there’s no obligation for the BM to fix bugs or continue the development after I disappear. It would be unreasonable to request a person to commit to such a responsibility at an indefinite point in the future.

I assume that if a project is important, there will be people willing to take care of it; and if it isn’t, then it doesn’t matter anyway. The main responsibility of the BM is thus to make it easy for such a hypothetical person to take over.

As to what it means to be «unreachable», I completely trust my BM’s judgement here. I don’t want them to follow any bureaucratic procedures. The risk of something going wrong is very low and easily outweighed by the benefits of timely response to problems.

One package that doesn’t have a BM yet is standalone-haddock. If you use it and would like to become a BM, please get in touch.

I also encourage other package maintainers to follow this example and appoint BMs for their popular packages.

Resources in Tasty (update)

Sun, 29 Dec 2013 20:00:00 +0000

In a recent article I described how resources were introduced to the Tasty test framework, as well as some alternative approaches. This article describes the new API, introduced in Tasty 0.7.

To recap, there was a function, withResource, that handled creation/acquisition and disposal of resources, but if you needed to access the resource directly in the tests, you had to store the resource in an IORef (or similar) as part of the initialization routine.

At the time it seemed acceptable, but later I discovered that when the number of resources was bigger than one or two, or even not known in advance (when tests are generated rather than just written down), this was inconvenient enough to start looking for a different solution.

One of the major problems with tests receiving the resource value directly, as in

withResource
  :: IO a
  -> (a -> IO ())
  -> (a -> TestTree)
  -> TestTree

… was that the resource could be used not only in the tests themselves, but to construct the tests, which is bad/wrong for a number of reasons. For instance, we don’t want to create the resources when we’re not running tests, but we still want to know which tests we have.

The solution I found is to pass not the value of the resource, but an IO action yielding the resource.

withResource
  :: IO a
  -> (a -> IO ())
  -> (IO a -> TestTree)
  -> TestTree

Even though it’s an IO action, it doesn’t acquire the resource, because such a resource wouldn’t be shared across multiple tests, which is the semantics we’re after. Instead, it returns the resource which has been acquired (think: reads from an IORef or MVar). But thanks to it being an IO action, it can only be used inside a test, and not to construct or alter tests based on the resource value.

Here’s a modified example from the last article which works with this new API:

import Test.Tasty
import Test.Tasty.HUnit

-- assumed defintions
data Foo
acquire :: IO Foo
release :: Foo -> IO ()
testWithFoo :: Foo -> Assertion
(acquire, release, testWithFoo) = undefined

main = do
  defaultMain $
    withResource acquire release tests

tests :: IO Foo -> TestTree
tests getResource =
  testGroup "Tests"
    [ testCase "x" $ getResource >>= testWithFoo
    ]

Custom options in Tasty

Fri, 20 Dec 2013 20:00:00 +0000

Tasty 0.6 is released, making it possible to create custom options just for your test suite!

Add your own option in three easy steps:

Define a datatype to represent the option, and make it an instance of IsOption
Register the options with the includingOptions ingredient
To query the option value, use askOption.

Examples follow.

Ignoring a test

My use case is a test suite that has a number of tests that fail on a certain build bot. I can’t fix the build bot configuration ATM, so I’d like to be able to mark these tests as known-fail in the build script for this particular build bot. — 23Skidoo

To some extent this is just a way around Tasty’s limited pattern language (which will improve, too!), but I still find it pretty nice.

With the following code, you can disable the second test by passing a --buildbot command-line option.

{-# LANGUAGE DeriveDataTypeable #-}

import Test.Tasty
import Test.Tasty.Options
import Test.Tasty.HUnit
import Data.Typeable (Typeable)
import Data.Tagged
import Data.Proxy
import Options.Applicative

newtype BuildBot = BuildBot Bool
  deriving (Eq, Ord, Typeable)

instance IsOption BuildBot where
  defaultValue = BuildBot False
  parseValue = fmap BuildBot . safeRead
  optionName = return "buildbot"
  optionHelp = return "Running under a build bot"
  optionCLParser =
    fmap BuildBot $
    switch
      (  long (untag (optionName :: Tagged BuildBot String))
      <> help (untag (optionHelp :: Tagged BuildBot String))
      )

main = defaultMainWithIngredients ings $
  askOption $ \(BuildBot bb) ->
  testGroup "Tests" $
  [ testCase "Successful test" $ return () ] ++
  if bb
    then []
    else [ testCase "Failing test" $ assertFailure "build bot" ]
  where
    ings =
      includingOptions [Option (Proxy :: Proxy BuildBot)] :
      defaultIngredients

Controlling the depth

When running the tests is there any general solution to set the Depth parameter for the (individual) tests, or better yet, a fine grained solution to set the Depth parameter for individual fields? — jules

Not that I recommend doing this — see this answer.

But here’s how you can do it if you’re sure you want it.

(This was also possible to hack with earlier versions of Tasty — see this gist).

{-# LANGUAGE DeriveDataTypeable #-}

import Test.Tasty
import Test.Tasty.Options
import Test.Tasty.SmallCheck
import Test.SmallCheck.Series
import Control.Applicative
import Data.Proxy
import Data.Typeable

data T1 = T1 { p1 :: Int,
               p2 :: Char,
               p3 :: Int
             } deriving (Eq, Show)

newtype P1Depth = P1Depth { getP1Depth :: Int }
  deriving Typeable

instance IsOption P1Depth where
  defaultValue = P1Depth 5
  parseValue = fmap P1Depth . safeRead
  optionName = return "smallcheck-depth-p1"
  optionHelp = return "Depth to use for p1"

t1Series
  :: Monad m
  => Int -- depth of p1
  -> Series m T1
t1Series d = decDepth $
  T1 <$> localDepth (const d) series <~> series <~> series

main :: IO ()
main = defaultMainWithIngredients (optsIng : defaultIngredients) $
  askOption $ \(P1Depth p1d) ->
    testProperty "Test1" $
      over (t1Series p1d) $
        \x -> x == x
  where
    optsIng = includingOptions [Option (Proxy :: Proxy P1Depth)]

To increase the depth of p1 to 20, pass --smallcheck-depth-p1 20 on the command line.

On column positions in vim

Sat, 14 Dec 2013 20:00:00 +0000

This post describes some of my findings about how locations (specifically, column positions) work in vim. My interest in this originated from the work on ariadne-vim, a TAGS-like plugin for vim.

ariadne-vim works by sending source locations back and forth to ariadne, the server process that does all the intellectual work — parsing and resolving the code. Those source locations had better be computed identically on the server side and the vim side.

Byte offsets («columns»)

The main way column positions are represented in vim is by byte offsets. That’s what functions like getpos, col, and cursor work with.

As an example, consider the following code:

data    Maybe α = Just α | Nothing

To make things interesting, I used a Greek variable name and put a tab after data. These two things — multibyte characters and tabs — are what we will be concerned with.

So, let’s calculate the byte offset of the capital N in Nothing. The tab is just one byte. The Greek alpha, on the other hand… may occupy any number of bytes, depending on the encoding being used.

Assuming the UTF-8 encoding, where alpha occupies two bytes, the byte offset of N is 27. But if the file was encoded using ISO/IEC 8859-7, where alpha is just one byte, then, as seen by vim, the position would be… still 27. That’s because besides the file encoding (as specified by the fileencoding option) vim also has its own internal encoding (the encoding option), and that’s what is used to compute those byte offsets.

The internal encoding is global for vim (unlike the file encoding, which is local to buffers) and is typically set after the locale’s encoding.

Isn’t it great that byte offsets do not depend on file encodings? Not at all. It means that you cannot simply compute offsets externally just by counting bytes in the file. Instead, you have to decode the file using fileencoding, and then re-encode it using encoding — and of course you need to know what those encodings are!

Besides, the parser used by the server process, haskell-src-exts, computes all locations as characters, not bytes. It would be nice if we didn’t have to perform tricky conversions on those locations.

Virtual columns

Virtual column of a position is where on the screen that position actually occurs. It can be obtained using the virtcol function. It’s much closer to the character count than the ordinary column (the byte offset), because even if a character is multibyte, it still takes one column on the screen. (I’m going to ignore combining characters here.)

The tabs are also interpreted differently by virtcol. They occupy variable number of columns — just as they do on the screen! For ariadne it’s a good thing, actually, because the column numbers are computed by the haskell-src-exts parser in the same way, using the tab stops placed every 8 characters.

In our example with Maybe above, the position of N is 28, and the position of M in Maybe is 9, because it comes after the tab character and the preceding text is shorter than 8 characters. (This is all assuming the tab stop size of 8.)

The only issue is that virtcol is computed based on the current value of the ts option, which specifies the tab stop size. Generally speaking, users may have any value of ts, while the Haskell report specifies that the tab stop size is 8, and haskell-src-exts computes locations based on that.

So, in ariadne-vim I temporarily set ts to 8 characters. Initially I was concerned that this will lead to screen flickering, because every time we change the ts value, vim reformats the buffer accordingly. But an experiment revealed that it is done after the full command is completed. As soon as we restore the ts value in the same command, the user won’t notice anything. It’s a hack, but the proper alternative — converting positions to byte counts on the server side — is very complicated.

That’s how we query the current position. How do we jump to a different one? Fortunately, the | motion operates with virtual columns, so we use that. We cannot use the cursor function, which deals with byte counts. And, of course, | is also sensitive to the value of ts, which again has to be modified temporarily.

Credits

Thanks to Ingo Karkat for explaining the situation to me.

Resources in Tasty

Tue, 10 Dec 2013 20:00:00 +0000

This article explores the new feature of the Tasty test framework — resources. It was added in the version 0.5 (and got a small fix in 0.5.1).

Update: this post conveys the general idea about resources, but in later versions of tasty (starting from 0.7), the API is a bit different. To learn more, see this subsequent article.

What problem it solves

Often a group of tests need access to a common resource, such as a connection over which they make requests, or a temporary directory in which they create files. When the tests are done, the resource should be released (the connection should be closed, the directory should be removed).

Previously, allocation of the resource could be performed outside of tasty like this:

import Test.Tasty as Tasty
import Control.Exception

main =
  bracket acquire release $ \resource ->
    Tasty.defaultMain ...

This solution, however, has several problems.

First, the resource is initialized in the beginning and released at the very end. Depending on the kind of resource we’re grabbing, this may be inconvenient or even infeasible.

Second, not all modes of running the test suite involve actually running the tests — like, for instance, --help or --list-tests. But because we’re acquiring the resource outside of tasty, we have no way to know that it’s not necessary.

A similar problem occurs when we do run tests, but not all of them. Remember, we can choose which tests to run with the --pattern option. Maybe for the tests we want to run right now that expensive resource isn’t needed, but again, we can’t know that.

So, to avoid these kinds of problems, special support for resources has been introduced.

How to use it

There’s just one new function you need to be aware of, and its signature is very simple:

withResource
  :: IO a         -- acquire resource
  -> (a -> IO ()) -- release resource
  -> TestTree
  -> TestTree

withResouce annotates a TestTree (typically that will be a testGroup) with the actions that should be run before and after the tests, respectively.

Note the similarity to the bracket function from Control.Exception:

bracket
  :: IO a         -- computation to run first ("acquire resource")
  -> (a -> IO b)  -- computation to run last ("release resource")
  -> (a -> IO c)  -- computation to run in-between
  -> IO c         -- returns the value from the in-between computation

A major difference, however, is that the third argument of withResource isn’t a function and doesn’t have direct access to the resource.

Sometimes it’s not a big deal — if you create a temporary directory with a known name, then you just have to know it’s there.

But often you do need to access the resource from the tests. In that case, use an IORef (created outside of tasty) to store the resource once it’s initialized. Here’s an example from the docs:

import Test.Tasty
import Test.Tasty.HUnit
import Data.IORef

-- assumed defintions
data Foo
acquire :: IO Foo
release :: Foo -> IO ()
testWithFoo :: Foo -> Assertion

main = do
  ref <- newIORef $
    -- If you get this error, then either you forgot to actually write to
    -- the IORef, or it's a bug in tasty
    error "Resource isn't accessible"
  defaultMain $
    withResource (do r <- acquire; writeIORef ref r; return r) release (tests ref)

tests :: IORef Foo -> TestTree
tests ref =
  testGroup "Tests"
    [ testCase "x" $ readIORef ref >>= testWithFoo
    ]

Yeah, perhaps not the most elegant way to pass the resource around, but it’s simple and gets the job done.

In the next section I’ll explain why it’s done in this way.

Alternative designs

test-framework’s way

test-framework allows generating tests in the IO monad using the function

buildTest :: IO Test -> Test

This solves the first problem described in the beginning of this article, but not the second one. Indeed, if we want list all the tests, or perform some other action on them without actually running them, we still have to execute the IO action and acquire the resources unnecessarily.

bracket way

Due to the exact same reason we can’t mimic the bracket function and pass the resource to the tests like this

withResource
  :: IO a
  -> (a -> IO ())
  -> (a -> TestTree)
  -> TestTree

Again, to list the tests we’d have to perform the IO action. Or, when in non-running mode, we could pass error "Don't evaluate the resource in non-running mode" as an a. This last option may actually be not as bad — I’d be interested in what others think.

A hypocritical variation of this would be to replace a -> TestTree with Maybe a -> TestTree, and pass Nothing when in non-running mode. I call it hypocritical because the user still would have to use fromJust or a similar partial function to get the resource, but on our side it looks like everything is total. Pro: reminds the user that the resource may not be there and should not be accessed unless we’re running. Con: boilerplate pattern-matching on the resource.

Type-safe way

a -> TestTree is not the only way to indicate that a test tree depends on the resource.

We could use type-level tricks similar to extensible-effects or regions to record which resources can be accessed by tests. I decided not to do this, because such things make code harder to understand and generally confuse users.

In addition, the problem described in the next section, «Non-type-safe way», applies to any type-safe solution, too.

Non-type-safe way

Even without compile-time guarantee that the resources are acquired and of the right type, we still could automate the IORef business and store resources in a data structure like Map ResourceName Dynamic. We could then provide a monadic interface to accessing the properties, such as

getResource :: Typeable a => ResourceName -> TestM a

But then we’d have to teach every test provider how to run tests in our TestM monad. In some cases (HUnit) this is just not possible. On the other hand, all test providers seem to support running simple IO actions inside tests.

A note on parallelism

There are two possible reasons to share a resource across multiple tests. It could be an optimization (to avoid creating and destroying the resource for every single test), or it could be a semantic requirement. In the latter case, one might want not to enable parallelism to avoid tests running simultaneously or in the wrong order.

While it’s possible to use a TVar to ensure ordering of the tests, that would hurt actual parallelism. Tasty can’t know that the test waits on a TVar instead of running, so it won’t be executing other tests during that time either.

Why PVP doesn't work

Sat, 05 Oct 2013 20:00:00 +0000

The Package versioning policy is one of the controversial topics in the Haskell community. Specifically, the point of disagreement is the upper bounds on dependency versions. Some people consider them necessary, while others consider them harmful.

To remind the arguments of both parties, here’s a typical conversation between a proponent and an opponent of upper bounds:

pro-pvp: A major version change indicates incompatible changes in the API. Your package could fail to compile against that new version!

against-pvp: Well, it could fail, but most of the time it doesn’t, because it uses only 10% of the package’s functionality, and the incompatible change affects some other 5% of the API.

pro-pvp: Why take chances? If you pin down the versions with which you have tested your package, you know it will always compile, no matter which new versions will be released in the future.

I belong to the opponents of upper bounds. Instead, I simply make sure my packages work with the latest everything. This is similar to the Stackage philosophy.

I said «simply», while in practice this can be somewhat time-consuming, given the pace at which the language and libraries evolve. People have complained in the past that maintaining a Haskell package is like shooting a moving target.

But that’s the only way it’s going to work. Below are two reasons why the alternative — enforcing upper version bounds — doesn’t work.

Reason 1 — incompatible constraints

Consider this situation:

package-a depends on containers ==0.4.*
package-b depends on containers ==0.5.*

It’s great that I can compile both of these packages, but what’s the point if I cannot use them together in my application?

Interval constraints do not compose — the intersection of non-empty intervals may be empty. Having just the lower bounds is much better — if each one of them is satisfiable, then they all can be satisfied simultaneously (e.g. by the latest version).

Reason 2 — you can’t nail everything down

In theory, you pin down every dependency and in five years from now it’s going to build in the same way as it just did today.

In practice, people would want to build your package with modern compilers, on modern operating systems, and with external libraries provided by their distributions. These external changes, which you cannot control, will likely cause some of your dependencies to stop building in the future.

To give you a specific example, at some point GHC became more strict (and rightly so) about type synonym instance declarations.

This code

{-# LANGUAGE TypeSynonymInstances #-}

class C a

instance C String

compiles with GHC 7.0, but GHC 7.2 requires also to enable FlexibleInstances, because it’s needed when you expand the synonym.

Every maintained package which has this problem would sooner or later get the fix, but probably only the latest major version would be fixed. If you depend on some older version of a problematic package, you won’t be able to build on anything newer than GHC 7.0.

Conclusion

My first argument shows that putting upper dependency bounds can create obstacles to using your packages, and the second argument shows that upper bounds in practice don’t deliver on their promise of eternal buildability.

So, dear package maintainers, please don’t put the upper bounds on your dependencies without a specific need.

Update

Thanks for everyone who responded. You helped me understand the problem better.

Turns out there are two separate issues here:

Whether to use upper bounds, and
Whether to support the latest versions of your dependencies.

They are related in the sense that in order to avoid supporting the latest versions of your dependencies you have to use upper bounds to keep your library compilable. However, simply using upper bounds does not mean that you do not support the latest versions.

So, my request to the library developers is: please support the latest versions of your dependencies. Failing to do so (and relying on upper bounds to have the illustion that it’s fine) is exactly what causes the problem with incompatible constraints.

As soon as you support the latest versions of your dependencies, upper bounds become harmless but also useless.

They may give you a grace period when a new version of a dependency comes out, but if you stretch this period, then you’re breaking your responsibility to support the latest versions.

On the other hand, having upper bounds puts more strain on you to keep those bounds up-to-date. Plus, not having upper bounds makes it more obvious when you don’t actually support the latest versions.

But at the end, if you are diligent enough, it’s up to you whether to use upper bounds. My personal choice is still not to put them.

Also note that this doesn’t apply to application developers (at least when the application doesn’t support dynamically loaded plugins). In fact, application developers can compensate for the absence of upper bounds in their dependencies by putting the constraints in their own cabal files.

Flavours of free applicative functors

Sun, 31 Mar 2013 20:00:00 +0000

Six months ago Tom Ellis wrote the article Towards Free Applicatives, where he described his implementation of a free applicative functor. In the reddit discussion of that article a few more implementations were suggested.

In this article I would like to look in more detail how these implementations work and how they differ from each other.

The implementations are rewritten in a common style where possible, to highlight the differences and similarities. Since a picture is worth a thousand words, for each implementation there’s a diagram showing how a simple applicative expression f <$> lift ax <*> lift ay <*> lift az is represented.

Notational conventions

x, y, z, f — simple (non-functorial) values
ax, ay, az — values of the base functor type
tx, ty, tz — values of the free applicative functor type

On the diagrams

F — base functor
Tx, Ty, Tz — inner types of functorial values (e.g. ax has type F Tx). The function f has type Tx -> Ty -> Tz -> Tr.
A square denotes unit (i.e. ()) as a type, term and pattern.

Ørjan Johansen’s free applicative

data Free f a where
  Pure :: a -> Free f a
  Ap :: Free f (a -> b) -> f a -> Free f b

instance Functor f => Functor (Free f) where
  fmap f (Pure x) = Pure $ f x
  fmap f (Ap tx ay) = Ap ((f .) <$> tx) ay

instance Functor f => Applicative (Free f) where
  pure = Pure
  tx <*> Pure y = fmap ($ y) tx
  tx <*> Ap ty az = Ap ((.) <$> tx <*> ty) az

lift :: f a -> Free f a
lift = Ap (Pure id)

lower :: Applicative f => Free f a -> f a
lower (Pure x) = pure x
lower (Ap tx ay) = lower tx <*> ay

The tree grows to the left. This can be easily seen on the diagram, and follows from the fact that the first argument of Ap is a free applicative itself and the second is not. (How the tree grows depends, obviously, on the order of Ap’s arguments. The convention here is that Ap’s arguments are in the applicative order, so that we can meaningfully talk about the tree’s associativity.)

<*> pattern-matches on its right argument, effectively re-associating the tree to the left.

Note that Free f a is an applicative functor regardless of f. It stores f-values in the tree without changes. We’ll see later that this is not always the case.

Twan van Laarhoven’s free applicative

data Free f a where
  Pure :: a -> Free f a
  Ap :: Free f (a -> b) -> f a -> Free f b

instance Functor (Free f) where
  fmap f (Pure x) = Pure $ f x
  fmap f (Ap tx ay) = Ap ((f .) <$> tx) ay

instance Applicative (Free f) where
  pure = Pure
  Pure f <*> tx = fmap f tx
  Ap tx ay <*> tz = Ap (flip <$> tx <*> tz) ay

lift :: f a -> Free f a
lift = Ap (Pure id)

lower :: Applicative f => Free f a -> f a
lower (Pure x) = pure x
lower (Ap tx ay) = flip id <$> ay <*> lower tx

This is a variation of Ørjan’s implementation. As can be seen from the diagrams, the only difference is that it stores the values in the opposite order, and modifies the function to accept them in that order.

As in Ørjan’s implementation, the tree grows to the left, but <*> now pattern-matches on its left argument, in order to push its right argument to the leftmost position in the tree.

The way <*> does pattern matching directly affects its algorithmic complexity. Ørjan’s implementation is linear in the size of its right argument and thus works better with left-associated applicative expressions. Twan’s version is dual: it is linear in the size of its left argument and works better with right-associated expressions. In both cases, unfortunate nesting increases complexity from linear to quadratic.

Also pay attention to the lower function. It has to take care of the effects — the right subtree must be «executed» before the left subtree in order to restore the original value faithfully.

Paolo Capriotti’s free applicative

data Free f a where
  Pure :: a -> Free f a
  Ap :: f (a -> b) -> Free f a -> Free f b

instance Functor f => Functor (Free f) where
  fmap f (Pure x) = Pure $ f x
  fmap f (Ap ax ty) = Ap (fmap (f.) ax) ty

instance Functor f => Applicative (Free f) where
  pure = Pure
  Pure f <*> tx = fmap f tx
  Ap ax ty <*> tz = Ap (fmap uncurry ax) ((,) <$> ty <*> tz)

lift :: Functor f => f a -> Free f a
lift ax = Ap (const <$> ax) (Pure ())

lower :: Applicative f => Free f a -> f a
lower (Pure x) = pure x
lower (Ap ax ty) = ax <*> lower ty

The diagram looks more complicated here. The first thing to notice is that the tree here grows to the right, unlike in the previous versions. The actual application of our f function happens now near the top of the tree rather than at the bottom. To make that possible, the function has to be converted to the uncurried form.

All left nodes except the topmost one are functions (wrapped in the base functor). All they do is take a nested tuple of the arguments downstream and add their own value to that tuple. The topmost left node already knows the final required argument, ax, and simply applies the uncurried function to the tuple.

The functorial values are stored in a modified form, which requires f to be an actual Functor. (But note that any f can be turned into a functor using Yoneda.)

Tom Ellis’s free applicative

The code here is quite different from the previous versions, so I decided to reproduce it verbatim (except the lift and lower functions, which I’ve added myself).

data ChainA f a b = Single (f (a -> b)) | forall c. Many (f (c -> b)) (ChainA f a c)

instance Functor f => Functor (ChainA f a) where
    fmap f (Single t) = Single (fmap (f .) t)
    fmap f (Many g v) = Many (fmap (f .) g) v

chain :: ChainA f b c -> ChainA f a b -> ChainA f a c
chain (Single t) v = Many t v
chain (Many t ts) v = Many t (chain ts v)

data FreeA f a = Pure a | ChainA (ChainA f () a)

instance Functor f => Functor (FreeA f) where
    fmap f (Pure a) = Pure (f a)
    fmap f (ChainA a) = ChainA (fmap f a)

instance Functor f => Applicative (FreeA f) where
    pure = Pure
    Pure f <*> g = fmap f g
    f <*> Pure g = fmap ($ g) f
    ChainA f <*> ChainA g = ChainA $ chain (pullUnit f) g

pullUnit :: Functor f => ChainA f () (b -> c) -> ChainA f b c
pullUnit = removeUnit . pull

pass :: Functor f => ChainA f a b -> ChainA f (a,c) (b,c)
pass (Single t) = Single (fmap (\f -> \(a,c) -> (f a, c)) t)
pass (Many t ts) = Many (fmap (\f -> \(a,c) -> (f a, c)) t) (pass ts)

pull :: Functor f => ChainA f a (b -> c) -> ChainA f (a,b) c
pull (Single t) = Single (fmap uncurry t)
pull (Many t ts) = Many (fmap uncurry t) (pass ts)

removeUnit :: Functor f => ChainA f ((), a) b -> ChainA f a b
removeUnit (Single t) = Single (fmap (\f -> \a -> f ((),a)) t)
removeUnit (Many t ts) = Many t (removeUnit ts)

lift :: Functor f => f a -> FreeA f a
lift ax = ChainA $ Single $ const <$> ax

lower :: Applicative f => FreeA f a -> f a
lower (Pure x) = pure x
lower (ChainA chain) = lowerChain chain <*> pure ()
  where
    lowerChain :: Applicative f => ChainA f x y -> f (x -> y)
    lowerChain (Single ax) = ax
    lowerChain (Many ax ty) = (.) <$> ax <*> lowerChain ty

Despite the superficial dissimilarity of the code, this is really a variation on Paolo’s implementation. The only difference is that unit is not passed as a part of the nested tuples. Two-level types and auxiliary functions are really just the cost of eliminating that unit.

Which one to use?

The free package currently uses the Twan’s version, which is the reason of the reverse effect that I mentioned in the previous article. Edward Kmett points out that the reason for choosing Twan’s implementation is that you can see the «next instruction» in O(1) when walking left to right, which is most common.

So start with that one, but also look at your problem (how exactly you are going to analyze the free applicative) and see if any of the alternatives fit better.

gtraverse vs. gfoldl

Fri, 29 Mar 2013 20:00:00 +0000

In Generalizing generic fold, I introduced the gtraverse function as an alternative to gfoldl for generic programming.

To remind the important points:

gtraverse allows more instances to be written, being more flexible about which elements can be treated as being on the same level;
on the other hand, everything that is possible to do with gfoldl seems to be possible to do with gtraverse, too.

It turns out that my intuition was wrong. There is a beautiful correspondence between gfoldl and gtraverse which shows that they are, in fact, equivalent in power. Read on to find out why, and what this means for practical generic programming.

gtraverse through gfoldl

As a quick warm-up, let’s implement gtraverse through gfoldl.

For simplicity, let’s assume that gfoldl and gtraverse are both methods of the same class, Data:

{-# LANGUAGE RankNTypes, GADTs, NoMonoLocalBinds #-}

class Data a where
  gfoldl
    :: (forall d b. Data d => c (d -> b) -> d -> c b)
    -> (forall g. g -> c g)
    -> a -> c a

  gtraverse
    :: Applicative w
    => (forall d . Data d => d -> w d)
    -> a -> w a

Then

gtraverse f = gfoldl g pure
  where
    g acc x = acc <*> f x

In plain English, when folding, we inject each element into c using the supplied function f, and then combine it with the accumulator using <*>.

gfoldl through gtraverse

It turns out that gfoldl can also be interpreted as an applicative traversal for a particular functor — namely, the free applicative functor. We’ll use the one from the free package for now.

If we run gtraverse with liftAp Identity (where Identity wraps its element into a trivial applicative functor), we’ll get a list rather than a tree, which can be easily folded in a gfoldl-like fashion.

One minor issue is that Identity has to capture the Data dictionary of the elements. We can easily achieve that using a GADT

data I a where I :: forall a . Data a => a -> I a

Now, gtraverse (liftAp . I) has type Data a => a -> Ap I a. It transforms any value for which gtraverse is defined to a (heterogeneous) list Ap I a, which contains all immediate children of that value, together with their Data dictionaries.

The next step is to fold that list:

foldAp
  :: (forall d b. Data d => c (d -> b) -> d -> c b)
  -> (forall g. g -> c g)
  -> Ap I a -> c a
foldAp f z (Pure x) = z x
foldAp f z (Ap (I x) k) = (foldAp f z k) `f` x

foldAp takes the same two functional arguments as gfoldl. But instead of working on the original value, it works on the flattened Ap-list.

After gluing the two steps together, we get

gfoldl f z = foldAp f z . gtraverse (liftAp . I)

Nice!

Consequences

So, what does this mean? Do you not need gtraverse since gfoldl can do all the same things?

No!

It is much easier to write instances for your data types using gtraverse, especially when you want to present an alternative view on the type from what is directly implied by its definition. (If you doubt it, you can still try to encode the uniform Data instance for lists and compare it with the gtraverse-based one.)

On the other hand, it is nice to know that we don’t really lose any power by adopting gtraverse instead of gfoldl.

You can try it out right now: I’ve just uploaded the first version of traverse-with-class on hackage.

Remaining issues

There are a couple of issues in our gfoldl implementation above¹:

Because of the free applicative functor usage, it has the same quadratic complexity problems as free monads.
Because of the way this free applicative works, our gfoldl is really a gfoldr: it traverses the elements in the reverse order.

I’ll talk more about these in subsequent articles.

to make it clear, there are no such issues in the library, because it doesn’t provide a gfoldl function. It provides unrelated functions gfoldl' and gfoldr with different types.↩︎

Ergative verbs in programming

Thu, 14 Mar 2013 20:00:00 +0000

For a long time I felt uncomfortable with phrases like «my program doesn’t compile».

Of course it doesn’t compile, stupid! It’s not a compiler!

I understood that this phrase was supposed to mean «The compiler fails to compile my program». But nevertheless it seemed like a wrong usage of the verb «compile».

Well, today I learned (thanks to Edwin Ashworth) that this is a common phenomenon in English. It’s called ergative usage. There is a long list of ergative verbs on Wiktionary, and «compile» is there!

This is a list of programming-related verbs that I’ve seen being used ergatively:

parse
type-check
compile
build

You can notice that they all follow the same pattern — something that one program (usually a compiler) does to another program. I’d be curious to find computing-related ergative verbs that do not fall into this pattern.

Generalizing generic fold

Mon, 11 Mar 2013 20:00:00 +0000

Uniform folding

Suppose we’ve created a new container data structure in Haskell, such as Data.Sequence.Seq. We now want to write a Data instance for it, to allow generic queries and transforms inside our container.

Let’s recall gfoldl definition for lists:

gfoldl f z [] = z []
gfoldl f z (x:xs) = z (:) `f` x `f` xs

We can define gfoldl similarly for sequences (in fact, this is the precise definition from Data.Sequence):

gfoldl f z s = case viewl s of
  EmptyL  -> z empty
  x :< xs -> z (<|) `f` x `f` xs

There are, however, downsides to this definition:

Generic map over our container would be less efficient than hand-written map, even if we ignore the costs associated with run-time type casts.

Indeed, in the generic definition we do not take advantage of the fact that we already have a proper container structure for our elements. Instead, we construct the structure from scratch, adding elements one by one. This overhead may be small for Seq, whose cons is O(1). But in case of an array-based structure, such as Vector, we’re out of luck.
If we only care about the closed recursion combinators, such as everywhere, we wouldn’t know the difference. But when open recursion is needed (e.g. gmapT), the semantics is not the one we’d expect. The intuition is that gmapT maps over the immediate «subterms» of an expression. One might expect that the subterms of Seq a are all of its elements (of type a). But no, we’re forced into this recursive list mindset, where the immediate subterms of a sequence are its head and tail.

(To be fair, it does make sense if we imagine data structures really being inductively-defined terms. But it is a very limited worldview.)

Can we define gfoldl in the uniform fashion, folding over all the elements at once? Try this yourself to understand better why it’s not easy.

I haven’t actually done it, but I think it’s achievable using some amount of type class hackery. Anyway, having to come up with such hacks defeats the point of an easy-to-use generic programming library.

In essence, the limitation of gfoldl is that it forces us to know the number of elements in advance. We cannot (easily) construct the result of the fold by recursive computation.

Generic traversals

In order to resolve the problem stated above, we need to restrict the type of gfoldl¹.

Recall its type:

gfoldl :: (forall d b. Data d => c (d -> b) -> d -> c b)
       -> (forall g. g -> c g)
       -> a -> c a

What might the first functional argument of gfoldl do? It probably transforms d to something like c d by using its Data instance, and then combines c (d -> b) with c d to obtain c b. This is not a sure thing, of course: the caller of gfoldl knows what c exactly is, so it might combine them in unexpected ways. Let us proceed with this assumption, however; as I said above, we need to restrict the caller in order to give more power to the callee.

So, if our guess is correct, we can separate the two activities into two different functions:

gfoldl :: (forall d. Data d => d -> c d)
       -> (forall d b. c (d -> b) -> c d -> c b)
       -> (forall g. g -> c g)
       -> a -> c a

The whole thing now suddenly becomes very familiar: c is an applicative functor! The second and third arguments are <*> and pure, respectively. And the applicative laws had better hold — there are now many ways to express the same thing, and we don’t expect there to be a difference.

Using the Applicative class, we can rewrite the type as

gtraverse :: Applicative c => (forall d . GTraversable d => d -> c d) -> a -> c a

Note that I took this chance to rename our restricted gfoldl to gtraverse and Data to GTraversable. The reason is that it is a generic version of traverse from Data.Traversable, whose type is

traverse :: Applicative c => (a -> c b) -> t a -> c (t b)

Read the type of gtraverse as «traverse this structure, applying the supplied function to the immediate subterms; then use the Applicative instance to reconstruct the original structure».

We can now write the uniform instance for lists (and any other algebraic data type) without a hassle:

gtraverse f [] = pure []
gtraverse f (x:xs) = (:) <$> f x <*> gtraverse f xs

But how does it restrict the caller of gfoldl? Not much. All the usual combinators, such as gmapT, gmapQ, gmapM etc. are expressible with gtraverse. I’d be curious to see an application of gfoldl for which gtraverse is too restrictive².

Yes, the article’s title is a lie. Thanks for reading this far. What we are really generalizing, by allowing more instances, is the Data class.↩︎
See gtraverse vs. gfoldl which shows that gtraverse and gfoldl are in fact equivalent in power.↩︎

Open your name resolution

Mon, 04 Mar 2013 20:00:00 +0000

What is name resolution?

Name resolution is a program analysis which for every name decides what that name refers to.

Consider the following Haskell program as an example:

n :: Integer
n = let k = 10 in k * k

There are three names in this program: n, k and Integer. The result of name resolution in this case would be:

n refers to the global value defined in the program itself, on lines 1-2;
Integer refers to the type (implicitly) imported from Prelude;
k on line 2 is a local value defined on the same line.

This article is concerned with the following question: how should the API of a name resolution library be structured?

Annotations

The standard approach is a function that transforms a simple AST to the annotated AST, where every name is annotated with its binding information (just like we did above).

(Note that a simple map from names to binding information won’t do: the same name can refer to different things in different parts of the program.)

This is enough for most applications of name resolution, such as type checking, compilation or static analysis.

Significance of lexical environment

However, simple annotations are not enough for source-to-source transformations, like the ones performed by HaRe or HasFix.

Such transformations typically require access to the whole symbol table to avoid variable capture. As an example, imagine that in the new version of some module Mod foo has been renamed to bar, and we want to adapt our own code to use this new version. And our code looks like this:

baz x = let bar = x + 1 in foo bar

Name resolution tells us that foo on that line indeed comes from Mod and should be renamed. But if we blindly apply the renaming, we’ll get

baz x = let bar = x + 1 in bar bar

where bar no longer refers to the global value imported from Mod — it has been captured by let!

The solution is simple: rename foo to qualified name Mod.bar whenever simple bar is already defined in the enclosing lexical scope. It requires us, however, to know the lexical environment at every point.

Annotating with symbol tables

Why not store the whole symbol table at every point in the AST? This could work. In a lazy language this is not even as bad, time- and space-wise, as it might seem.

It is cumbersome, however. Every time we replace a subtree, we need manually to re-run name resolution on the new subtree (or even on the whole tree, if there’s any possibility that other annotations may become invalid as a result of the replacement).

Open recursion

In order to maintain the symbol table during the traversal, we can try to keep track of the local bindings ourselves, only to discover that it’s not that trivial, and that we’re replicating a large part of the resolution code.

The core of the problem is that the name resolution algorithm is one big-step tree traversal, and we can only observe its final result (or a trace of its execution in the form of annotations).

If we could split it up into small-step, one layer at a time, resolution functions, we could use them in our own traversals and interleave them with transformations. Another name for this approach is open recursion, hence the article’s title.

Single-sorted AST

If our AST is a single recursive data type (parameterised by the annotation type)

type VarName = String

data Exp a
  = Var a VarName
  | App a (Exp a) (Exp a)
  | Lambda a VarName (Exp a)

type SymbolTable a = Map.Map VarName a

then such a small-step resolution function would have type

resolve
  :: (SymbolTable a -> Exp -> Exp)
  ->  SymbolTable a -> Exp -> Exp

It takes an expression, the symbol table (or environment) that corresponds to the enclosing lexical context of that expression, and a transformation function. It then applies the transformation to all immediate children of the expression.

Note that the transformation function is in turn parameterised by the symbol table. It is the responsibility of resolve to update the symbol table accordingly before using it to transform the children.

An implementation of resolve might look like this

resolve f tbl e =
  case e of
    Var l n -> Var l n
    App l e1 e2 -> App l (f tbl e1) (f tbl e2)
    Lambda l n body -> Lambda l n (f (Map.insert n l tbl) body)

Now that we have resolve, it is easy to tie the knot and write the big-step recursive traversal:

resolveRec :: Exp a -> Exp a
resolveRec = fix resolve Map.empty

resolveRec doesn’t do anything interesting — it’s an identity function. But we can compose resolve with an interesting transformation before “closing” the recursion:

annotate
  :: (SymbolTable a -> Exp a -> Exp a)
  ->  SymbolTable a -> Exp a -> Exp a
annotate f tbl e =
  case e of
    Var _ n -> f tbl $ Var (tbl Map.! n) n
    _ -> f tbl e

annotateRec :: Exp a -> Exp a
annotateRec = fix (resolve . annotate) Map.empty

Let’s try it out:

*Main> annotateRec $ Lambda 1 "x" $ App 2 (Var 3 "x") (Var 4 "x")
Lambda 1 "x" (App 2 (Var 1 "x") (Var 1 "x"))

Hence, the annotation interface to a name resolution library is easy to implement given the open recursion interface. But the open recursion interface is much more powerful, allowing the transformation function to depend on the lexical environment in non-trivial ways.

Multi-sorted AST

What if our AST is represented by a multitude of mutually recursive data types, just like in haskell-src-exts? Generic programming may help here — specifically, the “Scrap your boilerplate” approach.

A stock library, such as syb-with-class, might go a long way, but let’s roll out our own. It will be simpler this way, but also more flexible (for example, it’s possible to modify this presentation to allow the traversal to change the annotation type).

How does the type of resolve change? It now has to work for any AST node type, rather than just for Exp, i.e. be polymorphic. Furthermore, the functional argument to resolve, which used to have type SymbolTable a -> Exp -> Exp, also has to be polymorphic, because the types of node’s children may be different (and may differ from the node’s type itself).

To encode this polymorphism, one can use either a type class or a GADT sum type (see this StackOverflow entry for comparison of these approaches).

Although we are dealing with a finite, closed universe (consisting of haskell-src-exts node types) in this case, I prefer the class-based approach because it’s more practical — it doesn’t force us to define a giant GADT.

Thus, resolve now becomes a class method:

class Typeable a => Resolvable a where
  resolve
    :: (forall b . Resolvable b => SymbolTable -> b -> b)
    ->                             SymbolTable -> a -> a

The Typeable superclass constraint is needed here so that we can write type-specific cases (such as case for Var in the annotate function).

Otherwise, implementing and using this approach for a multi-sorted AST is quite similar to the single-sorted case.

Conclusions

The article argues that the open recursion approach to name resolution offers more power to the user and shows how to implement it.

The presentation here is simplified for the sake of clarity. A full implementation would allow queries (folds), effectful traversals and traversal that can change the annotation type.

For Haskell (and haskell-src-exts AST), I’ve just started implementing this idea in haskell-names.

Announcing SmallCheck 1.0

Tue, 19 Feb 2013 20:00:00 +0000

I am glad to announce a new major release of SmallCheck (hackage).

What is SmallCheck?

SmallCheck is a property-based testing library for Haskell.

If you are familiar with QuickCheck, you can think of SmallCheck as a better QuickCheck. The main way in which it is better is that it is non-random: whether your tests pass does not depend on the roll of the dice.

If you are not familiar with QuickCheck, the idea of SmallCheck is to check certain properties of your functions, as opposed to unit testing, where you just check the expected outcome of some particular tests.

For an introduction to property-based testing, check out Chapter 11 of Real World Haskell. It should be straightforward to translate all the examples there to SmallCheck.

What’s new in 1.0

Monadic properties

Properties are now parameterised on the monad they work in.

The two main ways a property can use the monad are:

generating test cases monadically
verifying the condition monadically

Monadic test cases generation can be used to read test cases from files. A well-known problem when testing compilers is that it is hard to enumerate or randomly generate valid programs. So here you go — just put some examples in the file system, and use them to verify some properties of your compiler.

An example of monadic verification is when you want to verify your code against an external (or otherwise impure) reference implementation.

Quantifiers

The semantics of quantifiers has changed. Previously, all variables were quantified universally by default, and a quantifier (like exists) changed the quantification of the next variable.

So, for example, exists $ \x y -> p x y was interpreted as $\exists x: \forall y. p\ x\ y$, which is probably not what one would expect.

Now the quantification operators (forAll, exists and existsUnique) set the quantification context for all subsequent variables, so the above property means $\exists x, y: p\ x\ y$.

Uniqueness quantification

There’s an interesting effect related to the uniqueness quantifier existsUnique (which was previously called exists1).

There are two ways to interpret existsUnique $ \x y -> p x y, “curried” and “uncurried”:

$\exists ! x: \exists ! y: p\ x\ y$
$\exists ! (x,y): p\ x\ y$

These two actually have different meaning. Suppose that $p\ x\ y=(|x|=|y|)$, where $|x|$ denotes the absolute value of an integer $x$. Then the first property above is true, and the second is false (be sure to check this yourself).

When mathematicians write $\exists ! x,y:p\ x\ y$, they really mean the second interpretation, so this is what SmallCheck implements (although the first interpretation would be much easier to implement).

Higher-order properties

Both SmallCheck and QuickCheck have a ==> operator, which tests an implication. The left argument of ==> has type Bool, and it’s usually a statement that the generated arguments satisfy some side condition (e.g. generated programs are valid).

In this new version of SmallCheck, the left argument of ==> can be an arbitrary property, with its own quantifiers etc. Here’s how we could test an isPrime function:

import Test.SmallCheck
import Test.SmallCheck.Series

isPrime :: Integer -> Bool
isPrime _ = True

main = 
  smallCheck 5 {- 5 is the testing depth -} $
    \(Positive n) ->
      (exists $ \(Positive d) -> d /= 1 && d /= n && n `mod` d == 0) ==> not (isPrime n)

Unsurprisingly, the result will be

Failed test no. 4.
there exists 4 such that
  condition is false

Ordered testing

SmallCheck has just got more efficient and less annoying!

Previously, if you wanted to get the minimal counterexample, you used the function smallCheck which tested the property first with depth 0, then with depth 1 and so on — up to the limit that you told it.

The reason was that the Serial instance of e.g. Integer generated the following integers for depth 3:

[-3,-2,-1,0,1,2,3]

They are tried in exactly this order, and, if all of them were counterexamples, the user would be presented with -3 as the first one.

Now the instances generate examples in the order of increasing depth, e.g.

[0,1,-1,2,-2,3,-3]

so there’s no need to try multiple depths.

Better reporting

The way SmallCheck reports its findings is now greatly improved, especially in some complex cases.

For example, consider the following property:

smallCheck 5 $ existsUnique $ \x -> (forAll $ \y -> x * y >= 0) ==> (forAll $ \y -> x * y == (0 :: Integer))

Here’s what SmallCheck 0.6.2 reports (or, to be precise, would report if it supported higher-order properties):

Depth 0:
  Completed 1 test(s) without failure.
Depth 1:
  Failed test no. 12. Test values follow.
  non-uniqueness
  -1
  0

And this is the output of SmallCheck 1.0:

Failed test no. 12.
there are at least two arguments satisfying the property:
  for 0
    condition is true
  for 1
    property is vacuously true because
      there exists -1 such that
        condition is false

Required type annotations

Because the Testable class is now multi-parameter (parameterised on the type and the monad), ambiguous types that involve this class do not get defaulted. To put it simply, most non-trivial properties now require some type annotations.

This turned out to be a very good thing for SmallCheck (which I couldn’t envision). Defaulting for properties often works in an unexpected way. For example, here’s what an older SmallCheck would tell you if you asked it whether all things are the equal:

> depthCheck 5 $ \x y -> x == y
Depth 5:
  Completed 1 test(s) without failure.

The result is unexpected, but at least we get some clue from the fact that there was just one test: the type of the variables got defaulted to ()!

Alas, QuickCheck doesn’t give us even this hint:

> quickCheck $ \x y -> x == y
+++ OK, passed 100 tests.

Luckily, with the new SmallCheck you actually get a type error:

<interactive>:10:26:
    No instance for (Eq a0) arising from a use of `=='
    The type variable `a0' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)

And indeed, the problem is easy to fix once you know it’s there:

> smallCheck 5 $ \x y -> x == (y :: Integer)
Failed test no. 2.
there exist 0 1 such that
  condition is false

Simpler interface

I took this chance to clean the API and make it more orthogonal.

Previously SmallCheck had functions like

forAllElem :: (Show a, Testable b) => [a] -> (a -> b) -> Property

which did three things at once:

made the variable universally quantified
converted the list to a depth-independent series
quantified the variable over that series

No wonder the API suffered from the combinatorial explosion. Now these things are all done separately:

generate :: (Depth -> [a]) -> Series m a

can be used to turn a list into a series;

over :: (Monad m, Show a, Testable m b) => Series m a -> (a -> b) -> Property m

sets the domain of quantification; and

forAll :: Testable m a => a -> Property m

sets the quantification context.

Generic uncurry

Tue, 29 Jan 2013 20:00:00 +0000

For the next version of SmallCheck I needed a generic uncurry function in Haskell. To my surprise, I couldn’t google anything simple and ready to use.

So here it is, for future reference.

{-# LANGUAGE TypeFamilies, DefaultSignatures, UndecidableInstances #-}
class Uncurry a where
  type Uncurried1 b a
  type Uncurried1 b a = b -> a

  unc1 :: (b -> a) -> Uncurried1 b a
  default unc1 :: (b -> a) -> (b -> a)
  unc1 = id

  type Uncurried a
  type Uncurried a = a

  unc :: a -> Uncurried a
  default unc :: a -> a
  unc = id

instance Uncurry y => Uncurry (x -> y) where
  type Uncurried1 z (x -> y) = Uncurried1 (z, x) y
  unc1 = unc1 . uncurry

  type Uncurried (x -> y) = Uncurried1 x y
  unc = unc1

You also need to make the final type an instance of Uncurry. For example, if you need to uncurry Int -> Bool -> Double, Double has to be an instance of Uncurry. But thanks to default method signatures and associated type synonym defaults, it is as easy as

instance Uncurry Double

Uncurry1 and unc1 are implementation details. The useful parts are Uncurry, which converts the type to its uncurried version, and unc, which uncurries the function itself.

After loading the code in ghci, you can see that it works by:

*Uncurry> :kind! Uncurried (Int -> Bool -> Char -> Double)
Uncurried (Int -> Bool -> Char -> Double) :: *
= ((Int, Bool), Char) -> Double

*Uncurry> :t unc $ \x y z -> (x + y * z :: Double)
unc $ \x y z -> (x + y * z :: Double)
  :: ((Double, Double), Double) -> Double

*Uncurry> unc (\x y z -> (x + y * z :: Double)) ((1,2),3)
7.0

I tested this with GHC 7.6, but I think it should work with GHC 7.4 as well.

Subtractable values are torsors

Tue, 08 Jan 2013 20:00:00 +0000

A group is an abstraction for things that can be «added» and «subtracted» (in some sense).

Most of the groups that programmers deal with are numeric, although some interesting groups arise in graphics and cryptography.

If we only require that things can be added, but not subtracted, we get semigroups and monoids (the difference is that a monoid also has a «zero» element). These are plentiful, with applications ranging from diagrams to regular expressions.

The reason is, of course, that addition is very natural — we can consider any way to combine objects together an addition as soon as it is associative — while subtraction, the inverse of addition, often doesn’t make sense.

Dates

But there are also things that can be subtracted, but not added. The simplest example is dates. You can subtract two dates to get the time interval between them, but you cannot add two dates. It doesn’t make sense to ask what today plus tomorrow is. You could try to add the numeric values corresponding to the dates and convert the answer back to a date, but the result would depend on whether you count from the start of the Common Era, the Unix epoch, or something else.

While adding two dates is not possible, it is possible to add a time interval to a date («five days from today»). This suggests that we should not confound dates and time intervals — they are different types of values.

For example, in the Haskell time package, there are types UTCTime and NominalDiffTime, and the following functions:

addUTCTime :: NominalDiffTime -> UTCTime -> UTCTime
diffUTCTime :: UTCTime -> UTCTime -> NominalDiffTime

Time intervals can also be added and subtracted, as witnessed by the Num NominalDiffTime instance.

Ask mathematics

What is the mathematical notion that characterizes dates? It should involve two sets: the set $V$ of values and the set $D$ of value differences.

The set $D$ must form an additive group, so that we can add and subtract the differences. (The word «additive» means that we assume that the group operation, identity element and inverse operation are called $+$, $0$ and $−$ respectively. In group theory it is more common to call them $\cdot$, $1$ and ${}^{-1}$.)

Thus we have the “zero” difference $0\in D$ and operations $+\colon D\times D\to D$, $-\colon D\to D$ which satisfy the usual group laws:

$+/0$ (identity element): $0+x=x+0=x$
$+/+$ (associativity): $(x+y)+z=x+(y+z)$
$+/−$ (inverse element): $x+(−x)=(−x)+x=0$

(We can write $x−y$ as a shorthand for $x+(−y)$.)

We do not require $+$ to be commutative to preserve generality, even though it is commutative for our current example.

We need to be able to add a difference to a value. Thus, we need a function $\mathrm{add}:D\times V\to V$. We would expect it to play well with the group structure of $D$:

$\mathrm{add}/0$: $\mathrm{add}(0,v)=v$
$\mathrm{add}/+$: $\mathrm{add}(x+y,v)=\mathrm{add}(x,\mathrm{add}(y,v))$

An alternative view on these laws is that the curried version of $\mathrm{add}$, $\mathrm{curry}(\mathrm{add}):D→(V→V)$, is a group homomorphism from $D$ to the group of bijections on $V$.

The mathematical abstraction corresponding to our $\mathrm{add}$ function is an action of group $D$ on the set $V$. But it doesn’t yet capture the fact that we can subtract two values to get a difference. For that we need a function $\mathrm{diff}:V×V→D$ inverse to $\mathrm{add}$:

$\mathrm{diff}/\mathrm{add}$: $\mathrm{diff}(\mathrm{add}(x,v),v)=x$
$\mathrm{add}/\mathrm{diff}$: $\mathrm{add}(\mathrm{diff}(u,v),v)=u$

If such a function exists, the set $V$ is called a torsor over the group $D$.

Thus, it is the mathematical notion of torsor that characterizes subtractable values.

Conal Elliott defines torsors in Haskell under the name of affine spaces.

Other examples

Points

Another simple example of a torsor is points. In linear algebra we get used to conflating points and vectors, because we consider vectors originating at zero.

But the geometrical intuition suggests that vectors do not have to be tied to zero. A vector is a directed segment which can be freely translated (shifted) on the plane (or in the space).

We can subtract two points to get a vector from one point to another. We can add a vector to a point, by translating it so that its origin becomes the given point. The end point of the vector then becomes the result of addition.

Vectors form an additive group, and points form a torsor over the vectors.

Like with dates, results of torsor operations on points and vectors do not depend on (and do not require the existence of) a coordinate system, while the «sum» of two points will change if you change the origin.

Files

There are also some examples that look very much like torsors, but do not satisfy all the laws.

One of such examples is files. The difference of two files is what we call a patch, or a diff. We can diff two files and apply the resulting patch to a third file («cherry-picking»).

Although patches do not form a group, they can be modelled as inverse semigroups. This motivates us to consider torsors based on inverse semigroups rather than on groups.

Haskell dialects

Haskell dialects can be considered as a torsor-like structure over Haskell extensions. We can add extensions to languages:

\[\mathrm{add}(\mathrm{RankNTypes}+\mathrm{GADTs}−\mathrm{MonomorphismRestriction}, \mathrm{Haskell2010}).\]

We can also subtract two languages to find out the difference between them in terms of extensions: \[ \begin{align*} \mathrm{diff}(\mathrm{Haskell2010}, \mathrm{Haskell98}) & = \mathrm{DoAndIfThenElse} \\ & +\mathrm{PatternGuards} \\ & +\mathrm{ForeignFunctionInterface} \\ & +\mathrm{EmptyDataDecls} \\ & −\mathrm{NPlusKPatterns}. \end{align*} \]

Extensions do not form a group. The easiest way to see that is to observe that extensions are idempotent, i.e. $X+X=X$ for any extension $X$. However, in a group there is only one idempotent element, zero.

Extensions do form an inverse semigroup, though. Thus we again arrive at inverse semigroup torsors.

Acknowledgements

A recent discussion with Niklas Broberg about Haskell extensions motivated me to explore these concepts. His HIW’12 lightning talk on Haskell Modular Mindset is also relevant.

Oleksandr Manzyuk kindly pointed me to the concept of a torsor.

Monads in dynamic languages

Wed, 26 Dec 2012 20:00:00 +0000

Are monads possible outside of Haskell?

Of course, it depends on what you call a monad.

The definition that is usually accepted in the Haskell context — that a monad is an instance of the particular type constructor class, Monad, with two methods satisfying three laws — is very much tied to Haskell’s language features — most importantly, types, type constructors and type constructor classes. Hence, this or similar notion of monad can only exist in a language rather close to Haskell.

If we drop the type constructor class requirement, then a monad is a triple consisting of a type constructor M and two polymorphic functions

return :: forall a . a -> M a
(>>=) :: forall a b . M a -> (a -> M b) -> M b

This is the definition originally used by Wadler in The essence of functional programming. By this definition, Monads can be implemented in other Hindley-Milner based languages, like Miranda and ML.

But what if we take this to extreme? What if we consider a dynamically typed functional language?

A dynamically typed language is nothing else than a static language with a single type, which we shall call here U. Thus, the signatures of >>= and return become

return :: U -> U
(>>=) :: U -> (U -> U) -> U

(We could as well say that >>= and return both have type U — remember, there’s only one type!) The functions themselves are still presumed to satisfy the usual laws.

This has been done many times before, most recent example being Douglas Crockford’s keynote at YUIConf. Douglas’s presentation has received a lot of criticism (and I’m not going to defend him), but some critics even have expressed the doubt that these are not «genuine» monads.

To settle this, let’s turn to the category theory — after all, that’s where monads come from. Haskell’s monads are category theory monads, where the type constructor is considered as an endofunctor on Hask, the category of Haskell types and functions.

What changes when we go from Haskell to a unityped language is that all the diversity of Haskell types collapses into the single type, U. That doesn’t stop it from forming a category, though. Because it has a single object, we don’t need a type constructor to represent the functorial action on objects — it’s always identity. We only need to know the functorial action on arrows (called fmap in Haskell) — but it is already determined by return and >>=.

To summarize, there doesn’t seem to be any reason to think that monads are somehow impossible in dynamic functional languages. And, although Haskellers usually consider monads as something closely tied to types, in a dynamic language all you need to define a monad is to define a pair of functions satisfying the monad laws.

Surprises of the Haskell module system (part 1)

Tue, 25 Dec 2012 20:00:00 +0000

While the basics of the Haskell module system are easy to understand, there are certainly interesting corner cases and non-obvious (mis)features.

All of these are documented in the Haskell report, but the tricky details may be hard to notice. Another good read on this subject is A Formal Specification for the Haskell 98 Module System by Iavor S. Diatchki, Mark P. Jones, and Thomas Hallgren.

Empty module header

What happens when the export list is omitted, like in the following example?

module M where

foo = 42

This one is easy: all the locally defined types, classes and values are exported.

We might assume that something similar happens when we omit the module header altogether, like if we put

foo = 42

in a file by itself.

Contrary to this intuition, and according to the standard, when you omit the module header, it is assumed to be module Main(main) where.

Sometimes you want to check whether a piece of Haskell code compiles. You may put it into a file and try ghc test.hs. But even if the code itself doesn’t contain errors, you’ll get a message

test.hs:1:1: The function `main' is not defined in module `Main'

The message may seem confusing at first: what’s the module Main it’s talking about, and why do I need this main function? Well, now you know the answer.

Module re-exports

You probably know that you can re-export a whole module from your own module. The syntax is

module M (module N) where
...

The semantics of this re-export is a bit tricky, though. For example, what can we say about the following module?

module M (module Data.List) where
import qualified Data.List

You’d probably think that module M now exports everything that is exported by Data.List. Not at all! The standard says (emphasis is mine):

The form module M names the set of all entities that are in scope with both an unqualified name e and a qualified name M.e.

Since we imported Data.List qualified, nothing from it is in scope under an unqualified name. So, M doesn’t export anything. Right? Not quite.

The last thing we need to observe here is that although the single module imported by M explicitly is Data.List, there’s always an implicit import — Prelude. And these two share quite a lot in common — functions like foldr, map, length etc. All these satisfy the criterion above — they are in scope with both unqualified and qualified names, although these names come from different modules.

To summarize, the module M defined above will export exactly the intersection of Prelude and Data.List, plus instances exported by either of these modules.

Haskell recipe: reading list of lines

Thu, 25 Oct 2012 20:00:00 +0000

import Prelude hiding (readList)
import Control.Applicative
import Control.Monad
import Control.Monad.Trans
import Control.Monad.Trans.Maybe
import Data.Maybe

Suppose we need to ask the user to enter a list of lines, terminated by an empty line. The usual way to do it is through recursion:

readList :: IO [String]
readList = do
  l <- getLine
  if null l
    then return []
    else (l :) <$> readList

This is fine, but can we do better?

A good style in Haskell is to replace explicit recursion with high-level recursion combinators where it is possible and doesn’t harm readability.

What is the pattern here? Those who are familiar with parser combinator libraries will easily recognize the many combinator. It tries to recognize as many items as possible and returns the list of recognized items (in this case successful recognition means that we got a non-empty line).

A generalization of the many combinator works for any applicative functor which is an instance of the Alternative class:

many :: Alternative f => f a -> f [a]

The IO functor is not an instance of Alternative, but we can fix it by layering a MaybeT transformer on top of it. MaybeT will be our means to distinguish failure from success.

readList :: IO [String]
readList = fmap (fromMaybe []) $ runMaybeT $ many $ do
  l <- lift getLine
  guard $ not $ null l
  return l

The inner do-block is the code which reads one line, and, depending on whether that line is null, returns it or signals about the failure.

Some more details about what’s going on there.

lift getLine lifts getLine from IO String to MaybeT IO String by assuming that it always succeeds (in MaybeT sense).
guard is a function from the MonadZero class that checks the condition and, if it does not hold, declares failure. It is somewhat similar to an assert function, except it is used for control flow rather than debugging purposes.
Finally, if guard did not abort the computation, we get to the last line that simply returns the line we’ve just read.

(If you want to learn more about how monad transformers work, please refer to Composing monadic effects.)

The outer code does boring stuff: runMaybeT transforms MaybeT IO String to IO (Maybe String) (which is the same thing but newtyped), and fromMaybe interprets Nothing as the empty list.

It is interesting that only the first line of readList definition gives us a hint that MaybeT works here. The rest of the code just uses overloaded functions many, lift, guard and return.

Was the rewrite worth it? It is debatable, since a function did get a bit more complicated. But in my opinion, the answer is yes — we got rid of explicit recursion, and the idea of the function became more clear.

Reasoning about space usage in Haskell

Sun, 08 Apr 2012 20:00:00 +0000

This article describes why it’s hard to reason about space usage of Haskell programs.

Operational semantics

The standard (“The Haskell Report”) does not specify an operational semantics for Haskell. Thus, in order to talk about space usage of Haskell programs, we need to fix a particular semantics. A good choice is the STG semantics as defined in the fast curry paper. First, it is formally defined; second, it has significant practical value, since it is used in GHC, the de-facto standard Haskell compiler.

Note, however, that the STG operational semantics is defined for STG programs, not Haskell programs (STG defines its own small functional language). It’s not always obvious what STG program corresponds to a given Haskell program because of numerous optimizations performed by the compiler, but GHC allows to see the resulting STG code using the -ddump-stg flag.

Operational properties are context-dependent

In eager languages, usually we can easily express time and space complexity, e.g. using O-notation.

In Haskell, operational properties are largely context-dependent, and there’s no existing framework to express them.

Laziness

One source of context dependency is a direct consequence of lazy evaluation: the function result may be evaluated to different extents by different callers.

For example, it may seem that [1..n] requires O(n) memory, but when called from take m [1..n], it takes only O(min(m,n)) memory.

Garbage collection

Another source of context dependency is interaction between laziness and automatic memory management.

For example, the following two functions look equivalent in terms of space usage (although the second one is somewhat less time-efficient):

f1 list = foldl' (\(!a,!b) x -> (a + x, b * x)) (0, 1) list

f2 list = (foldl' (\a x -> a + x) 0 list, foldl' (\b x -> b * x) 1 list)

It would seem that these functions run in constant space. Of course, by “constant space” we really mean “constant additional space” – we do not count the space occupied by the list itself, because the function doesn’t create the list. Well, in a lazy language it does!

The difference becomes clear if we consider f1 [1..n] and f2 [1..n]: the former indeed runs in constant space, while the latter requires linear space.

The reason is that f1 can process the list incrementally. It demands the next value from the list, updates its accumulators and throws the value away.

In f2, the first fold evaluates the list, but it has to be kept in memory to be used for the second fold. As a result, by the time the first fold is evaluated, the whole list is evaluated and none of it is garbage collected.

Thunks

It’s not enough to track how much space our data occupies. Another kind of heap inhabitants is thunks. When more thunks are created than necessary, we have a thunk leak.

By the way, how many thunks may be necessary? Are there examples where having more than O(1) thunks at some point of program execution is the best solution to the problem? I don’t know yet.

In any case, we need to be able to reason about the number of thunks that are created by a function.

To complicate things further, creation of thunks depends on how the function result is evaluated. For example, a function returns a (lazy) tuple. We force the first element of that tuple. During the evaluation, everything that is needed to evaluate that first element is evaluated immediately, but values needed only for the second element are stored in thunks.

If we evaluated the second element, the picture would be dual, and the number of thunks probably would be different.

Conclusions

Currently, if we want to reason about space usage of Haskell programs, we need:

study the source code of each function. A function documentation typically includes its type, denotational semantics and perhaps some O-figures, but that’s not enough, as we’ve seen;
for each function, understand how its result is evaluated by all callers.

Clearly, this breaks abstraction and modularity.

In practice, GHC’s strictness analyser is very smart, so most of the time we don’t notice any problems. Still, it’s very important to have a framework in which we could analyse and document functions’ operational behaviour in a modular and composable way.

Composing monadic effects

Mon, 02 Jan 2012 20:00:00 +0000

In Haskell, monad transformers are used to combine effects provided by multiple monads.

Boring example: Writer, Reader, State

For instance, if we want to have a read-only environment plus a write-only logging facility, we can start with the Reader monad and add a logging facility to it with the WriterT monad transformer:

type MyMonad w r a = WriterT w (Reader r) a

Or, we could start with the Writer monad and put the ReaderT transformer on top of it:

type MyMonad w r a = ReaderT r (Writer w) a

In the end, it doesn’t matter which way we choose, as both are isomorphic to

type MyMonad w r a = r -> (a, w)

If we want to have a state as well, we can plug in the StateT transformer anywhere in the stack, and again the order doesn’t matter. The resulting monad is now isomorphic to the standard RWS (Reader-Writer-State) monad.

So, we say about those transformers that they commute (recall that Reader is just the ReaderT transformer applied to the Identity monad).

More interesting example: Either + Writer

Not every two transformers commute, though.

Suppose we want to write log messages, plus to be able to abort the computation.

There are two quite different ways to compose these effects.

There’s

type MyMonad e w a = ErrorT e (Writer w) a

isomorphic to (Either e a, w), and then there’s

type MyMonad e w a = WriterT w (Either e) a

isomorphic to Either r (a, w).

We can already see the difference from the types. In the first case, the w is there independently of the value of Either. In the second case, if the computation failed, the whole thing is Left msg, and we don’t get our log w back.

Let’s try the following to confirm our expectations:

> runWriter $ runErrorT $ tell "foo" >> throwError "err" >> tell "bar"
(Left "err","foo")
> runErrorT $ runWriterT $ tell "foo" >> throwError "err" >> tell "bar"
Left "err"

Developing intuition

When reasoning about a stack of monad transformers, it’s useful to keep in mind that a transformer knows nothing about its inner monad. It has to work “inside” that monad. It has no ability to alter or cancel the effects that happened before in that monad. (It can prevent those effects from happening, though – see tell "bar" above.)

In the first example above, which corresponds to ErrorT e (Writer w), the Error monad had to work inside the Writer monad. It simply had no way to tell the writer monad to “forget” what had been told to it before.

In the second example, corresponding to WriterT w (Either e) a, the main monad is now Either e. If it decides to fail, it will return just the error, and nothing else.

The same logic answers the question why there’s no IOT, an IO monad transformer. Imagine this:

do
    targetHit <- launchMissiles
    when targetHit $
        throwError "oops I did it again"

Since throwError happens in the base monad, the whole thing is just Left "oops I did it again" – nothing mentions any IO. But what about the missiles?

If this shallow explanation is not enough to make you comfortable with transformers, I’d advise to read (or even to write) the implementation of some standard transformers.

Even more interesting example: Parsec + State

The Parsec monad already allows us to have our own state, but let’s see how we can add one independently.

Again, we have two ways to layer ParsecT and StateT transformers on top of Identity: ParsecT s () (State u) a and StateT u (Parsec s ()) a. Is there a difference?

Recall that Parsec is a backtracking monad. It can execute one branch, fail, and then execute another branch. Do we observe the state changes that happened in the aborted branch?

This will be our test code:

((put 1 >> char 'b') <|> char 'a') >> get

We’ll try to run it with the input "a" and initial state 0. First, Parsec executes the left branch of <|>, fails, and proceeds with the right branch. If the state is “global”, then the final result will be 1. If the state is subject to backtracking, then we’ll see 0.

In the ParsecT s () (State u) a case, the base monad is State. ParsecT works inside that monad and cannot revert the effects that happened there. (It may seem that it could remember the state of a branch using get and then restore it using put; but a monad transformer has to be polymorphic in the underlying monad and thus cannot rely on existence of get and put.)

And indeed, the code

import Text.Parsec
import Control.Monad.State

main = print $ flip evalState 0 $ runParserT p () "-" "a"
    where p = ((put 1 >> char 'b') <|> char 'a') >> get

prints Right 1.

Let’s now try the second option: layering StateT on top of Parsec. We cannot use the code above as is, because types do not match: e.g. char 'b' has type Parsec s () Char, but we need StateT u (Parsec s ()) Char. Such an upgrade is done by the lift function.

Sometimes, however, we need to “downgrade” computations from StateT u (Parsec s ()) a to Parsec s () a.

For example, to implement a <|> operator of type

(<|>) :: StateT u (Parsec s ()) a -> StateT u (Parsec s ()) a -> StateT u (Parsec s ()) a

we need first to downgrade the arguments to plain Parsec computation, invoke Parsec’s own <|> and then upgrade the result back to StateT.

import Text.Parsec as P
import Control.Monad.State

main = print $ runParser (evalStateT p 0) () "-" "a"
  where
    p = ((put 1 >> char 'b') <|> char 'a') >> get

    char = lift . P.char

    a <|> b = StateT $ \s ->
        runStateT a s P.<|> runStateT b s

Notice how we explicitly run both branches of <|> with the same state. It’s no surprise now that the state will be “backtracked” and the result will be Right 0. By the way, this is the same result as would be achieved using Parsec’s internal state.

A	\(\mathscr{A}\)	N	\(\mathscr{N}\)
B	\(\mathscr{B}\)	O	\(\mathscr{O}\)
C	\(\mathscr{C}\)	P	\(\mathscr{P}\)
D	\(\mathscr{D}\)	Q	\(\mathscr{Q}\)
E	\(\mathscr{E}\)	R	\(\mathscr{R}\)
F	\(\mathscr{F}\)	S	\(\mathscr{S}\)
G	\(\mathscr{G}\)	T	\(\mathscr{T}\)
H	\(\mathscr{H}\)	U	\(\mathscr{U}\)
I	\(\mathscr{I}\)	V	\(\mathscr{V}\)
J	\(\mathscr{J}\)	W	\(\mathscr{W}\)
K	\(\mathscr{K}\)	X	\(\mathscr{X}\)
L	\(\mathscr{L}\)	Y	\(\mathscr{Y}\)
M	\(\mathscr{M}\)	Z	\(\mathscr{Z}\)

Roman Cheplyaka

Рейтинги шахових розрядів і звань

Alice and Bob flipping coins puzzle

Sum of random cards puzzle

Solution

Bonus: calculating the probabilities

AeroPress Go measurements. What volume do AeroPress marks correspond to?

Chamber volume

Coffee grounds weight

Brewed coffee yield

How much current does an active bass guitar draw from a 9V battery?

A data structure puzzle

Understanding modern Linux routing (and wg-quick)

Routing tables (ip route)

Routing policies (ip rule)

Understanding wg-quick

Rule 32764

Rule 32765

My podcast editing workflow in REAPER

Create the project; import and align the tracks

Normalize

Dynamic split

Edit at item boundaries

Re-listen and make final edits

StateT vs. IORef: a benchmark

Laptop vs. desktop for compiling Haskell code

Specs

Methodology

Results

Conclusions

Acknowledgments

system2 in R considered inadequate

Sort directories by the newest file

Don't buy tempered glass PC cases

Avoid the months() and years() functions from lubridate

If anyone drove a time machine, they would crash

Configuring PulseAudio for Audient EVO 4

How I integrate ghcid with vim/neovim

ghcid setup

vim/neovim setup

Which dynamic mic should you buy?

Visualizing Haskell heap profiles in 2020

hp2ps

hp2pretty

hp/D3.js

Perl & R

hp2any

Compile and link a Haskell package against a local C library

Binomial coefficients via integer division

Does micro-USB cable affect charging speed?

Break on NaN in gdb

Significance level vs. the type I error chance, or how to interpret conditionals

Bioinformatics events in Europe in 2020

Long-term reminders

A blind microphone comparison

A curious associativity of the <$> operator

Decompose ContT

Motivation

A beautiful but useless solution

A more practical solution

Why would you use ContT?

llvm-hs

Reading from a list of files

Allocating a linked list

How (not) to convert CDouble to Double

Fix a torrent file encoding

Lazy validation

Why lazy validation?

An implementation

Comparing different implementations

Surprises of the Haskell module system (part 2)

Exporting constructors without data type

Exporting fields without constructor

Hidden types and type synonyms

Self-exporting module

A better noise gate

Not enough reduction

Too much reduction

A better noise gate

Choosing the parameters

Routing tables (`ip route`)

Routing policies (`ip rule`)