<?xml version="1.0"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:planet="http://planet.intertwingly.net/" xmlns:indexing="urn:atom-extension:indexing" indexing:index="no"><access:restriction xmlns:access="http://www.bloglines.com/about/specs/fac-1.0" relationship="deny"/>
  <title>Planet Haskell</title>
  <updated>2026-04-12T02:04:52Z</updated>
  <generator uri="http://intertwingly.net/code/venus/">Venus</generator>
  <author>
    <name>Haskell Admin Team</name>
    <email>planet@haskell.org</email>
  </author>
  <id>http://planet.haskell.org/atom.xml</id>
  <link href="http://planet.haskell.org/atom.xml" rel="self" type="application/atom+xml"/>
  <link href="http://planet.haskell.org/" rel="alternate"/>

  <entry>
    <id>tag:blogger.com,1999:blog-9757377.post-3700094658306171494</id>
    <link href="https://wadler.blogspot.com/feeds/3700094658306171494/comments/default" rel="replies" title="Post Comments" type="application/atom+xml"/>
    <link href="https://www.blogger.com/comment/fullpage/post/9757377/3700094658306171494" rel="replies" title="0 Comments" type="text/html"/>
    <link href="https://www.blogger.com/feeds/9757377/posts/default/3700094658306171494" rel="edit" type="application/atom+xml"/>
    <link href="https://www.blogger.com/feeds/9757377/posts/default/3700094658306171494" rel="self" type="application/atom+xml"/>
    <link href="https://wadler.blogspot.com/2026/04/wet-sidewalks-and-odd-numbers.html" rel="alternate" title="Wet Sidewalks and Odd Numbers" type="text/html"/>
    <title>Wet Sidewalks and Odd Numbers</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p/><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEij_CIKOMnrRH3Aa4_K7emngkVT1D39qIa_4v8foa-HMuWLPQMR8BL7kHbLbO1luZz7XK_8WKvHatJctvdSqo6LBXgKtENq55oDuLCgCBD8HdFXlup_VPR66psESR0Kjh9YQh6R0FXNPfRmBqWg5K33tmPdM28ArIxzvbEtC2VWHMpyFV2OsUwR" style="margin-left: 1em; margin-right: 1em;"><img alt="" height="334" src="https://blogger.googleusercontent.com/img/a/AVvXsEij_CIKOMnrRH3Aa4_K7emngkVT1D39qIa_4v8foa-HMuWLPQMR8BL7kHbLbO1luZz7XK_8WKvHatJctvdSqo6LBXgKtENq55oDuLCgCBD8HdFXlup_VPR66psESR0Kjh9YQh6R0FXNPfRmBqWg5K33tmPdM28ArIxzvbEtC2VWHMpyFV2OsUwR=w640-h334" width="640"/></a></div><br/>Phil Crissman <a href="https://philcrissman.net/posts/wet-sidewalks-and-odd-numbers/">explains</a> Propositions as Types with a dialogue between Achilles and the Tortoise, in the style of Douglas Hofstadter (who in turn was inspired by Lewis Carrol). Lambda Man makes an appearance.<p/></div>
    </content>
    <updated>2026-04-09T15:22:05Z</updated>
    <published>2026-04-09T15:22:00Z</published>
    <category scheme="http://www.blogger.com/atom/ns#" term="Logic"/>
    <category scheme="http://www.blogger.com/atom/ns#" term="Programming Languages"/>
    <author>
      <name>Philip Wadler</name>
      <email>noreply@blogger.com</email>
      <uri>http://www.blogger.com/profile/12009347515095774366</uri>
    </author>
    <source>
      <id>tag:blogger.com,1999:blog-9757377</id>
      <category term="Politics"/>
      <category term="Programming Languages"/>
      <category term="Functional Programming"/>
      <category term="Computing"/>
      <category term="Scotland"/>
      <category term="Independence"/>
      <category term="Haskell"/>
      <category term="Yes!"/>
      <category term="Academia"/>
      <category term="UK"/>
      <category term="Edinburgh"/>
      <category term="Science"/>
      <category term="Cycling"/>
      <category term="Graphics"/>
      <category term="Types"/>
      <category term="Mathematics"/>
      <category term="US"/>
      <category term="Logic"/>
      <category term="Covid-19"/>
      <category term="Security"/>
      <category term="Web"/>
      <category term="Comics"/>
      <category term="EU"/>
      <category term="Israel"/>
      <category term="Theory"/>
      <category term="Climate Change"/>
      <category term="Comedy"/>
      <category term="Blockchain"/>
      <category term="Education"/>
      <category term="Writing"/>
      <category term="AI"/>
      <category term="IOHK"/>
      <category term="University"/>
      <category term="Agda"/>
      <category term="Cryptocurrency"/>
      <category term="Developers"/>
      <category term="Internet"/>
      <category term="SIGPLAN"/>
      <category term="Scala"/>
      <category term="Status"/>
      <category term="ACM"/>
      <category term="Concurrency"/>
      <category term="DSL"/>
      <category term="Databases"/>
      <category term="Dynamic and Static Typing"/>
      <category term="F#"/>
      <category term="Green"/>
      <category term="JavaScript"/>
      <category term="Lego"/>
      <category term="Privacy"/>
      <category term="Recursion"/>
      <category term="Session Types"/>
      <category term="Strange Loop"/>
      <category term="Category Theory"/>
      <category term="Formal Methods"/>
      <category term="Java"/>
      <category term="Object-Oriented"/>
      <category term="Technology"/>
      <category term="Architecture"/>
      <category term="Copyright"/>
      <category term="Distributed Computing"/>
      <category term="Europe"/>
      <category term="Productivity"/>
      <category term="Racket"/>
      <category term="Science Fiction"/>
      <category term="Theatre"/>
      <category term="BLM"/>
      <category term="Brexit"/>
      <category term="Communication"/>
      <category term="DRM"/>
      <category term="Environment"/>
      <category term="Erlang"/>
      <category term="Finance"/>
      <category term="Gender"/>
      <category term="Palestine"/>
      <category term="BDS"/>
      <category term="Books"/>
      <category term="Cinema"/>
      <category term="Japan"/>
      <category term="Net Neutrality"/>
      <category term="Open Access"/>
      <category term="Pyret"/>
      <category term="Scheme"/>
      <category term="Sweden"/>
      <author>
        <name>Philip Wadler</name>
        <email>noreply@blogger.com</email>
        <uri>http://www.blogger.com/profile/12009347515095774366</uri>
      </author>
      <link href="https://wadler.blogspot.com/feeds/posts/default" rel="http://schemas.google.com/g/2005#feed" type="application/atom+xml"/>
      <link href="https://www.blogger.com/feeds/9757377/posts/default?alt=atom" rel="self" type="application/atom+xml"/>
      <link href="https://wadler.blogspot.com/" rel="alternate" type="text/html"/>
      <link href="http://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <link href="https://www.blogger.com/feeds/9757377/posts/default?alt=atom&amp;start-index=26&amp;max-results=25" rel="next" type="application/atom+xml"/>
      <title>Wadler's Blog</title>
      <updated>2026-04-09T15:22:05Z</updated>
    </source>
  </entry>

  <entry>
    <id>tag:blogger.com,1999:blog-11295132.post-5093748306360724234</id>
    <link href="http://blog.sigfpe.com/2026/03/what-does-it-take-to-be-hero-revisited.html" rel="alternate" type="text/html"/>
    <title>"What does it take to be a hero?" revisited</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><h2>Biased posteriors the hard way</h2>
<p>I was <a href="http://blog.sigfpe.com/2024/09/what-does-it-take-to-be-hero-and-other.html">previously</a> interested to see how die rolls in an RPG appear when conditioned on you having survived an unlikely situation. As might have been predicted, if the die rolls contribute to that survival in a largely additive way, for example by being damage scored against a large opponent, then the posterior distribution of the rolls looks exponentially tilted.
</p>

<h2>Biased posteriors an easier way</h2>
<p>
But the only virtue of the brute force Monte Carlo method I used was that it was easy to code. It's computationally wasteful. So I wrote a much more performant DSL in Python which I have put on <a href="https://github.com/dpiponi/dice-nine">github</a>.
</p>

<p>
  It uses <code>numpy</code> to achieve tolerable numerical performance, but in addition it uses two techniques beyond brute force to make it usable.
</p>
<p>
One challenge with a probabilistic language is to manage state. 
</p>
First there's state "in the past": If you're computing probabilities that are sums of many large intermediate states, for example 100 die rolls, you run the risk of running foul of combinatorial explosion.
<p>
If you write a loop like:
</p>
<code>
for i in range(100):
  t += d(6)
</code>
<p>
you want to be sure that the <code>+=</code> operation erases history (ie. previous values of <code>t</code>) so you aren't tracking all <code>6^100</code> individual histories. In this case it's easy but in other cases it might not be so obvious that you have unneeded state lying around. So the code has a simple (and incomplete) backward liveness pass to insert deletions of data that won't be used again. Whenever state is deleted, you can merge histories that are now indistinguishable.
</p>

<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOVKs7SPg4Va84Rq4g6pL3aTuEYti8GdW6zXtxxcBGKXSB8ywFJ62RVHxoz9Y3hfroRPmtykYIl1mFrgPzmQNy63CMaXUshKHUOkLuQ1lK5KAA3gu7BeREiUrXlf2kBlV0HAOMbLk0gVkZJ97HpvBgb_bw1MXxfWx7Qpz2hFH99wOXOUQk7odzzg/s614/Image%203-30-26%20at%2010.02%E2%80%AFAM.png" style="display: block; padding: 1em 0; text-align: center;"><img alt="" border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOVKs7SPg4Va84Rq4g6pL3aTuEYti8GdW6zXtxxcBGKXSB8ywFJ62RVHxoz9Y3hfroRPmtykYIl1mFrgPzmQNy63CMaXUshKHUOkLuQ1lK5KAA3gu7BeREiUrXlf2kBlV0HAOMbLk0gVkZJ97HpvBgb_bw1MXxfWx7Qpz2hFH99wOXOUQk7odzzg/s600/Image%203-30-26%20at%2010.02%E2%80%AFAM.png"/></a></div>
<p>
And then there is state "in the future": sometimes you'd like to compute probabilities of data structures like lists but materializing a list results in state that can cause combinatorial explosion. So I support Python style generators allowing you to generate data lazily - for example permutations of cards. So we can bring into existence state just before we need it.
</p>

<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm6xN9qhfgiwqjJV5dswflOo3NO6VpMlpRzeb2_KDBxlavBiIxjNfhyphenhyphenxl5gveFkOpTgS8s7DXiteyPic7URhYmJSYhPVWYsDLjPvohZ8k-PAwXkfwlXKGDnHcGO2KiQqbM_6wknzHBc72tyevLg3Nw_-xAji2FPIoPwYDWD5YLrVlM04KmyQ0fFw/s981/image.png" style="display: block; padding: 1em 0; text-align: center;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm6xN9qhfgiwqjJV5dswflOo3NO6VpMlpRzeb2_KDBxlavBiIxjNfhyphenhyphenxl5gveFkOpTgS8s7DXiteyPic7URhYmJSYhPVWYsDLjPvohZ8k-PAwXkfwlXKGDnHcGO2KiQqbM_6wknzHBc72tyevLg3Nw_-xAji2FPIoPwYDWD5YLrVlM04KmyQ0fFw/s600/image.png" width="400"/></a></div>

  <h2>Auto batching</h2>
<p>
  There's another important technique I used: this code uses brute force (though at this point maybe I should stop calling it brute force) so there are many states, each corresponding to a possible set of values for some <code>numpy</code> objects. And we often want to perform a <code>numpy</code> operation for each of these values. We don't need to loop. In many cases a parameterised family of <code>numpy</code> operations is in fact a single <code>numpy</code> operation. This kind of transformation is <a href="https://github.com/tensorflow/probability/tree/v0.23.0/tensorflow_probability/python/experimental/auto_batching">ubiquitous</a> in GPU computing. So we can interpret the following &amp;&amp;D fight, summing over the combinatorialy large number of ways it could happen, in a few seconds:
</p>

<pre>@d9.dist
def f():
    # Brachiosaurus (Monster Manual 1e p. 24)
    hp1 = lazy_sum(36 @ d(8))
    # Tyrannosaurus Rex (Monster Manual 1e p.28)
    hp2 = lazy_sum(18 @ d(8))

    for i in range(14):
        print("round", i)
        if hp1 &gt; 0 and d(20) &gt; 1:
            hp2 = max(0, hp2 - lazy_sum((x for x in 5 @ d(4))))

        if hp2 &gt; 0:
            # Two claws...
            if d(20) &gt; 1:
                hp1 -= d(6)
            if d(20) &gt; 1:
                hp1 -= d(6)
            # ...and a bite
            if d(20) &gt; 1:
                hp1 -= lazy_sum((x for x in 5 @ d(8)))
            hp1 = max(hp1, 0)

    win1 = hp2 == 0
    win2 = hp1 == 0

    return win1, win2
</pre>

<p>
Besides its own test suite I also used a large number of questions on the <a href="https://rpg.stackexchange.com/questions/tagged/anydice">RPG Stack Exchange</a> to build a library of <a href="https://colab.research.google.com/drive/1sOh3Ie_uD2RXVKGoFZ3MZwXN-t9_5KCQ?usp=sharing">examples</a> for testing.
</p>

<h2>Let's work in a general semiring</h2>
<p>
Of mathematical interest: most of the probability computations take place in a semiring. So most of the numerical computing is simply addition and multiplication. When that's all you're doing, there is the well known technique for working with large integers where you work modulo <code>p[i]</code> for some array of primes and only at the end reconstruct your final result using the Chinese remainder theorem. Less will known is that this works for rationals also. (I conjectured this was true, started deriving it myself, and then learnt there are published methods.) This means we can work with exact rational arithmetic using <code>numpy</code> without the need for a bignum library. This code turns out being related to <a href="https://web.cs.ucdavis.edu/~green/papers/pods07.pdf">provenance semirings</a> as it effectively becomes a simple database tracking the provenance of each record.
</p>

<p>
  I originally wrote this code to target GPUs. On my Mac, <code>numpy</code> turned out to be comparable in speed to <code>PyTorch</code> and way faster than <code>TensorFlow</code>. I think this is because those libraries are optimised around data of fairly fixed shape passing through fixed pipelines whereas my code is very ad hoc. I've a feeling a few custom kernels would speed it up a lot. (I may be wrong about this but I do know I can write CUDA/Metal code directly that is many times faster than some of my dice-nine examples.)
</p>

<h2>Not just Python</h2>
<p>
  And one final note. This is a deeply embedded DSL. I use Python as a host to give me an AST that I interpret. This isn't simply overloading of Python operators.
</p></div>
    </summary>
    <updated>2026-04-09T04:32:49Z</updated>
    <published>2026-03-30T17:17:00Z</published>
    <author>
      <name>sigfpe</name>
      <email>noreply@blogger.com</email>
    </author>
    <source>
      <id>tag:blogger.com,1999:blog-11295132</id>
      <category term="mathematics"/>
      <category term="haskell"/>
      <category term="physics"/>
      <category term="monad"/>
      <category term="programming"/>
      <category term="types"/>
      <category term="astronomy"/>
      <category term="quantum"/>
      <category term="comonads"/>
      <category term="self-reference"/>
      <category term="category theory"/>
      <category term="lawvere theories"/>
      <category term="optimisation"/>
      <category term="probability"/>
      <author>
        <name>sigfpe</name>
        <email>noreply@blogger.com</email>
      </author>
      <link href="http://blog.sigfpe.com/" rel="alternate" type="text/html"/>
      <link href="http://sigfpe.blogspot.com/rss.xml" rel="self" type="application/rss+xml"/>
      <title>A Neighborhood of Infinity</title>
      <updated>2026-04-10T20:10:11Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://r6.ca/blog/20260402T135216Z.html</id>
    <link href="https://r6.ca/blog/20260402T135216Z.html" rel="alternate" type="text/html"/>
    <title>Nash Equilibrium for Terminal Maneuvers</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Last year Ethan Heilman wrote about a simple game he calls <a href="https://www.ethanheilman.com/x/31/index.html" title="Missile vs. Laser: The Game of Terminal Maneuvers">Terminal Maneuvers</a>.
This game simulates a missile attacking an interstellar ship.
The ship has a laser defence system.
One player controls the missile, and the other player controls the laser.
If the missile hits the ship, Missile wins.
If the laser hits the missile, the missile is destroyed and Laser wins.

</p><p>The complicating factor is that, due to the relative motion of the laser and the ship being a significant fraction of the speed of light, Laser has to aim not at the missile but where the missile will be.
This distance allows the missile to perform erratic manoeuvers to prevent Laser from knowing what its future position will be.
However, Missile must expend fuel to perform these manoeuvers.

</p><p>The Terminal Maneuvers game proceeds in five rounds, giving the laser five opportunities to hit the missile.
In each round, Missile secretly commits to an amount of fuel they will expend.
Laser must “aim” by guessing the amount of fuel expended by Missile.
If they guess correctly, there is some probability of destroying the missile, which depends on how far away the missile is and how much fuel the missile expended.
The table below shows the probabilities of the missile being destroyed in the various rounds.

<table summary="A matrix of probabilities with rows being fuel burned and columns being the round number.">
<caption>Probability of Laser destroying the missile when correctly guessing Missile’s fuel expenditure</caption>
<thead>
<tr>
<th>Fuel Cost</th>
<th>Round 1</th>
<th>Round 2</th>
<th>Round 3</th>
<th>Round 4</th>
<th>Round 5</th>
</tr>
</thead>
<tbody>
<tr>
<th>0 Fuel</th>
<td>100%</td>
<td>100%</td>
<td>100%</td>
<td>100%</td>
<td>100%</td>
</tr>
<tr>
<th>1 Fuel</th>
<td>1/6</td>
<td>2/6</td>
<td>3/6</td>
<td>4/6</td>
<td>5/6</td>
</tr>
<tr>
<th>2 Fuel</th>
<td>0%</td>
<td>1/6</td>
<td>2/6</td>
<td>3/6</td>
<td>4/6</td>
</tr>
<tr>
<th>3 Fuel</th>
<td/>
<td>0%</td>
<td>1/6</td>
<td>2/6</td>
<td>3/6</td>
</tr>
<tr>
<th>4 Fuel</th>
<td/>
<td/>
<td>0%</td>
<td>1/6</td>
<td>2/6</td>
</tr>
<tr>
<th>5 Fuel</th>
<td/>
<td/>
<td/>
<td>0%</td>
<td>1/6</td>
</tr>
<tr>
<th>6 Fuel</th>
<td/>
<td/>
<td/>
<td/>
<td>0%</td>
</tr>
</tbody>
</table>

</p><p>The missile has a limited amount of fuel at the start of the game.
Fuel spent earlier in the game means less fuel available later in the game when it is most needed.
The amount of starting fuel selects the difficulty of the game.
Ethan suggests <a href="https://www.ethanheilman.com/x/31/index.html" title="Missile vs. Laser: The Game of Terminal Maneuvers">starting with seven fuel</a>, which empirically gives Missile about a 25% chance of winning.

</p><p>Laser knows how much fuel the missile has at the start of each round, so it is imperative that Missile does not run out of fuel in the middle of the game.
If Laser knows the missile is out of fuel, Laser will predict zero fuel used and will always successfully destroy the missile.
That said, as long as Missile has some fuel, choosing to burn zero fuel is still a legitimate option.

</p><p>Starting with seven fuel, one strategy for Missile would be to burn one fuel on the first four rounds and burn the remaining three fuel on the last round.
However, if Laser realizes this is Missile’s strategy, Laser can always predict the correct amount of fuel that will be used by the missile.
Taking the product of all the probabilities of Missile’s survival in each round, Missile only has a 4.6% chance of winning.
Clearly, Missile’s optimal strategy should be non-deterministic.

</p><p>I figured this game would be a fun exercise in learning about mixed-strategy (i.e., non-deterministic) Nash equilibrium.
This game is a small finite game, so it is reasonably easy to analyze, but it is significantly more complicated than trivial games often used in Nash equilibrium examples.

</p><p>For these calculations, it is best to find the Nash equilibrium strategy at the endgame and work backward from there.
To that end, let us start with the simplest non-trivial endgame.
Missile has survived to Round 5 and has 1 fuel left.
Missile can choose to burn their last fuel or not, and Laser can choose to aim at no fuel burned or not.
This yields the following, game-theoretic payoff matrix, listing the probabilities of missile or laser winning:

<table summary="Matrix has rows for Missile&#x2019;s burn choice and columns for Laser&#x2019;s predictions.">
<caption>Payoff matrix for Round 5 with 1 fuel remaining</caption>
<tbody>
<tr>
<td/>
<th>Predict 0</th>
<th>Predict 1</th>
</tr>
<tr>
<th>Burn 0</th>
<td class="payoff">0, 1</td>
<td class="payoff">1, 0</td>
</tr>
<tr>
<th>Burn 1</th>
<td class="payoff">1, 0</td>
<td class="payoff">1⁄6, 5⁄6</td>
</tr>
</tbody>
</table>

</p><p>This is a constant-sum game, because the total score of all players is always the same, no matter the outcome.
Constant-sum games are also known as zero-sum games since they can be translated into games where the sum of each outcome is zero without affecting any strategy.

</p><p>The definition of Nash equilibrium is a pair of strategies, one for each player, where neither player individually can change strategies to improve their outcome.
Therefore, one potential way for Missile to devise a strategy is to find one where Laser’s chance of winning is the same regardless of the move that they make.
Such a strategy is not necessarily going to be possible, but we can give it a try.

</p><p>Let <var>p</var> be the probability that Missile will burn 0, and let <var>q</var> be the probability that Missile will burn 1.
If Laser predicts 0, the probability of them winning is <var>p</var>.
If Laser predicts 1, the probability of them winning is 5⁄6 ⁢<var>q</var>.
If Laser cannot make a choice between these two options to improve their odds, then <var>p</var> = 5⁄6 ⁢<var>q</var>.
Missile’s probabilities must add up to 1, so we also require <var>p</var> + <var>q</var> = 1.

</p><p>We have a linear system of two equations and two unknowns, so we can try to solve it.
The solution is <var>p</var> = 5⁄11 and <var>q</var> = 6⁄11.
Missile burns no fuel with probability 5⁄11 and burns its one fuel with probability 6⁄11.
This provides Missile a 6⁄11 chance of winning, regardless of which prediction Laser makes.

</p><p>On the flip side, Laser’s Nash equilibrium can be computed by choosing a set of probabilities so that Missile’s outcome is the same regardless of whether they choose to burn fuel or not.
This time, let <var>p</var> be the probability that Laser will predict 0, and let <var>q</var> be the probability that Laser will predict 1.
If Missile burns 0, the probability of them winning is <var>q</var>.
If Missile burns 1, the probability of them winning is <var>p</var> + 1⁄6 ⁢<var>q</var>.
Again, if Missile cannot make a choice between these two options to improve their odds, then <var>q</var> = <var>p</var> + 1⁄6 ⁢<var>q</var>.
Laser’s probabilities also must add up to 1, so we also require <var>p</var> + <var>q</var> = 1.

</p><p>Rearranging <var>q</var> = <var>p</var> + 1⁄6 ⁢<var>q</var>, we get 5⁄6 ⁢<var>q</var> = <var>p</var>, which happens to be the exact same equation Missile had.
Thus, their solutions are identical.
Laser predicts no fuel burned with a probability of 5⁄11 and predicts one fuel burned with a probability of 6⁄11.
This provides Laser a 5⁄11 chance of winning no matter whether Missile has chosen to burn their fuel or not.
This chance is the complement to Missile’s 6⁄11 chance of winning, as it has to be.

<table summary="Matrix has rows for Missile&#x2019;s burn choice and columns for Laser&#x2019;s predictions.">
<caption>Payoff matrix for Round 5 with 2 fuel remaining</caption>
<tbody>
<tr>
<td/>
<th>Predict 0</th>
<th>Predict 1</th>
<th>Predict 2</th>
</tr>
<tr>
<th>Burn 0</th>
<td class="payoff">0, 1</td>
<td class="payoff">1, 0</td>
<td class="payoff">1, 0</td>
</tr>
<tr>
<th>Burn 1</th>
<td class="payoff">1, 0</td>
<td class="payoff">1⁄6, 5⁄6</td>
<td class="payoff">1, 0</td>
</tr>
<tr>
<th>Burn 2</th>
<td class="payoff">1, 0</td>
<td class="payoff">1, 0</td>
<td class="payoff">2⁄6, 4⁄6</td>
</tr>
</tbody>
</table>

</p><p>If the game ends in Round 5 with Missile having 2 fuel left, we have the above payoff matrix.
We can solve similar linear algebra problems on three variables to find strategies for each player so that the other player’s outcome is the same no matter which of their three choices they make.
The solution has Missile burn 0 fuel with probability 10⁄37, burn 1 fuel with probability 12⁄37, and burn 2 fuel with probability 15⁄37, giving Missile a 27⁄37 chance of winning regardless of what Laser’s prediction is.

</p><p>Laser makes the predictions with the same probability distribution, giving Laser a 10⁄37 chance of winning no matter how much fuel Missile chooses to burn.
This sort of distribution is what we might expect: somewhat evenly distributed with a bias towards burning more fuel, which provides some evasion for Missile.

</p><p>What I found surprising is how when Laser plays at their Nash equilibrium, they simply do not care how much fuel Missile has secretly chosen to burn.
Their odds of winning are the same regardless of what Missile reveals.
It is as if Laser is no longer playing against Missile at all.
Missile’s choices no longer matter.
This result is called the “indifference principle.”
Later we will see that Missile’s choices sometimes can matter.

</p><p>Missile feels the same when playing at their equilibrium.
No matter what prediction Laser ultimately makes, Missile’s odds of winning have already been fixed by playing at the Nash equilibrium.

</p><p>In theory, this is what playing poker at a Nash equilibrium should feel like.
Based on the state of the board, you make a random selection of calls, folds, or raises according to some appropriate distribution, and your distribution has fixed your expected payout at that point, independent of the choices the other players are going to make.
No need to stress over whether your bluff will be called or not.

</p><p>Before moving on to analyzing Round 4, we can complete a chart of the probability of Missile winning when playing at their Nash equilibrium depending on how much fuel they have remaining.

<table summary="Table columns are for the two players, Missile and Laser, with a row for each possible amount of remaining fuel at the start of the round.">
<caption>Probability table for Round 5</caption>
<tbody>
<tr>
<th>Remaining Fuel</th>
<th>Missile Win Probability</th>
<th>Laser Win Probability</th>
</tr>
<tr>
<td>0</td>
<td class="number">0%</td>
<td class="number">100%</td>
</tr>
<tr>
<td>1</td>
<td class="number">6⁄11 ≈ 54.5%</td>
<td class="number">5⁄11 ≈ 45.5%</td>
</tr>
<tr>
<td>2</td>
<td class="number">27⁄37 ≈ 73.0%</td>
<td class="number">10⁄37 ≈ 27.0%</td>
</tr>
<tr>
<td>3</td>
<td class="number">47⁄57 ≈ 82.5%</td>
<td class="number">10⁄57 ≈ 17.5%</td>
</tr>
<tr>
<td>4</td>
<td class="number">77⁄87 ≈ 88.5%</td>
<td class="number">10⁄87 ≈ 11.5%</td>
</tr>
<tr>
<td>5</td>
<td class="number">137⁄147 ≈ 93.2%</td>
<td class="number">10⁄147 ≈ 6.8%</td>
</tr>
<tr>
<td>6+</td>
<td class="number">100%</td>
<td class="number">0%</td>
</tr>
</tbody>
</table>

</p><p>If Missile starts Round 4 with 1 fuel remaining, they are in big trouble.
They can only burn fuel in at most one of the two remaining rounds.
Therefore, Laser can win by predicting 0 fuel burned in both Round 4 and Round 5.
Laser is guaranteed to destroy the missile on one of those two rounds.
Missile must start Round 4 with at least 2 fuel remaining if they are to have a chance of winning.

</p><p>Since the game in each round depends only on the state of Missile’s remaining fuel and not on the specific choices of how that state came to be, we can simplify the analysis of Round 4’s payoff matrix by using each player’s probability of winning Round 5 as their scores in Round 4.
Note that Missile does not have the option of burning all their fuel in Round 4, since starting Round 5 with 0 fuel is a guaranteed loss for them, and Laser knows it.

<table summary="Matrix has rows for Missile&#x2019;s burn choice and columns for Laser&#x2019;s predictions.">
<caption>Payoff matrix for Round 4 with 2 fuel remaining</caption>
<tbody>
<tr>
<td/>
<th>Predict 0</th>
<th>Predict 1</th>
</tr>
<tr>
<th>Burn 0</th>
<td class="payoff">0, 1</td>
<td class="payoff">27⁄37, 10⁄37</td>
</tr>
<tr>
<th>Burn 1</th>
<td class="payoff">6⁄11, 5⁄11</td>
<td class="payoff">2⁄11, 9⁄11</td>
</tr>
</tbody>
</table>

</p><p>To compute Missile’s strategy, we define <var>p</var> and <var>q</var> as before.
This time Missile needs to solve the equations
<var>p</var> + 5⁄11 ⁢<var>q</var> = 10⁄37 ⁢<var>p</var> + 9⁄11 ⁢<var>q</var> and <var>p</var> + <var>q</var> = 1.
The solution has Missile burn 0 fuel with a probability of 148⁄445 and burn 1 fuel with a probability of 297⁄445, which is roughly a 1⁄3<sup>rd</sup>–2⁄3<sup>rd</sup> split.
This provides Missile a chance of winning with a probability of 162⁄445, or about 36.4%.

</p><p>Meanwhile, Laser needs to solve the equations 27⁄37 ⁢<var>q</var> = 6⁄11 ⁢<var>p</var> + 2⁄11 ⁢<var>q</var> and <var>p</var> + <var>q</var> = 1.
The solution has Laser predict 0 fuel with probability 223⁄445 and predict 1 fuel with probability 222⁄445, which is nearly evenly split.
This provides Laser a chance of winning of 283⁄445, or about 63.6%.

</p><p>In Round 5, each player’s individual payoff matrix was symmetric, which led to Missile and Laser having identical strategies.
In Round 4, the individual player’s payoff matrices are no longer symmetric, and Missile and Laser end up with different strategies.
Laser picks a nearly 50–50 split because the differences of column scores, 5⁄11 − 1 vs. 9⁄11 − 10⁄37, are nearly equal in magnitude.
Whereas Missile picks a 1⁄3<sup>rd</sup>–2⁄3<sup>rd</sup> split because the difference of row scores, 27⁄37 vs. 2⁄11 − 6⁄11, differs in magnitude by close to a factor of two.

</p><p>We can proceed as before, using linear algebra to compute equilibrium strategies for Round 4 with various states of remaining fuel for the missile.
However, we run into a problem when Missile has 4 fuel remaining.

<table summary="Matrix has rows for Missile&#x2019;s burn choice and columns for Laser&#x2019;s predictions.">
<caption>Payoff matrix for Round 4 with 4 fuel remaining.</caption>
<tbody>
<tr>
<td/>
<th>Predict 0</th>
<th>Predict 1</th>
<th>Predict 2</th>
<th>Predict 3</th>
</tr>
<tr>
<th>Burn 0</th>
<td class="payoff">0, 1</td>
<td class="payoff">77⁄87, 10⁄87</td>
<td class="payoff">77⁄87, 10⁄87</td>
<td class="payoff">77⁄87, 10⁄87</td>
</tr>
<tr>
<th>Burn 1</th>
<td class="payoff">47⁄57, 10⁄57</td>
<td class="payoff">47⁄171, 124⁄171</td>
<td class="payoff">47⁄57, 10⁄57</td>
<td class="payoff">47⁄57, 10⁄57</td>
</tr>
<tr>
<th>Burn 2</th>
<td class="payoff">27⁄37, 10⁄37</td>
<td class="payoff">27⁄37, 10⁄37</td>
<td class="payoff">27⁄74, 47⁄74</td>
<td class="payoff">27⁄37, 10⁄37</td>
</tr>
<tr>
<th>Burn 3</th>
<td class="payoff">6⁄11, 5⁄11</td>
<td class="payoff">6⁄11, 5⁄11</td>
<td class="payoff">6⁄11, 5⁄11</td>
<td class="payoff">4⁄11, 7⁄11</td>
</tr>
</tbody>
</table>

</p><p>Let us try to solve for Laser’s equilibrium strategy, the probability distribution where Missile’s outcome is the same no matter what move they make.
We let <var>p</var>, <var>q</var>, <var>r</var>, and <var>s</var> be the probabilities of predicting 0 through 3 fuel burned, respectively.
In addition to having <var>p</var> + <var>q</var> + <var>r</var> + <var>s</var> = 1, we require</p><ul>
<li>77⁄87 ⁢(<var>q</var> + <var>r</var> + <var>s</var>),</li>
<li>47⁄57 ⁢(<var>p</var> + <var>r</var> + <var>s</var>) + 47⁄171 ⁢<var>q</var>,</li>
<li>27⁄37 ⁢(<var>p</var> + <var>q</var> + <var>s</var>) + 27⁄74 ⁢<var>r</var>, and</li>
<li>6⁄11 ⁢(<var>p</var> + <var>q</var> + <var>r</var>) + 4⁄11 ⁢<var>s</var></li>
</ul>
all be equal to each other.
Solving this system of equations gives us
<ul>
<li><var>p</var> ≈ 34.4%,</li>
<li><var>q</var> ≈ 44.3%,</li>
<li><var>r</var> ≈ 40.8%, and</li>
<li><var>s</var> ≈ −19.5%.</li>
</ul>

<p>Apparently, predicting 3 fuel used is such a terrible move for Laser that our “optimal” solution wants us to predict it with a <em>negative</em> 19.5% probability!
Unfortunately, Laser cannot actually select moves with negative probability.
We have to add constrains to our acceptable solutions to ensure all probabilities are non-negative.

</p><p>Adding linear constraints to our problem brings us into the realm of linear programming.
Since we are entering this realm, we can take this opportunity to compute the minimax solution for each player.
For Laser, the minimax solution is to compute a probability distribution that minimizes Missile’s score, i.e., their probability of winning, which we will denote by <var>z</var>, subject to the constraint that Missile will choose the move that maximizes their score for that distribution.
This leads to the following system of linear constraints:

</p><ul>
<li>77⁄87 ⁢(<var>q</var> + <var>r</var> + <var>s</var>) ≤ <var>z</var></li>
<li>47⁄57 ⁢(<var>p</var> + <var>r</var> + <var>s</var>) + 47⁄171 ⁢<var>q</var> ≤ <var>z</var></li>
<li>27⁄37 ⁢(<var>p</var> + <var>q</var> + <var>s</var>) + 27⁄74 ⁢<var>r</var> ≤ <var>z</var></li>
<li>6⁄11 ⁢(<var>p</var> + <var>q</var> + <var>r</var>) + 4⁄11 ⁢<var>s</var> ≤ <var>z</var></li>
<li>0 ≤ <var>p</var></li>
<li>0 ≤ <var>q</var></li>
<li>0 ≤ <var>r</var></li>
<li>0 ≤ <var>s</var></li>
<li><var>p</var> + <var>q</var> + <var>r</var> + <var>s</var> = 1</li>
</ul>
where we want to minimize <var>z</var>.

<p>Using <a href="https://online-optimizer.appspot.com/" title="Linear Optimization">linear programming</a>, we can optimize this system.
The <a href="https://online-optimizer.appspot.com/?model=https:%2F%2Fr6.ca%2Fblog%2Ftext%2F77ad58fbd4a4ff615adeae04c64de276340e079f23affac9337f42f52e9e7f9e.txt" title="Linear Optimization">optimal solution</a> is</p><ul>
<li><var>p</var> ≈ 30.5%,</li>
<li><var>q</var> ≈ 38.1%,</li>
<li><var>r</var> ≈ 31.4%,</li>
<li><var>s</var> ≈ 0%, and</li>
<li><var>z</var> ≈ 61.5%.</li>
</ul>
That is, Laser’s strategy is to
predict 0 fuel burned 30.5% of the time,
predict 1 fuel burned 38.1% of the time,
predict 2 fuel burned 31.4% of the time,
and <em>never</em> predict 3 fuel burned.
This lets Missile win at most 61.5% of the time, or equivalently, it lets Laser win at least 38.5% of the time.

<p>Is this minimax strategy really an optimal strategy?
Let us look at Missile’s minimax strategy.
For Missile, we need to optimize the following system of linear constraints:

</p><ul>
<li><var>p</var> + 10⁄57 ⁢<var>q</var> + 10⁄37 ⁢<var>r</var> + 5⁄11 ⁢<var>s</var> ≤ <var>z</var></li>
<li>10⁄87 ⁢<var>p</var> + 124⁄171 ⁢<var>q</var> + 10⁄37 ⁢<var>r</var> + 5⁄11 ⁢<var>s</var> ≤ <var>z</var></li>
<li>10⁄87 ⁢<var>p</var> + 10⁄57 ⁢<var>q</var> + 47⁄74 ⁢<var>r</var> + 5⁄11 ⁢<var>s</var> ≤ <var>z</var></li>
<li>10⁄87 ⁢<var>p</var> + 10⁄57 ⁢<var>q</var> + 10⁄37 ⁢<var>r</var> + 7⁄11 ⁢<var>s</var> ≤ <var>z</var></li>
<li>0 ≤ <var>p</var></li>
<li>0 ≤ <var>q</var></li>
<li>0 ≤ <var>r</var></li>
<li>0 ≤ <var>s</var></li>
<li><var>p</var> + <var>q</var> + <var>r</var> + <var>s</var> = 1</li>
</ul>
to minimize <var>z</var>.

<p>The <a href="https://online-optimizer.appspot.com/?model=https:%2F%2Fr6.ca%2Fblog%2Ftext%2F799b154aeb4b2b968b0d8b0895c20334dab3d7a5244d3b5a26a3df2b94e29f6e.txt" title="Linear Optimization">optimal solution</a> is</p><ul>
<li><var>p</var> ≈ 19.9%,</li>
<li><var>q</var> ≈ 32.0%,</li>
<li><var>r</var> ≈ 48.2%,</li>
<li><var>s</var> ≈ 0%, and</li>
<li><var>z</var> ≈ 38.5%.</li>
</ul>
That is, Missile’s strategy is to
burn 0 fuel 19.9% of the time,
burn 1 fuel 32.0% of the time,
burn 2 fuel 48.2% of the time,
and <em>never</em> burn 3 fuel.
This lets Laser win at most 38.5% of the time, or equivalently, it lets Missile win at least 61.5% of the time.

<p>This pair of strategies is optimal because Laser wins at least 38.5% of the time by their strategy, and Missile wins at least 61.5% of the time by their strategy, which adds up to 100%.
It turns out that <a href="https://www.cs.mcgill.ca/~cai/COMP_MATH_553_2016/lec3.pdf" title="6.891 Games, Decision, and Computation: Lecture 3">for zero-sum games, the minimax, maximin, and Nash equilibrium strategy sets are all identical, and furthermore, these strategies form a convex set</a>.
Using <a href="https://cgi.csc.liv.ac.uk/~rahul/bimatrix_solver/" title="Solve a Bimatrix Game">a general-purpose Nash equilibrium solver</a> will produce the same pair of optimal strategies.


</p><p>Still, these strategies surprised me.
Laser is not even aiming at Missile burning 3 fuel.
Shouldn’t Missile avoid being hit by Laser entirely by choosing to burn 3 fuel?
But Missile’s optimal strategy also says to avoid burning 3 units of fuel.
Why?

</p><p>Upon closer examination, we see that with Missile’s computed optimal strategy, they have a 61.5% chance of winning.
If Missile were to burn 3 fuel, yes, they would avoid being hit by Laser in Round 4.
However, they would begin Round 5 with only 1 remaining fuel.
In that state they would only have a 54.5% chance of winning, worse odds than their optimal strategy that avoids burning 3 fuel.

</p><p>Laser is not aiming at Missile burning 3 fuel because Laser would <em>love</em> for Missile to burn 3 fuel.
Doing so would increase Laser’s odds of winning from 38.5% to 45.5%.
We see that Laser’s strategy does not entirely rule out all consequences of Missile’s choices.
It only makes it indifferent to Missile’s choices within the support of Missile’s optimal mixed set of moves.
Technically an opponent’s choices can affect the outcome of the game; they can still make moves that benefit the other player.

</p><p>Continuing with linear programming, we can fill out the  table for the probability of winning for Round 4.

<table summary="Table columns are for the two players, Missile and Laser, with a row for each possible amount of remaining fuel at the start of the round.">
<caption>Probability table for Round 4</caption>
<tbody>
<tr>
<th>Remaining Fuel</th>
<th>Missile Win Probability</th>
<th>Laser Win Probability</th>
</tr>
<tr>
<td>1-</td>
<td class="number">0%</td>
<td class="number">100%</td>
</tr>
<tr>
<td>2</td>
<td class="number">≈ 36.4%</td>
<td class="number">≈ 63.6%</td>
</tr>
<tr>
<td>3</td>
<td class="number">≈ 50.5%</td>
<td class="number">≈ 49.5%</td>
</tr>
<tr>
<td>4</td>
<td class="number">≈ 61.5%</td>
<td class="number">≈ 38.5%</td>
</tr>
<tr>
<td>5</td>
<td class="number">≈ 69.9%</td>
<td class="number">≈ 30.1%</td>
</tr>
<tr>
<td>6</td>
<td class="number">≈ 76.4%</td>
<td class="number">≈ 23.6%</td>
</tr>
<tr>
<td>7</td>
<td class="number">≈ 81.6%</td>
<td class="number">≈ 18.4%</td>
</tr>
</tbody>
</table>

</p><p>Continuing this way, we can work backwards and compute probability tables for all the rounds until we reach round 1.

<table summary="Table columns are for the two players, Missile and Laser, with a row for each possible amount of remaining fuel at the start of the round.">
<caption>Probability table for Round 1</caption>
<tbody>
<tr>
<th>Starting Fuel</th>
<th>Missile Win Probability</th>
<th>Laser Win Probability</th>
</tr>
<tr>
<td>7</td>
<td class="number">1005005076075⁄3110959445024 ≈ 32.3%</td>
<td class="number">2105954368949⁄3110959445024 ≈ 67.7%</td>
</tr>
</tbody>
</table>

</p><p>In conclusion, we found that playing optimally, Missile has an approximately 32.3% chance of winning, which is a little higher than the 25% estimate given by Ethan.
I leave it as an exercise to determine the most fair amount of starting for Missile to start with.

</p></div>
    </content>
    <updated>2026-04-02T13:52:16Z</updated>
    <published>2026-04-02T13:52:16Z</published>
    <source>
      <id>https://r6.ca/blog/</id>
      <author>
        <name>Russell O’Connor</name>
        <uri>https://r6.ca/</uri>
      </author>
      <link href="https://r6.ca/blog/" rel="alternate" type="text/html"/>
      <link href="https://r6.ca/blog/feed.atom" rel="self" type="application/atom+xml"/>
      <title>Russell O’Connor’s Blog</title>
      <updated>2026-04-02T13:52:16Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://tweag.io/blog/2026-04-02-making-bazel-builds-more-reliable/</id>
    <link href="https://tweag.io/blog/2026-04-02-making-bazel-builds-more-reliable/" rel="alternate" type="text/html"/>
    <title>Accessing external resources reliably with Bazel</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>We say that a system is reliable if it continues to function correctly
when events outside the system affect it.
Many factors can impact the reliability of Bazel builds,
especially dependencies on external services.
In this post, we’ll focus on what can go wrong
when your build needs resources you don’t control
and what you can do to reduce the risk of build failures.</p>
<h2 id="depending-on-external-resources"><a class="anchor before" href="https://www.tweag.io/rss.xml#depending-on-external-resources"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Depending on external resources</h2>
<p>Some build actions triggered by Bazel might be accessing resources
that are external to your organization.
For Bazel builds, this typically applies to build rules (to build your first-party code) or
repository rules (utilities and tools those rules might need).
When Bazel starts a build, it emits data about network requests,
and you need to make those external requests visible
so that you know what external resources your builds depend on.
You can access this information via the <a href="https://bazel.build/remote/bep">Build Event Protocol (BEP)</a>
which can be written to disk or,
if you operate a remote cache service, your provider might have a BEP viewer.
You can also use the <a href="https://bazel.build/versions/8.2.0/reference/command-line-reference#flag--experimental_repository_resolved_file"><code class="language-text">--experimental_repository_resolved_file</code> flag</a>
to produce resolved information about all Starlark repository rules that were executed.</p>
<p>Building a target that depends on a repository rule such as this:</p>
<div class="gatsby-highlight"><pre class="language-python"><code class="language-python">http_archive<span class="token punctuation">(</span>
    name <span class="token operator">=</span> <span class="token string">"yq_cli"</span><span class="token punctuation">,</span>
    build_file <span class="token operator">=</span> <span class="token string">"@//tools/yq:BUILD.bazel.gen"</span><span class="token punctuation">,</span>
    sha256 <span class="token operator">=</span> <span class="token string">"7583d471d9bfe88e32005e9d287952382df0469135f691e044443f610d707f4d"</span><span class="token punctuation">,</span>
    url <span class="token operator">=</span> <span class="token string">"https://github.com/mikefarah/yq/releases/download/v4.47.1/yq_linux_amd64.tar.gz"</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span></code></pre></div>
<p>would result in the following build event (the snippet below is copied from the BEP output):</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">...
children {
  fetch {
    url: "https://github.com/mikefarah/yq/releases/download/v4.47.1/yq_linux_amd64.tar.gz"
  }
}
...</code></pre></div>
<p>To get an idea of what kinds of artifacts a Bazel build for a reasonably large project might fetch,
let’s build a few open-source projects — <a href="https://github.com/envoyproxy/envoy">Envoy</a>, <a href="https://github.com/redpanda-data/redpanda">Redpanda</a>, and <a href="https://github.com/DataDog/datadog-agent">datadog-agent</a>.
These are some of the domains from which at least one resource was fetched
when building all targets from these projects:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">bcr.bazel.build             cdn.azul.com                dl.google.com
dl.grafana.com              dl.min.io                   download.gnome.org
files.pythonhosted.org      gcr.io                      github.com
go.dev                      mirror.bazel.build          mirrors.kernel.org
raw.githubusercontent.com   pkgconfig.freedesktop.org   pypi.org
static.crates.io            static.rust-lang.org        s3.amazonaws.com
www.antlr.org               www.colm.net                www.lua.org
www.sqlite.org              www.tcpdump.org</code></pre></div>
<p>While most of your external dependencies are going to be declared in build metadata files
such as <code class="language-text">MODULE.bazel</code> (or legacy <code class="language-text">WORKSPACE</code>),
some network requests are going to be made by build targets
such as <code class="language-text">genrule</code>s (e.g., by calling <code class="language-text">curl</code>) or toolchains (e.g., a <code class="language-text">pip</code> call to the PyPI index).
We’ll see a worked example of this later in the post.</p>
<h2 id="common-problems"><a class="anchor before" href="https://www.tweag.io/rss.xml#common-problems"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Common problems</h2>
<p>In general, it is advised to rely on <code class="language-text">MODULE.bazel</code> or <code class="language-text">WORKSPACE</code> mechanisms
for accessing external dependencies instead of doing so via build or test actions.
Bazel by design lacks support and features for downloads to take place within build actions,
and when attempting to interact with external systems this way,
you will be limited in how you can manage and account for those requests.</p>
<p>Therefore, when building, the complete list of accessed online resources —
those that are accounted for by BEP and those that are not — might be much longer.
After doing a full build, it might be helpful to audit the network requests made
to discover what resources were fetched
and a complete inventory of external hosts your build depends on.</p>
<p>Given these external dependencies, these are common problems that could happen to any of them:</p>
<ul>
<li>Outages:
no service provides 100% uptime guarantee
and some providers, sadly, have incidents all too often.</li>
<li>Removed artifacts:
an archive file might be deleted due to retention policy.</li>
<li>Rate limiting:
many concurrent builds coming from the same cluster can accidentally trigger API
or download rate limits, especially with public registries.</li>
<li>Checksum drift:
content of an artifact at a given URL can change, intentionally or maliciously,
causing checksum mismatches.</li>
</ul>
<p>This post focuses on strategies to either remove these external
dependencies from the critical path, or make failures graceful and recoverable.</p>
<h2 id="remedies"><a class="anchor before" href="https://www.tweag.io/rss.xml#remedies"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Remedies</h2>
<p>The remedies below are intentionally “stackable”: you can start with low-effort
safeguards (e.g., checksums and retries) and progress toward stronger guarantees
(e.g., mirrors and network blocking).
If you’re skimming, you can pick one external host that concerns you
(e.g., <code class="language-text">github.com</code> or <code class="language-text">pypi.org</code>) and follow the options
that would let you depend on it more reliably.</p>
<h3 id="using-checksums"><a class="anchor before" href="https://www.tweag.io/rss.xml#using-checksums"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Using checksums</h3>
<p>External resources may not only vanish or become inaccessible, but also change in place.
Any artifact you download (unless there’s a strong guarantee from a provider), might change its contents
such as when a provider does in-place updates of their releases
(or it could also be a malicious attempt to inject code).
To prevent this issue, SHA-256 digests must be coupled with any artifact you download from the Internet.
Even though when declaring dependencies on external resources
such as with <code class="language-text">http_archive</code>, providing <code class="language-text">sha256</code> attribute is optional,
it is considered a security risk to omit specifying the SHA-256 for remote files to be fetched.</p>
<h3 id="using-github-releases"><a class="anchor before" href="https://www.tweag.io/rss.xml#using-github-releases"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Using GitHub releases</h3>
<p>As the majority of build rules and open-source tools used by projects built with Bazel are hosted on GitHub,
there are some special concerns that are worth mentioning.</p>
<p>A public GitHub repository might be moved, deleted, or become private (this happened in 2025 with <code class="language-text">rules_mypy</code>).
If you do have to rely on external rulesets hosted on GitHub,
make sure they are hosted under the <a href="https://github.com/bazel-contrib/">bazel-contrib</a> organization
(or help get them migrated at some point) to avoid surprises.</p>
<p>Checksums of <a href="https://docs.github.com/en/repositories/working-with-files/using-files/downloading-source-code-archives">dynamically generated archives</a> might change;
this has caused Bazel outages before, in <a href="https://github.com/orgs/community/discussions/45830">2023</a>.
There was some <a href="https://github.blog/open-source/git/update-on-the-future-stability-of-source-code-archives-and-hashes/">confusion</a> about whether the stability of archives is guaranteed or not.
There might be some edge cases such as when a <a href="https://github.com/orgs/community/discussions/46034#discussioncomment-10572847">Git repository is renamed</a>,
and since Bazel builds rely on stability of archives (for reproducibility and caching among other reasons)
it might be best to play it safe and only use <a href="https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases">releases</a> instead of using source downloads.</p>
<h3 id="using-retries"><a class="anchor before" href="https://www.tweag.io/rss.xml#using-retries"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Using retries</h3>
<p>It is possible that some of your dependencies need to be obtained from an online resource
that is known to be unstable.
What’s worse, you may not even be able to cache it (or host yourself):
for example, imagine needing to download a short-lived license file for a commercial product from the manufacturer’s server
when starting a build.
To make downloading this file (via a repository rule) more likely to succeed,
consider using the <a href="https://bazel.build/reference/command-line-reference#common_options-flag--experimental_repository_downloader_retries"><code class="language-text">--experimental_repository_downloader_retries</code> flag</a>
to specify the maximum number of attempts to retry upon a download error.</p>
<h3 id="placing-binaries-under-version-control"><a class="anchor before" href="https://www.tweag.io/rss.xml#placing-binaries-under-version-control"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Placing binaries under version control</h3>
<p>This varies a lot between organizations and the programming languages concerned,
but a common approach that is adopted by most organizations is
to check in the source code that is used to build a binary, and not the binary itself.</p>
<p>Many engineers would be strongly opposed to checking in any binary,
as Version Control Systems (VCS) are designed and optimised for managing the source code.
However, it is known that some organizations choose to place binary libraries
that are external dependencies of their first-party code
under version control.
This has been seen occasionally in Java projects where <code class="language-text">.jar</code> libraries
(that nowadays can be managed with Maven / Gradle) were checked in.
Today, this, arguably, might make sense only for legacy projects,
air-gapped or classified networks, and for vendored native libraries
that are hard to rebuild.</p>
<p>Unless you are able to provide top-notch automation for keeping your third-party dependencies
checked in under version control up-to-date, patched, and compliant with any licensing constraints,
it might be best to rely on a private artifact cache for hosting third-party dependencies.</p>
<h3 id="internal-repository-manager"><a class="anchor before" href="https://www.tweag.io/rss.xml#internal-repository-manager"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Internal repository manager</h3>
<p>As your organization grows, you will likely need to invest in a tool
that would allow you to organize your resources
such as external tools and third-party code packages into repositories.
There are lots of commercial solutions on the market such as <a href="https://jfrog.com/artifactory/">JFrog Artifactory</a>,
<a href="https://www.sonatype.com/products/sonatype-nexus-repository">Sonatype Nexus</a>, <a href="https://aws.amazon.com/codeartifact/">AWS CodeArtifact</a>, and <a href="https://docs.gitlab.com/user/packages/">GitLab package registry</a> to name a few.</p>
<p>With a repository manager, once you discover a dependency on an external artifact,
you would upload it manually in your internal binary repository
and update your build metadata accordingly:</p>
<div class="gatsby-highlight"><pre class="language-python"><code class="language-python"><span class="token comment"># MODULE.bazel</span>
http_archive<span class="token punctuation">(</span>
  name <span class="token operator">=</span> <span class="token string">"tool"</span><span class="token punctuation">,</span>
  <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>
  urls <span class="token operator">=</span> <span class="token punctuation">[</span>
    <span class="token string">"https://artifacts.company.com/artifactory/project/tools/tool-1.2.3.tar.gz"</span><span class="token punctuation">,</span>
    <span class="token string">"https://www.project.org/source/1.2.3/tool-1.2.3.tar.gz"</span><span class="token punctuation">,</span>
  <span class="token punctuation">]</span>
<span class="token punctuation">)</span></code></pre></div>
<p>URLs from the <a href="https://bazel.build/rules/lib/repo/http#http_archive-urls"><code class="language-text">urls</code></a> attribute are tried in order until one succeeds.
It is recommended to specify the local binary repository artifact first,
and if the hosted mirror happens to be down, your build would still succeed
provided that, in this case, <code class="language-text">project.org</code> is up and running.</p>
<h3 id="bazel-downloader-configuration"><a class="anchor before" href="https://www.tweag.io/rss.xml#bazel-downloader-configuration"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Bazel downloader configuration</h3>
<p>You could also let your binary repository manager be the only place where Bazel builds can fetch resources from
if you don’t want to depend on external artifacts in any way at all.
This can be achieved by providing a configuration file for the remote downloader
using the <a href="https://bazel.build/reference/command-line-reference#common_options-flag--downloader_config"><code class="language-text">--downloader_config</code> flag</a>.</p>
<p>For example, a simple use case may be to block GitHub and instead rewrite fetches to go to an Artifactory instance.
This can be done with the following downloader configuration:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">rewrite github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(.*) artifacts.my-company.com/artifactory/github-releases-mirror/$1/$2/releases/download/$3/$4

# if you still have to rely on dynamically generated archives instead of releases
rewrite github.com/([^/]+)/([^/]+)/archive/(.+).(tar.gz|zip) artifacts.my-company.com/artifactory/github-releases-mirror/$1/$2/archive/$3.$4</code></pre></div>
<p>However, support for using Bazel’s downloader needs to be enabled in Bazel rulesets by their authors.
For instance, in <code class="language-text">rules_python</code>, the <a href="https://github.com/bazel-contrib/rules_python/blob/main/docs/pypi/download.md#bazel-downloader-and-multi-platform-wheel-hub-repository"><code class="language-text">pip</code> extension</a>
now supports pulling information from a PyPI compatible mirror
which means that the Bazel downloader can be used for downloading Python wheels.</p>
<p>Take a look at some downloader configurations used in other projects
(e.g., <a href="https://github.com/monogon/monogon/blob/main/build/bazel/bazel_downloader.cfg">1</a>, <a href="https://github.com/bazelbuild/bazel/blob/master/bazel_downloader.cfg">2</a>, <a href="https://github.com/google/tensorstore/blob/master/bazel/rewrite.config">3</a>)
to explore how others set up access to external resources
and learn the nuances of the configuration declaration syntax.</p>
<h3 id="blocking-network-requests"><a class="anchor before" href="https://www.tweag.io/rss.xml#blocking-network-requests"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Blocking network requests</h3>
<p>Additional control of network access can be achieved by blocking some network requests in CI agents
using custom firewall rules or other tools of that nature.
However, as mentioned earlier, Bazel’s downloader configuration can only rewrite or block requests that Bazel
is aware of.
This means that not all network traffic in a Bazel build is Bazel-managed traffic.</p>
<p>To illustrate this, let’s declare a dependency on the <a href="https://github.com/bazelbuild/bazel-central-registry/tree/5978cdffb9fdb566dc0eca759f1aa57cb3604914/modules/gawk/5.3.2"><code class="language-text">gawk</code> binary</a>.
When running <code class="language-text">gawk</code>, its <a href="https://github.com/bazelbuild/bazel-central-registry/blob/5978cdffb9fdb566dc0eca759f1aa57cb3604914/modules/gawk/5.3.2/source.json#L2">sources</a> are going to be fetched from the <a href="https://ftp.gnu.org/gnu/gawk/gawk-5.3.2.tar.xz">GNU FTP server</a>.
Let’s also add a <code class="language-text">genrule</code> that will download an archive from the same FTP server:</p>
<div class="gatsby-highlight"><pre class="language-python"><code class="language-python"><span class="token comment"># MODULE.bazel</span>
bazel_dep<span class="token punctuation">(</span>name <span class="token operator">=</span> <span class="token string">"gawk"</span><span class="token punctuation">,</span> version <span class="token operator">=</span> <span class="token string">"5.3.2"</span><span class="token punctuation">)</span>

<span class="token comment"># BUILD.bazel</span>
genrule<span class="token punctuation">(</span>
    name <span class="token operator">=</span> <span class="token string">"diffutils"</span><span class="token punctuation">,</span>
    outs <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"diffutils-3.12.tar.xz"</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
    cmd <span class="token operator">=</span> <span class="token triple-quoted-string string">"""wget -O "$@" https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.xz"""</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span></code></pre></div>
<p>We’ll configure Bazel to use a downloader configuration that blocks fetches from that FTP server:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text"># bazel_downloader.cfg
block ftp.gnu.org

# .bazelrc
common --downloader_config=bazel_downloader.cfg</code></pre></div>
<p>When attempting to run the <code class="language-text">gawk</code> binary from the ruleset, an error is expectedly raised
since accessing the server is blocked:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">$ bazel run @gawk
...
ERROR: java.io.IOException: Configured URL rewriter blocked all URLs:
[https://ftp.gnu.org/gnu/gawk/gawk-5.3.2.tar.xz]</code></pre></div>
<p>However, building a <code class="language-text">genrule</code> still succeeds
because the downloader configuration does not apply here:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">$ bazel build //src:diffutils
...
INFO: From Executing genrule //src:diffutils:
--2026-01-19 10:48:54--  https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.xz
Resolving ftp.gnu.org (ftp.gnu.org)... 209.51.188.20, 2001:470:142:3::b
Connecting to ftp.gnu.org (ftp.gnu.org)|209.51.188.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Saving to: 'bazel-out/k8-fastbuild/bin/src/diffutils-3.12.tar.xz'</code></pre></div>
<p>External network requests of this nature are hard to audit in a large codebase
since they won’t show up as structured fetch events in BEP output.
To mitigate this, prefer using repository rules and <a href="https://bazel.build/external/extension">Bzlmod extensions</a>
for any downloads instead of ad hoc shell commands.
Going a step further, you might want to consider forbidding direct calls to applications
that might make network requests (such as <code class="language-text">curl</code> or <code class="language-text">wget</code>) in <code class="language-text">genrule</code> targets, unless explicitly approved.
Where unavoidable, configure targets to access internal repositories instead of public endpoints.</p>
<h3 id="sandboxing"><a class="anchor before" href="https://www.tweag.io/rss.xml#sandboxing"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Sandboxing</h3>
<p>When triggering builds in a <a href="https://bazel.build/docs/sandboxing#sandboxing-strategies">Bazel sandbox</a>, they are run in a container (using Linux Namespaces)
to isolate the build actions from the host.
In addition to making your entire filesystem read-only (except for the sandbox directory),
you can also forbid actions access the network.
This is useful in some scenarios when you want to confirm that a build doesn’t make any network requests
such as when running unit tests or integration tests that are not supposed to make any network calls.
See <a href="https://bazel.build/reference/be/common-definitions">Bazel tags</a> <code class="language-text">requires-network</code> and <code class="language-text">block-network</code>
to learn how to control network access for individual build targets.</p>
<p>Keep in mind that cached results of build actions can still be fetched even when blocking the network in a sandbox.
So if artifacts needed for a build were uploaded to the Bazel cache previously,
you won’t know whether a particular build needs any network resources unless you run the build without cache access.
Also, none of the sandbox flags affect any cache as it’s expected
that these flags should not affect the output of hermetic actions
and making them part of a cache key would worsen the effectiveness of the cache.</p>
<p>With the network disabled in a sandbox, the <code class="language-text">genrule</code> target we declared earlier fails to build:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">$ bazel build //src:diffutils --spawn_strategy=linux-sandbox --nosandbox_default_allow_network
...
ERROR: Executing genrule //src:diffutils failed: (Exit 4): bash failed: ...
Resolving ftp.gnu.org (ftp.gnu.org)... failed: Temporary failure in name resolution.
wget: unable to resolve host address 'ftp.gnu.org'
Target //src:diffutils failed to build
...</code></pre></div>
<h3 id="mirrors"><a class="anchor before" href="https://www.tweag.io/rss.xml#mirrors"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Mirrors</h3>
<p>Since Bazel 8.4, you can also use the <a href="https://bazel.build/reference/command-line-reference#common_options-flag--module_mirrors"><code class="language-text">--module_mirrors</code> flag</a>
to mirror the source archives.
To take advantage of this, add <code class="language-text">--module_mirrors=https://bcr.cloudflaremirrors.com</code> in your <code class="language-text">.bazelrc</code> file.
Keep in mind that this only applies to registry sources and not to other resources fetched by Bazel
(such as downloads happening in the <a href="https://bazel.build/rules/lib/builtins/repository_ctx#download">repository rules context</a>).</p>
<p>Note that for Bazel builds, the <a href="https://registry.bazel.build/">Bazel Central Registry (BCR)</a>
only stores metadata for a Bazel module;
the actual artifacts are usually fetched from URLs
that point to files hosted online (most often on GitHub).</p>
<p>BCR itself is a sort of external dependency for your builds, too.
Even though it’s hosted on production-grade infrastructure at Google, it can still be impacted by outages and operational mishaps.
The SSL certificate for <code class="language-text">mirror.bazel.build</code> has expired, causing worldwide CI breakages, at least twice:
once in <a href="https://github.com/bazelbuild/bazel/issues/15515">2022</a> and again in <a href="https://github.com/bazelbuild/bazel/issues/28101">2025</a>.
Refer to <a href="https://blog.bazel.build/2026/01/16/ssl-cert-expiry.html">Postmortem for bazel.build SSL certificate expiry</a> to learn more.</p>
<p>Configuring Bazel to use <code class="language-text">https://bcr.cloudflaremirrors.com</code> as a mirror for modules from the BCR helps,
but the Cloudflare mirror doesn’t cover the registry itself.
So if you want to go the extra mile, you might also consider setting up your <a href="https://bazel.build/external/registry#index_registry">own BCR index registry</a>
and point Bazel at that instead.
But if this is not feasible, <a href="https://github.com/DataDog/datadog-agent/pull/44621">write a playbook</a> for incident response
around build outages caused by external dependencies, so teams don’t have to improvise under pressure.</p>
<h3 id="pull-through-cache"><a class="anchor before" href="https://www.tweag.io/rss.xml#pull-through-cache"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Pull-through cache</h3>
<p>If your repository manager supports it, you could let your builds download external resources,
but every resource that is being fetched is saved into the cache as well.
On subsequent builds, the resources are going to be fetched from the cache, if available.
This would let you turn random external downloads into a controlled internal dependency
without requiring you to pre-vendor everything up front.</p>
<p>If your CI agents are in the same network or cloud region (depending on your infrastructure setup),
this could also speed up the builds by having downloads complete faster.
Not relying on external resources makes your Bazel builds also a lot more
<a href="https://bmitch.net/blog/2025-08-22-ghrc-appears-malicious/">secure</a> as your CI agents will only download data from a trusted source.</p>
<p>If using an off-the-shelf solution, such as the popular JFrog Artifactory, is not possible,
there are some other options.
Bazel picks up <a href="https://bazel.build/versions/8.2.0/external/advanced#using_proxies">proxy addresses</a> from the <code class="language-text">HTTP_PROXY</code> and <code class="language-text">HTTPS_PROXY</code> environment variables
and uses these to download files over HTTP and HTTPS, respectively (if specified).
This means you might have success with caching proxy solutions such as <a href="https://www.squid-cache.org/">Squid</a> and <a href="https://www.charlesproxy.com">Charles</a>
or by combining <a href="https://nginx.org/">Nginx</a> and <a href="https://www.varnish-software.com/products/varnish-cache/">Varnish</a> HTTP reverse proxies.
Routing requests through a proxy might also help to avoid rate limiting issues
since the external service will see fewer direct requests.</p>
<p>With this configuration, your downloader configuration file would look something like this:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text"># point all downloads at the mirror
rewrite (.*) {caching-service-url}/$1

# use the original location if the mirror is down
rewrite (.*) $1</code></pre></div>
<p>For a completely custom solution, take a look at the <a href="https://github.com/monogon/monogon/tree/main/build/mirror_proxy">Bazel downloader mirror from Monogon</a>
which can be used to mirror Bazel dependencies to a cloud bucket storage such as S3 or GCS.
Bazel’s <a href="https://github.com/bazelbuild/remote-apis">remote asset API</a> lets you use an existing remote cache
(content-addressable storage: CAS) as a downloader cache as well.
The cache provider service needs to support it, but many existing solutions, both commercial and open-source ones, are compatible.</p>
<p>The <a href="https://bazel.build/reference/command-line-reference#common_options-flag--experimental_remote_downloader"><code class="language-text">--experimental_remote_downloader</code> flag</a>
can be specified to provide a Remote Asset API endpoint URI to be used as a remote download proxy.
To get started, consider using <a href="https://github.com/buchgr/bazel-remote">bazel-remote</a>, which has out-of-the-box support for this use case.
Make sure to provide the <a href="https://bazel.build/rules/lib/repo/http#http_archive-sha256"><code class="language-text">sha256</code></a> for the assets to fetch
so that they can be cached just like any other CAS object.
A remote caching service will automatically download the assets from the URL if they are found in the CAS and cache it thereafter.</p>
<p>Bazel 9 adds support for <a href="https://github.com/bazelbuild/bazel/discussions/27509">remote repository caches</a>
which make Bazel builds (at least those requiring previously cached assets) extra resilient to external access issues.
During outages of external hosting services, those organizations that didn’t have a central repository manager
where repository rules artifacts could be stored had to extract files from cache directories on local developer machines
and save them to an accessible location within the internal network.</p>
<p>Now these artifacts will be saved into a remote cache similarly to build output results.
To confirm that your remote repository cache works as expected,
you can use the <a href="https://bazel.build/reference/command-line-reference#common_options-flag--repository_disable_download"><code class="language-text">--repository_disable_download</code> flag</a>
after doing a clean build (which should succeed as it will reuse the remote cache entries uploaded in the previous build).</p>
<h3 id="chaos-testing"><a class="anchor before" href="https://www.tweag.io/rss.xml#chaos-testing"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Chaos testing</h3>
<p>Finally, instead of waiting for the next GitHub outage, you can test your resilience
by intentionally breaking access to certain external hosts.
In a staging CI environment, temporarily block access to key external systems with firewall rules and verify
that your mirrors and caches are used as expected, builds either still succeed,
or fail fast with clear error messages, and your runbooks are correct and sufficient.</p>
<h2 id="conclusion"><a class="anchor before" href="https://www.tweag.io/rss.xml#conclusion"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Conclusion</h2>
<p>Bazel projects often depend on external services in subtle ways,
and any instability or change in those services can break otherwise healthy builds.
You can significantly improve build reliability by making all downloads explicit and verifiable,
routing them through managed infrastructure,
and tightening how and when network access is allowed.
Resilient Bazel builds come from treating external dependencies as first‑class operational risks
and turning unpredictable third‑party failures into controlled, recoverable events.</p></div>
    </summary>
    <updated>2026-04-02T00:00:00Z</updated>
    <published>2026-04-02T00:00:00Z</published>
    <source>
      <id>https://tweag.io</id>
      <author>
        <name>Tweag I/O</name>
      </author>
      <link href="https://tweag.io" rel="alternate" type="text/html"/>
      <link href="http://www.tweag.io/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Scale your engineering power. We enable deep-tech startups to achieve
their vision, from research to product delivery.</subtitle>
      <title>Tweag - Engineering blog</title>
      <updated>2026-04-02T12:24:07Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>http://www.well-typed.com/blog/2026/03/hs-bindgen-alpha2</id>
    <link href="https://well-typed.com/blog/2026/03/hs-bindgen-alpha2" rel="alternate" type="text/html"/>
    <title>Second pre-release of hs-bindgen</title>
    <summary>With heartfelt thanks to the many people who have already tried hs-bindgen and
given us feedback, we have steadily been working towards the first official
release (see Contributors for the full list). In case you missed
the announcement of the first alpha, hs-bindgen is a
tool for automatic construction [...]</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>With heartfelt thanks to the many people who have already tried <code>hs-bindgen</code> and
given us feedback, we have steadily been working towards the first official
release (see <a href="https://github.com/well-typed/hs-bindgen/tree/release-0.1-alpha2?tab=readme-ov-file#contributors">Contributors</a> for the full list). In case you missed
the <a href="https://well-typed.com/blog/2026/02/hs-bindgen-alpha/">announcement of the first alpha</a>, <code>hs-bindgen</code> is a
tool for automatic construction of Haskell bindings for C libraries: just point
it at a C header and let it handle the rest. Because we have fixed some critical
bugs in this alpha release, but we’re not quite ready yet for the first full
official release, we have <a href="https://github.com/well-typed/hs-bindgen/releases/tag/release-0.1-alpha2">tagged a second alpha release</a>. In the
remainder of this blog post we will briefly highlight the most important
changes; please refer to the <code>CHANGELOG.md</code> of
<a href="https://github.com/well-typed/hs-bindgen/blob/release-0.1-alpha2/hs-bindgen/CHANGELOG.md">hs-bindgen</a> and of
<a href="https://github.com/well-typed/hs-bindgen/blob/release-0.1-alpha2/hs-bindgen-runtime/CHANGELOG.md">hs-bindgen-runtime</a> for the full list of changes, as well as
for migration hints where we have introduced some minor backwards incompatible
changes.</p>

<h2 id="bugfixes">Bugfixes</h2>
<p>The most important fixes for bugs in the generated code are:</p>
<ul>
<li>The implementation of <code>peek</code> and <code>poke</code> for bitfields was broken, which could
lead to segfaults.</li>
<li>Duplicate record fields are now usable also in Template Haskell mode.</li>
<li>Patterns for unsigned enums now get the right value.</li>
</ul>
<p>We have also resolved a number of panics during code generation, but those would
not have resulted in incorrect generated code (merely in no code being generated
at all).</p>
<h2 id="new-features">New features</h2>
<ul>
<li><p><em>Implicit fields</em> arise when one struct (or union) is nested in another,
without any field name or tag:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb1-1"><a href="https://well-typed.com/blog/rss2.xml#cb1-1" tabindex="-1"/><span class="kw">struct</span> outer <span class="op">{</span></span>
<span id="cb1-2"><a href="https://well-typed.com/blog/rss2.xml#cb1-2" tabindex="-1"/>  <span class="dt">int</span> x<span class="op">;</span></span>
<span id="cb1-3"><a href="https://well-typed.com/blog/rss2.xml#cb1-3" tabindex="-1"/>  <span class="kw">struct</span> <span class="op">{</span></span>
<span id="cb1-4"><a href="https://well-typed.com/blog/rss2.xml#cb1-4" tabindex="-1"/>    <span class="dt">int</span> y<span class="op">;</span></span>
<span id="cb1-5"><a href="https://well-typed.com/blog/rss2.xml#cb1-5" tabindex="-1"/>    <span class="dt">int</span> z<span class="op">;</span></span>
<span id="cb1-6"><a href="https://well-typed.com/blog/rss2.xml#cb1-6" tabindex="-1"/>  <span class="op">};</span></span>
<span id="cb1-7"><a href="https://well-typed.com/blog/rss2.xml#cb1-7" tabindex="-1"/><span class="op">};</span></span></code></pre></div>
<p>We now support such implicit fields; both the inner (anonymous) struct as well
as the corresponding field of the outer struct will be named after the first
field of the inner struct<a class="footnote-ref" href="https://well-typed.com/blog/rss2.xml#fn1" id="fnref1"><sup>1</sup></a>:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb2-1"><a href="https://well-typed.com/blog/rss2.xml#cb2-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Outer</span> <span class="ot">=</span> <span class="dt">Outer</span> {</span>
<span id="cb2-2"><a href="https://well-typed.com/blog/rss2.xml#cb2-2" tabindex="-1"/><span class="ot">    x ::</span> <span class="dt">CInt</span></span>
<span id="cb2-3"><a href="https://well-typed.com/blog/rss2.xml#cb2-3" tabindex="-1"/>  ,<span class="ot"> y ::</span> <span class="dt">Outer_y</span></span>
<span id="cb2-4"><a href="https://well-typed.com/blog/rss2.xml#cb2-4" tabindex="-1"/>  }</span>
<span id="cb2-5"><a href="https://well-typed.com/blog/rss2.xml#cb2-5" tabindex="-1"/></span>
<span id="cb2-6"><a href="https://well-typed.com/blog/rss2.xml#cb2-6" tabindex="-1"/><span class="kw">data</span> <span class="dt">Outer_y</span> <span class="ot">=</span> <span class="dt">Outer_y</span> {</span>
<span id="cb2-7"><a href="https://well-typed.com/blog/rss2.xml#cb2-7" tabindex="-1"/><span class="ot">    y ::</span> <span class="dt">CInt</span></span>
<span id="cb2-8"><a href="https://well-typed.com/blog/rss2.xml#cb2-8" tabindex="-1"/>  ,<span class="ot"> z ::</span> <span class="dt">CInt</span></span>
<span id="cb2-9"><a href="https://well-typed.com/blog/rss2.xml#cb2-9" tabindex="-1"/>  }</span></code></pre></div>
<p>For this particular case we <em>could</em> also have chosen to flatten the structure
and add <code>y</code> and <code>z</code> directly to <code>Outer</code>, but that does not work in all cases
(for example, when we have an anonymous struct inside a union), so instead
we opt for consistency and always generate an explicit type for the
inner struct.</p></li>
<li><p>Unnamed bit-field declarations, which are used to control padding, are now
supported:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb3-1"><a href="https://well-typed.com/blog/rss2.xml#cb3-1" tabindex="-1"/><span class="kw">struct</span> bar <span class="op">{</span></span>
<span id="cb3-2"><a href="https://well-typed.com/blog/rss2.xml#cb3-2" tabindex="-1"/>  <span class="dt">signed</span> <span class="dt">char</span> x <span class="op">:</span> <span class="dv">3</span><span class="op">;</span></span>
<span id="cb3-3"><a href="https://well-typed.com/blog/rss2.xml#cb3-3" tabindex="-1"/>  <span class="dt">signed</span> <span class="dt">char</span>   <span class="op">:</span> <span class="dv">3</span><span class="op">;</span>  <span class="co">// Explicit padding</span></span>
<span id="cb3-4"><a href="https://well-typed.com/blog/rss2.xml#cb3-4" tabindex="-1"/>  <span class="dt">signed</span> <span class="dt">char</span> y <span class="op">:</span> <span class="dv">2</span><span class="op">;</span></span>
<span id="cb3-5"><a href="https://well-typed.com/blog/rss2.xml#cb3-5" tabindex="-1"/><span class="op">};</span></span></code></pre></div></li>
<li><p>We used to distinguish between <em>parse</em> predicates (which files should
<code>hs-bindgen</code> parse at all?) and <em>selection</em> predicates (for which C
declarations should we generate Haskell declarations?). This was confusing,
and as we are getting better at skipping over declarations with unsupported
features (and that list is dwinding anyway), parse predicates are not that
useful anymore. Parse predicates therefore have been removed entirely; we
simply always parse everything (<em>selection</em> predicates are still very much an
important feature of course).</p></li>
<li><p>Some infrastructure for and around binding specifications has been improved.
For example, we now distinguish between macros and non-macros of the same name,
and our treatment of arrays has changed slightly. For example, given</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb4-1"><a href="https://well-typed.com/blog/rss2.xml#cb4-1" tabindex="-1"/><span class="kw">typedef</span> <span class="dt">char</span> T <span class="op">[];</span></span>
<span id="cb4-2"><a href="https://well-typed.com/blog/rss2.xml#cb4-2" tabindex="-1"/><span class="dt">void</span> foo <span class="op">(</span>T xs<span class="op">);</span></span></code></pre></div>
<p>we now generate</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb5-1"><a href="https://well-typed.com/blog/rss2.xml#cb5-1" tabindex="-1"/><span class="ot">foo ::</span> <span class="dt">Ptr</span> (<span class="dt">Elem</span> <span class="dt">T</span>) <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span></code></pre></div>
<p>We do not use <code>Ptr CChar</code>, because <code>T</code> might have an existing binding in
another library (with an external binding specification), and we don’t know
what the type of the elements of <code>T</code> are (it could for example be some
newtype around <code>CChar</code>). <code>Elem</code> is a member of a new <code>IsArray</code> class, part of
the <code>hs-bindgen-runtime</code>.</p></li>
<li><p>Top-level anonymous enums are now supported. For example,</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb6-1"><a href="https://well-typed.com/blog/rss2.xml#cb6-1" tabindex="-1"/><span class="kw">enum</span> <span class="op">{</span>A<span class="op">,</span> B<span class="op">};</span></span></code></pre></div>
<p>results in</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb7-1"><a href="https://well-typed.com/blog/rss2.xml#cb7-1" tabindex="-1"/><span class="kw">pattern</span> <span class="dt">A</span><span class="ot"> ::</span> <span class="dt">CUInt</span></span>
<span id="cb7-2"><a href="https://well-typed.com/blog/rss2.xml#cb7-2" tabindex="-1"/><span class="kw">pattern</span> <span class="dt">A</span> <span class="ot">=</span> <span class="dv">0</span></span>
<span id="cb7-3"><a href="https://well-typed.com/blog/rss2.xml#cb7-3" tabindex="-1"/></span>
<span id="cb7-4"><a href="https://well-typed.com/blog/rss2.xml#cb7-4" tabindex="-1"/><span class="kw">pattern</span> <span class="dt">B</span><span class="ot"> ::</span> <span class="dt">CUInt</span></span>
<span id="cb7-5"><a href="https://well-typed.com/blog/rss2.xml#cb7-5" tabindex="-1"/><span class="kw">pattern</span> <span class="dt">B</span> <span class="ot">=</span> <span class="dv">1</span></span></code></pre></div>
<p>(Normally an <code>enum</code> results in a <code>newtype</code> around the enum’s underlying type,
and the patterns are for that <code>newtype</code> instead.)</p></li>
<li><p>We now generate bindings for static global variables (such globals are
sometimes used in headers that also contain static function bodies).</p></li>
<li><p>All definitions required by the generated code are now (re-)exported from
<code>hs-bindgen-runtime</code>, so that it becomes the only package dependency that
needs to be declared (no need for <code>ghc-prim</code> or <code>primitive</code> anymore).</p></li>
</ul>
<p>This list is not complete; some other less common edge cases have also been
implemented.</p>
<h2 id="conclusions">Conclusions</h2>
<p>Although we are still working on some finishing touches before we can release
the first official version of <code>hs-bindgen</code>, it is already being put to good use
on various projects. There are only a <a href="https://github.com/well-typed/hs-bindgen/issues?q=is%3Aissue%20state%3Aopen%20label%3Amissing-c-feature">handful of missing C
features</a> left, all of which low priority edge cases (though
if you have a specific use case for any of these, do let us know!). So if you
are interested, please do try it out, and let us know if you find any problems.
There should be no major breaking changes between now and the first official
release.</p>
<section class="footnotes footnotes-end-of-document" id="footnotes">
<hr/>
<ol>
<li id="fn1"><p>This is the version that uses the
<code>--omit-field-prefixes</code> option, which generates code that relies on
<code>DuplicateRecordFields</code> and <code>OverloadedRecordDot</code>.<a class="footnote-back" href="https://well-typed.com/blog/rss2.xml#fnref1">↩︎</a></p></li>
</ol>
</section></div>
    </content>
    <updated>2026-03-27T00:00:00Z</updated>
    <published>2026-03-27T00:00:00Z</published>
    <category term="hs-bindgen"/>
    <category term="open-source"/>
    <category term="foreign function interface"/>
    <author>
      <name>edsko</name>
    </author>
    <source>
      <id>https://well-typed.com/blog/</id>
      <logo>https://well-typed.com/img/wtnlogoplain.svg</logo>
      <link href="https://well-typed.com/blog/rss2.xml" rel="self" type="application/rss+xml"/>
      <link href="https://well-typed.com/blog/" rel="alternate" type="text/html"/>
      <subtitle>Because Well-Typed Programs Don't Go Wrong</subtitle>
      <title>Well-Typed Blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://haskell.org/ghc/blog/20260327-ghc-9.12.4-released.html</id>
    <link href="http://haskell.org/ghc/blog/20260327-ghc-9.12.4-released.html" rel="alternate" type="text/html"/>
    <title>GHC 9.12.4 is now available</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><h1>GHC 9.12.4 is now available</h1>
<h4 class="text-muted">wz1000 - 2026-03-27</h4>

<p>The GHC developers are very pleased to announce the release of GHC 9.12.4.
Binary distributions, source distributions, and documentation are available at
<a href="https://downloads.haskell.org/ghc/9.12.4">downloads.haskell.org</a> and via <a href="https://www.haskell.org/ghcup/">GHCup</a>.</p>
<p>GHC 9.12.4 is a bug-fix release fixing many issues of a variety of
severities and scopes, including:</p>
<ul>
<li><p>Fixed a critical code generation regression where sub-word division produced
incorrect results (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26711">#26711</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26668">#26668</a>), similar to the bug fixed in 9.12.2</p></li>
<li><p>Numerous fixes for register allocation bugs, preventing data corruption
when spilling and reloading registers
(<a href="https://gitlab.haskell.org/ghc/ghc/issues/26411">#26411</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26526">#26526</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26537">#26537</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26542">#26542</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26550">#26550</a>)</p></li>
<li><p>Fixes for several compiler crashes, including issues with CSE (<a href="https://gitlab.haskell.org/ghc/ghc/issues/25468">#25468</a>),
and the simplifier(<a href="https://gitlab.haskell.org/ghc/ghc/issues/26681">#26681</a>), implicit parameters (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26451">#26451</a>), and the type-class
specialiser (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26682">#26682</a>)</p></li>
<li><p>Fixed cast worker/wrapper incorrectly firing on INLINE functions (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26903">#26903</a>)</p></li>
<li><p>Fixed LLVM backend miscompilation of bit manipulation operations
(<a href="https://gitlab.haskell.org/ghc/ghc/issues/20645">#20645</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26065">#26065</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26109">#26109</a>)</p></li>
<li><p>Fixed associated type family and data family instance changes not triggering
recompilation (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26183">#26183</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26705">#26705</a>)</p></li>
<li><p>Fixed negative type literals causing the compiler to hang (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26861">#26861</a>)</p></li>
<li><p>Improvements to determinism of compiler output (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26846">#26846</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26858">#26858</a>)</p></li>
<li><p>Fixes for eventlog shutdown deadlocks (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26573">#26573</a>)
and lost wakeups in the RTS (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26324">#26324</a>)</p></li>
<li><p>Fixed split sections support on Windows (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26696">#26696</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26494">#26494</a>) and the LLVM backend (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26770">#26770</a>)</p></li>
<li><p>Fixes for the bytecode compiler, PPC native code generator, and Wasm backend</p></li>
<li><p>The runtime linker now supports COMMON symbols (<a href="https://gitlab.haskell.org/ghc/ghc/issues/6107">#6107</a>)</p></li>
<li><p>Improved backtrace support: backtraces for <code>error</code> exceptions are now
evaluated at throw time</p></li>
<li><p><code>NamedDefaults</code> now correctly requires the class to be standard or have an
in-scope default declaration, and handles poly-kinded classes (<a href="https://gitlab.haskell.org/ghc/ghc/issues/25775">#25775</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/25778">#25778</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/25882">#25882</a>)</p></li>
<li><p>… and many more</p></li>
</ul>
<p>A full accounting of these fixes can be found in the <a href="https://gitlab.haskell.org/ghc/ghc/-/blob/ghc-9.12/docs/users_guide/9.12.4-notes.rst?ref_type=heads&amp;plain=1">release notes</a>. As
always, GHC’s release status, including planned future releases, can be found on
the GHC Wiki <a href="https://gitlab.haskell.org/ghc/ghc/-/wikis/GHC-status">status</a>.</p>
<p>GHC development is sponsored by:</p>
<ul>
<li><a href="https://juspay.com/">Juspay</a></li>
<li><a href="https://qbaylogic.com/">QBayLogic</a></li>
<li><a href="https://www.channable.com/">Channable</a></li>
<li><a href="https://haskell.foundation/">Haskell Foundation</a></li>
<li><a href="https://serokell.io/">Serokell</a></li>
<li><a href="https://well-typed.com/">Well-Typed</a></li>
<li><a href="https://www.tweag.io/">Tweag</a></li>
<li><a href="https://www.dotcom-monitor.com/">Dotcom-Monitor</a></li>
<li><a href="https://www.loadview-testing.com/">LoadView</a></li>
<li><a href="https://webhostingbuddy.com/">Web Hosting Buddy</a></li>
<li><a href="https://www.findmyelectric.com/">Find My Electric</a></li>
<li><a href="https://www.sc.com">Standard Chartered</a></li>
<li><a href="https://upcloud.com">UpCloud</a></li>
<li><a href="https://mercury.com">Mercury</a></li>
</ul>
<p>We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.</p>
<p>As always, do give this release a try and open a <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/new">ticket</a> if you see
anything amiss.</p></div>
    </summary>
    <updated>2026-03-27T00:00:00Z</updated>
    <published>2026-03-27T00:00:00Z</published>
    <author>
      <name>ghc-devs</name>
    </author>
    <source>
      <id>http://haskell.org/ghc</id>
      <link href="http://haskell.org/ghc" rel="alternate" type="text/html"/>
      <link href="http://haskell.org/ghc/rss.xml" rel="self" type="application/rss+xml"/>
      <title>GHC Developer blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://medium.com/p/539cb2f40215</id>
    <link href="https://cdsmithus.medium.com/athena-loses-a-bet-539cb2f40215?source=rss-18bd5acaea78------2" rel="alternate" type="text/html"/>
    <title>Athena Loses a Bet</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*weyULcS-xuFt4Akf6f4vpA.png"/></figure><p>Athena and Ares argue over human nature, and agree to test three great minds of the age.</p><p>First, they approach Aristotle in the Lyceum and propose a bargain. “If you ask it of us, the one you love most in the world will perish, but you will be made rich beyond imagining.” Aristotle barely hesitates. “No,” he says. “To destroy the very purpose of living for the sake of the mere means is the mark of a man who lacks wisdom.”</p><p>Next, they approach Plato, finding him pacing in an olive grove of his Academy. They offer the same proposal. “I decline,” he says. “Love allows us to glimpse the ideal of pure beauty, but wealth is an anchor to the material world.”</p><p>Finally, they approach Socrates, wandering barefoot in the crowded dusty stalls of the Agora. The gods approach him with the same bargain: “If you ask it of us, Xanthippe, whom you love most in the world, will perish — ”</p><p>“I ask it!” he blurts out.</p><p>Athena blinks. “You did not even hear the rest. We were going to say you would be given wealth beyond measure.”</p><p>Socrates shrugs. “Keep it. This was never about money.”</p><p>Millenia later, Athena is still smarting from losing the bet, and she demands a rematch. Searching for another Greek philosopher, they instead find a middle aged woman writing a novel called <em>Atlas Shrugged</em>. She’s a philosopher, and Atlas was Greek, so that’s close enough.</p><p>“If you ask it,” Athena says to her, “we will make you wealthy beyond measure, but then in return, your true love will be taken from you.”</p><p>The woman looks up, bored, and asks “Why give me the money if you’re just going to take it right back?”</p><img alt="" height="1" src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=539cb2f40215" width="1"/></div>
    </content>
    <updated>2026-03-25T18:58:04Z</updated>
    <published>2026-03-25T18:58:04Z</published>
    <category term="jokes"/>
    <author>
      <name>Chris Smith</name>
    </author>
    <source>
      <id>https://medium.com/@cdsmithus?source=rss-18bd5acaea78------2</id>
      <logo>https://cdn-images-1.medium.com/fit/c/150/150/1*rU9ddF_bkph8Qg3jip7vfw.jpeg</logo>
      <link href="https://medium.com/@cdsmithus?source=rss-18bd5acaea78------2" rel="alternate" type="text/html"/>
      <link href="https://medium.com/@cdsmithus/feed" rel="self" type="application/rss+xml"/>
      <link href="http://medium.superfeedr.com" rel="hub" type="text/html"/>
      <subtitle>Stories by Chris Smith on Medium</subtitle>
      <title>Stories by Chris Smith on Medium</title>
      <updated>2026-04-12T01:59:24Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>Buzzsprout-18853260</id>
    <link href="https://www.buzzsprout.com/1817535/episodes/18853260-79-peter-thiemann.mp3" length="47963255" rel="enclosure" type="audio/mpeg"/>
    <title>79: Peter Thiemann</title>
    <summary>Peter is a professor at the University of Freiburg, and he was doing functional programming right when Haskell got started. So naturally we asked him about the early days of Haskell, and how from the start Peter pushed the envelope on what you could do with the type system and specifically with the type classes, from early web programming to program generation to session types. Come with us on a trip down memory lane!</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Peter is a professor at the University of Freiburg, and he was doing functional programming right when Haskell got started. So naturally we asked him about the early days of Haskell, and how from the start Peter pushed the envelope on what you could do with the type system and specifically with the type classes, from early web programming to program generation to session types. Come with us on a trip down memory lane!</p><p><br/></p></div>
    </content>
    <updated>2026-03-22T12:00:00Z</updated>
    <published>2026-03-22T12:00:00Z</published>
    <author>
      <name>Haskell Podcast</name>
    </author>
    <source>
      <id>https://haskell.foundation/podcast/</id>
      <logo>https://storage.buzzsprout.com/tnk1ztokn5vmeiufqmr4kkp37mw2?.jpg</logo>
      <category scheme="http://www.itunes.com/" term="Technology"/>
      <author>
        <name>Haskell Podcast</name>
      </author>
      <link href="https://rss.buzzsprout.com/1817535.rss" rel="self" type="application/rss+xml"/>
      <link href="https://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <link href="https://haskell.foundation/podcast/" rel="alternate" type="text/html"/>
      <rights>Â© 2026 The Haskell Interlude</rights>
      <subtitle>This is the Haskell Interlude, where the five co-hosts (Wouter Swierstra, Andres LÃ¶h, Alejandro Serrano, Niki Vazou, and Joachim Breitner) chat with Haskell guests!</subtitle>
      <title>The Haskell Interlude</title>
      <updated>2026-04-10T16:12:47Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>http://www.well-typed.com/blog/2026/03/haskell-ecosystem-report-q1-2026</id>
    <link href="https://well-typed.com/blog/2026/03/haskell-ecosystem-report-q1-2026" rel="alternate" type="text/html"/>
    <title>Haskell ecosystem activities report: December 2025â€“February 2026</title>
    <summary>This is the thirtieth edition of our Haskell ecosystem activities report,
which describes the work Well-Typed are doing on GHC, Cabal, HLS and other parts
of the core Haskell toolchain. The current edition covers roughly the months of
December 2025 to February 2026.

You can find the previous editions [...]</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>This is the thirtieth edition of our Haskell ecosystem activities report,
which describes the work Well-Typed are doing on GHC, Cabal, HLS and other parts
of the core Haskell toolchain. The current edition covers roughly the months of
December 2025 to February 2026.</p>
<p>You can find the previous editions collected under the
<a href="https://well-typed.com/blog/tags/haskell-ecosystem-report">haskell-ecosystem-report tag</a>.</p>
<h2 id="sponsorship">Sponsorship</h2>
<p>We offer <a href="https://well-typed.com/ecosystem/">Haskell Ecosystem Support Packages</a> to provide commercial
users with support from Well-Typed’s experts while investing in the Haskell
community and its technical ecosystem including through the work described in
this report. To find out more, read our <a href="https://well-typed.com/blog/2025/06/haskell-ecosystem-support-packages">announcement of these
packages</a> in partnership with
the Haskell Foundation. We need funding to continue this essential maintenance work!</p>
<p>Many thanks to our Haskell Ecosystem Supporters: <a href="https://www.sc.com/">Standard Chartered</a>,
<a href="https://www.channable.com/">Channable</a> and <a href="https://qbaylogic.com/">QBayLogic</a>,
as well as to our other clients who also contribute to making this work possible:
<a href="https://www.anduril.com/">Anduril</a>, <a href="https://juspay.in/">Juspay</a> and <a href="https://mercury.com/">Mercury</a>;
and to the <a href="https://opencollective.com/haskell-language-server">HLS Open Collective</a> for
supporting HLS release management.</p>
<h2 id="team">Team</h2>

<p><a href="https://well-typed.com/people/matthew">Matthew Pickering</a> announced that he will be leaving the company and moving to a non-Haskell
role at the end of March.
Working with Matt has been a joy – more than his deep technical insight
or sharp intuition, it’s the warmth of his vision for how to work together and
his generosity that has made him such a force within the team.
He was also a beacon that could rally the community in difficult times, perhaps
most memorably with his technical and social contributions in consolidating
Haskell IDEs with the creation of the Haskell Language Server.
His dedication to tooling has also been an inspiration, with his work on
<code>ghc-debug</code> and on profiling an invaluable contribution to our understanding
of memory usage of Haskell programs.</p>
<p>The Haskell toolchain team at Well-Typed currently includes:</p>
<ul>
<li><a href="https://well-typed.com/people/andreask">Andreas Klebinger</a></li>
<li><a href="https://well-typed.com/people/hannes">Hannes Siebenhandl</a></li>
<li><a href="https://well-typed.com/people/magnus">Magnus Viernickel</a></li>
<li><a href="https://well-typed.com/people/mikolaj">Mikolaj Konarski</a></li>
<li><a href="https://well-typed.com/people/rodrigo">Rodrigo Mesquita</a></li>
<li><a href="https://well-typed.com/people/sam">Sam Derbyshire</a></li>
<li><a href="https://well-typed.com/people/zubin">Zubin Duggal</a></li>
</ul>
<p>In addition, many others within Well-Typed contribute to GHC, Cabal, HLS
and other open source Haskell libraries and tools.
This report includes contributions from <a href="https://well-typed.com/people/alex">Alex Washburn</a>,
<a href="https://well-typed.com/people/duncan">Duncan Coutts</a>,
<a href="https://well-typed.com/people/wen">Wen Kokke</a> and <a href="https://well-typed.com/people/wolfgang">Wolfgang Jeltsch</a> in
particular.</p>
<p>We are active participants in community efforts for developing the Haskell language and libraries.
Rodrigo joined the <a href="https://github.com/ghc-proposals/ghc-proposals">GHC Steering Committee</a> in December,
alongside <a href="https://well-typed.com/people/adam">Adam Gundry</a>.
Wolfgang joined the <a href="https://github.com/haskell/core-libraries-committee">Core Libraries Committee</a> in February.</p>
<h2 id="highlights">Highlights</h2>
<h3 id="interactive-step-through-debugging">Interactive step-through debugging</h3>
<p>The <a href="https://well-typed.github.io/haskell-debugger/">Haskell Debugger</a> (<code>hdb</code>) has been made more robust and more features were implemented by Rodrigo, Matthew, and Hannes.
Most notably, the debugger now:</p>
<ul>
<li>Displays stack traces for bytecode and compiled code frames (provided the program and dependencies were compiled with <code>-finfo-table-map</code> for the latter)</li>
<li>Displays source locations and callstacks for exception breakpoints</li>
<li>Uses the external interpreter by default</li>
<li>Can be run on GHC itself!</li>
</ul>
<p>To run <code>hdb</code> you need to use GHC 9.14 and to configure the IDE accordingly. Please refer to the <a href="https://well-typed.github.io/haskell-debugger/#installation">installation instructions</a>. Apart from that, if HLS just works on your codebase, so should the debugger!</p>
<h3 id="live-monitoring-using-the-eventlog">Live monitoring using the eventlog</h3>
<p>GHC’s eventlog already lets Haskell programs emit rich runtime telemetry, but
the workflow has historically been to run the program to completion and inspect
the eventlog afterwards. <a href="https://github.com/well-typed/eventlog-live/"><code>eventlog-live</code></a>
allows us instead to monitor the program as it is running. Wen continued work on
this project, taking significant steps towards making it production-ready, including:</p>
<ul>
<li><p>extending <code>eventlog-live</code> with support for the OpenTelemetry protocol
(<a href="https://github.com/well-typed/eventlog-live/pull/119"><span>#119</span></a>),</p></li>
<li><p>bringing the underlying
<a href="https://github.com/well-typed/eventlog-socket/"><code>eventlog-socket</code></a> library
closer to being ready for general use, by
adding a testsuite (<a href="https://github.com/well-typed/eventlog-socket/pull/27"><span>#27</span></a>).
fixing a litany of issues with the C code
(<a href="https://github.com/well-typed/eventlog-socket/pull/38"><span>#38</span></a>), and
finalising the user-facing API (<a href="https://github.com/well-typed/eventlog-socket/pull/43"><span>#43</span></a>),</p></li>
<li><p>adding support for custom commands in
<a href="https://github.com/well-typed/eventlog-socket/"><code>eventlog-socket</code></a>
(<a href="https://github.com/well-typed/eventlog-socket/pull/36"><span>#36</span></a>).</p></li>
</ul>
<h3 id="trees-that-grow">Trees That Grow</h3>
<p>The <code>Language.Haskell.Syntax</code> module hierarchy is intended to be a stable,
public API for the Haskell AST — one that external tools could eventually depend
on without coupling themselves to GHC internals, reducing ecosystem breakage.
Right now, that goal is undermined by lingering dependencies on internal modules
under the <code>GHC</code> hierarchy.</p>
<p>Alex, with help from Rodrigo, has been systematically removing these edges in
the dependency graph:</p>
<ul>
<li><code>Language.Haskell.Syntax.Type</code> no longer depends <code>GHC.Utils.Panic</code>
(<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15134">!15134</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26626">#26626</a>).</li>
<li><code>Language.Haskell.Syntax.Decls</code> no longer depends on <code>GHC.Unit.Module.Warnings</code>
(<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15146">!15146</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26636">#26636</a>), nor on <code>GHC.Types.ForeignCall</code> (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15477">!15477</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26700">#26700</a>) or
<code>GHC.Types.Basic</code> (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15265">!15265</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26699">#26699</a>).</li>
<li><code>Language.Haskell.Syntax.Binds</code> no longer depends on <code>GHC.Types.Basic</code>
(<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15187">!15187</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26670">#26670</a>).</li>
</ul>
<p>Once this work is done, it will be possible to consider moving the AST into a
separate package, and taking further steps towards increasing modularity of the
compiler.</p>
<h3 id="towards-a-standalone-base-package">Towards a standalone <code>base</code> package</h3>
<p>Historically, the <code>base</code> package was used as both the user-facing standard
library and a repository of GHC-specific internals, with much special treatment
in the compiler. This means GHC and <code>base</code> versions are tightly coupled, and
makes upgrading to new compiler versions unnecessarily difficult.</p>
<p>GHC developers have made significant progress towards making <code>base</code> a normal Haskell
package: <code>ghc-internal</code> has been split out as a separate library, <code>base</code> no
longer has a privileged unit-id in the compiler, and Cabal now allows
reinstalling it.</p>
<p>Matt posted a <a href="https://github.com/haskell/core-libraries-committee/issues/375#issuecomment-3646780735">summary of progress</a>
and outlined possible next steps
to seek community consensus on the direction of travel.
The <a href="https://github.com/well-typed/reinstallable-base">reinstallable-base</a> repository
collects documents and discussion on the effort.</p>
<p>Wolfgang continued various pieces of technical groundwork:</p>
<ul>
<li><p>cleaning up many unused known-key names in the compiler
(<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15184">!15184</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15190">!15190</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15211">!15211</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15213">!15213</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15217">!15217</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15218">!15218</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15219">!15219</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15215">!15215</a>),</p></li>
<li><p>finishing the process of removing <code>GHC.Desugar</code> from <code>base</code> (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15433">!15433</a>),</p></li>
<li><p>refining the import list of <code>System.IO.OS</code> to aid in modularity (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15567">!15567</a>).</p></li>
</ul>
<p>Wolfgang improved the public API of <code>base</code> relating to OS handles, to make the
API more stable across platforms and avoid the need for users to depend on
GHC-internal implementation details (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14732">!14732</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14905">!14905</a>). While in the area, he
fixed a bug in the implementation of <code>hIsReadable</code> and <code>hIsWritable</code> for duplex
handles (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26479">#26479</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15227">!15227</a>), and a mistake in the documentation of <code>hIsClosed</code>
(<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15228">!15228</a>).</p>
<h3 id="incorrect-absence-analysis-in-ghc">Incorrect absence analysis in GHC</h3>
<p>GHC bug <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26416">#26416</a> has occupied the attention of the team for quite some time.
Initially thought to be an issue with specialisation, a reproducer that Sam and
Magnus created showed that the issue is in fact a bug in absence analysis
— an optimisation that identifies and removes unused function arguments —
in which GHC would erroneously conclude that a used argument was in fact absent.</p>
<p>Andreas helped investigate the root cause, before Zubin finally took the torch
and put up a solution (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15238">!15238</a>).</p>
<h3 id="ghc-changelogs">GHC changelogs</h3>
<p>GHC’s changelogs have not always been as complete or reliable as the
community deserves. Keeping changelogs accurate across backports has also been a
major source of frustration for release managers.</p>
<p>This is why, after a discussion initiated by Teo Camarasu in <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26002">#26002</a>, we have decided
to adopt the <a href="https://codeberg.org/fgaz/changelog-d"><code>changelog.d</code></a> system —
already in use by the Cabal project — in which each change is a separate file
in the changelog directory.
This eliminates the merge conflicts that make backporting painful, and makes it
easier to associate MRs with changelog entries.</p>
<p>Zubin has been spearheading the effort, with the intention to switch to this
new method of changelog generation right after the fork date for GHC 10.0.</p>
<h2 id="ghc">GHC</h2>
<h3 id="ghc-releases">GHC Releases</h3>
<ul>
<li>Zubin worked on 9.12.3, backporting patches and preparing release candidates,
with a final release on the 27th of December.</li>
<li>Magnus and Zubin worked on backports for 9.12.4.</li>
<li>Zubin worked on 9.14.1, putting out the final release on the 19th of December.</li>
</ul>
<h3 id="frontend">Frontend</h3>
<ul>
<li><p>Sam reviewed the implementation of the <code>QualifiedStrings</code>
extension by Brandon Chinn (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14975">!14975</a>).
This allows string literals of the form <code>ModName."foo"</code>
(interpreted as <code>ModName.fromString ("foo" :: String)</code>).</p></li>
<li><p>Sam made several changes to the treatment of <code>Coercible</code> constraints in the
typechecker (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14100">!14100</a>):</p>
<ul>
<li>Defaulting of representational equalities to nominal equalities, functionality
previously added to GHC by Sam, is now more robust (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/25825">#25825</a>).</li>
<li>Error messages involving unsolved <code>Coercible</code> constraints are greatly
improved, an oft-requested improvement (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/15850">#15850</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/20289">#20289</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/23731">#23731</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26137">#26137</a>).
Error messages now consistently mention relevant out-of-scope data
constructors, provide import suggestions, and include additional
explanations about roles (when relevant).</li>
</ul></li>
<li><p>Magnus implemented several fixes to the implementation of <code>ExplicitLevelImports</code>:</p>
<ul>
<li>a missing check for types (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26098">#26098</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15119">!15119</a>),</li>
<li>a GHC panic in the driver (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26568">#26568</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15118">!15118</a>).</li>
</ul></li>
<li><p>Sam improved the reporting of “valid hole fits”, adding support for suggesting
bidirectional pattern synonyms (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26339">#26339</a>) and properly dealing with data
constructors with linear arguments (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26338">#26338</a>).</p></li>
<li><p>Sam investigated a typechecking regression starting in GHC 9.2 with the
introduction of the <code>Assert</code> type family to improve error messages involving
comparison of type-level literals (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26190">#26190</a>), posting <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26190#note_646511">his analysis</a> to the ticket.
To tackle this, he opened <a href="https://github.com/ghc-proposals/ghc-proposals/pull/735">GHC proposal <span>#735</span></a>, which is still in need of further community feedback.</p></li>
<li><p>Sam minimised a bug with rewrite rules (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26682">#26682</a>), which allowed Simon Peyton Jones
to identify and fix the bug (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15208">!15208</a>).</p></li>
<li><p>Sam improved how existential variables are displayed in Haddock documentation
(<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15099">!15099</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26252">#26252</a>).</p></li>
</ul>
<h4 id="determinism">Determinism</h4>
<ul>
<li>Matt identified and fixed several ways in which GHC compilation was not deterministic:
<ul>
<li>an issue with non-deterministic documentation information (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26858">#26858</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15482">!15482</a>).</li>
<li>non-determinism of constraint solving impacting generated <code>Typeable</code> evidence (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26846">#26846</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15442">!15442</a>).</li>
<li>issues with the Template Haskell machinery of the <code>singletons</code> library producing non-deterministic names
(<a href="https://github.com/goldfirere/singletons/pull/629"><code>singletons</code> <span>#629</span></a>,
<a href="https://github.com/goldfirere/th-desugar/pull/240"><code>th-desugar</code> <span>#240</span></a>).</li>
</ul></li>
</ul>
<h4 id="plugins">Plugins</h4>
<ul>
<li><p>Sam finished up and landed a long-standing MR by Chris Wendt (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/10133">!10133</a>) which
fixed a plugin-related issue.</p></li>
<li><p>Sam fixed a regression in <a href="https://hackage.haskell.org/package/ghc-typelits-natnormalise"><code>ghc-typelits-natnormalise</code></a>
in which the plugin would cause GHC to fall into an infinite loop
(<a href="https://github.com/clash-lang/ghc-typelits-natnormalise/issues/116"><code>ghc-typelits-natnormalise</code> <span>#116</span></a>, <a href="https://github.com/clash-lang/ghc-typelits-natnormalise/pull/118"><span>#118</span></a>).</p></li>
</ul>
<h3 id="backend">Backend</h3>
<ul>
<li><p>Rodrigo announced that work described in <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/23218">#23218</a> evolved into the <a href="https://dl.acm.org/doi/10.1145/3776711">POPL 2026
paper “Lazy Linearity for a Core Functional Language”</a>,
which presents a way to type linearity in GHC Core that is robust to almost
all GHC optimisations, together with a GHC plugin validating programs at
each optimisation stage.</p></li>
<li><p>With the oversight of Andreas, Sam carefully reconsidered the treatment of
register formats in the register allocator and liveness analysis. This
culminated in <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15121">!15121</a>:</p>
<ul>
<li>Keep track of register formats in liveness analysis (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26526">#26526</a>).</li>
<li>Use the right format when reloading spilled register (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26411">#26411</a>).</li>
<li>Enforce the invariant that writes to a register re-defined the format that
this register is used at for the purposes of liveness analysis, fixing another
bug reported by <code>@aratamizuki</code> on <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15121#note_647685"><span>!15121</span></a>.</li>
</ul></li>
<li><p>Sam put up a small fix for the mapping of registers to stack slots, fixing
an oversight in the case that registers start off small and are subsequently
written at larger widths (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26668">#26668</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15185">!15185</a>).</p></li>
<li><p>Sam reviewed a GHC contribution by <code>@sgillespie</code> adding SIMD primops for
<code>abs</code> and <code>sqrt</code> operations (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15236">!15236</a>), suggesting more efficient implementations of
certain operations.</p></li>
<li><p>Andreas investigated potential missed specialisations,
which allowed Simon Peyton Jones to make further progress in
improving the specialiser (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26831">#26831</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15441">!15441</a>).</p></li>
<li><p>Sam investigated several bugs to do with the interactions of join points with
ticks (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/14242">#14242</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26157">#26157</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26642">#26642</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26693">#26693</a>) and casts (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/14610">#14610</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/21716">#21716</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26422">#26422</a>).
He fixed the main bug (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26642">#26642</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15538">!15538</a>), which was due to incorrect
transformations in <code>mergeCaseAlts</code>. He also undertook a general refactor of
the area and, pinning down the overall handling of casts and ticks under
join points in a Note.</p></li>
</ul>
<h3 id="runtime-system-and-linker">Runtime system and linker</h3>
<ul>
<li><p>Matt fixed a decoding failure for <code>stg_dummy_ret</code> by using <code>INFO_TABLE_CONSTR</code>
for its closure (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26745">#26745</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15303">!15303</a>).</p></li>
<li><p>Duncan fixed long-standing inconsistencies in eventlog <code>STOP_THREAD</code> status
codes (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26867">#26867</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15522">!15522</a>).</p></li>
<li><p>Andreas improved the documentation of the <code>-K</code> RTS flag in <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15365">!15365</a> (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26354">#26354</a>).</p></li>
</ul>
<h3 id="exception-backtraces-stack-annotations-and-stack-decoding">Exception backtraces, stack annotations and stack decoding</h3>
<ul>
<li><p>Matt and Hannes improved the reporting of backtraces when using <code>error</code>
(<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15306">!15306</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15395">!15395</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26751">#26751</a>). This involved opening two CLC proposals
(<a href="https://github.com/haskell/core-libraries-committee/issues/383">CLC <span>#383</span></a>,
<a href="https://github.com/haskell/core-libraries-committee/issues/387">CLC <span>#387</span></a>).</p></li>
<li><p>Hannes continued working on the implementation of stack annotations and stack
decoding (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26218">#26218</a>), including:</p>
<ul>
<li>integrating <a href="https://github.com/well-typed/ghc-stack-profiler"><code>ghc-stack-profiler</code></a>,
a profiler that relies on stack annotations instead of heavier profiling
mechanisms, with the <a href="https://github.com/well-typed/eventlog-socket"><code>eventlog-socket</code></a>
library; and</li>
<li>working on the <a href="https://github.com/well-typed/ghc-stack-annotations"><code>ghc-stack-annotations</code></a>
compatibility library for annotating the stack.</li>
</ul></li>
<li><p>Rodrigo removed an incorrect assertion that fired when decoding a BCO whose
bitmap has no payload (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26640">#26640</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15136">!15136</a>).</p></li>
</ul>
<h3 id="build-system-and-packaging">Build system and packaging</h3>
<ul>
<li><p>Zubin fixed a GHC 9.14.1 build issue due to missing <code>.cabal</code> files for
<code>ghc-experimental</code> and <code>ghc-internal</code> in the source tarball (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26738">#26738</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15391">!15391</a>).</p></li>
<li><p>Andreas investigated the use of Cabal’s <code>--semaphore</code> feature to speed up GHC builds slightly (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26876">#26876</a>, <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15483">!15483</a>).
There are some issues preventing us from enabling this unconditionally
(<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26977">#26977</a>, <a href="https://github.com/haskell/cabal/issues/11557"><code>Cabal</code> <span>#11557</span></a>).</p></li>
</ul>
<h3 id="ci-and-testing">CI and testing</h3>
<ul>
<li><p>Magnus ensured the user’s guide can be generated with old versions of Python
to fix CI build failures on some older containers (<a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/15127">!15127</a>).</p></li>
<li><p>Magnus updated the Debian images used for CI (<a href="https://gitlab.haskell.org/ghc/ci-images/-/merge_requests/183"><code>ci-images</code> <span>!183</span></a>,
<a href="https://gitlab.haskell.org/ghc/ci-images/-/merge_requests/178"><span>!178</span></a>).</p></li>
<li><p>Sam finished up the work of Sven Tennie on testing floating point expressions
in the <a href="https://gitlab.haskell.org/ghc/test-primops"><code>test-primops</code></a> test
framework for GHC (<a href="https://gitlab.haskell.org/ghc/test-primops/-/merge_requests/19"><code>test-primops</code> <span>!19</span></a>).
This is preparatory work for improving the robustness of GHC’s handling of
floating point (<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26919">#26919</a>).</p></li>
<li><p>Andreas updated the <code>nofib</code> GHC benchmarking suite to fix issues that Sam ran
into when trying to use it, updating the CI in the process
(<a href="https://gitlab.haskell.org/ghc/nofib/-/merge_requests/81"><code>nofib</code> <span>!81</span></a>, <a href="https://gitlab.haskell.org/ghc/nofib/-/merge_requests/82"><span>!82</span></a>, <a href="https://gitlab.haskell.org/ghc/nofib/-/merge_requests/83"><span>!83</span></a>).</p></li>
</ul>
<h3 id="infrastructure">Infrastructure</h3>
<ul>
<li><p>Magnus worked on the infrastructure for the GitLab instance used for the GHC
project, bringing up new runners for CI and switching to a new verification
system to approve new users which makes it easier for new contributors to
open issues.</p></li>
<li><p>Magnus and Andreas helped the Haskell infrastructure team address Gitlab outages on short notice in order to improve availability of the GHC Gitlab instance.</p></li>
<li><p>Andreas and Magnus organized temporary CI capabilities sponsored by WT during a temporary outage of one of GHC’s CI runners.</p></li>
</ul>
<h2 id="cabal">Cabal</h2>
<ul>
<li><p>Sam added support for setting the logging handle via the library interface of <code>Cabal</code>,
a significant milestone in updating <code>cabal-install</code> to compile packages with
the <code>Cabal</code> library without invoking external processes (<a href="https://github.com/haskell/cabal/pull/11077"><code>Cabal</code> <span>#11077</span></a>).</p></li>
<li><p>Matt helped Matthías Páll Gissurarson to fix a bug in which <code>cabal haddock</code> was looking for
files in the wrong directory (<a href="https://github.com/haskell/cabal/issues/11475"><code>Cabal</code> <span>#11475</span></a>, <a href="https://github.com/haskell/cabal/pull/11476"><span>#11476</span></a>).</p></li>
<li><p>Matt fixed a bug with broken Haddocks locally due to non-expanded <code>${pkgroot}</code>
variable (<a href="https://github.com/haskell/cabal/issues/11217"><code>Cabal</code> <span>#11217</span></a>,
<a href="https://github.com/haskell/cabal/pull/11218"><span>#11218</span></a>).</p></li>
<li><p>Matt fixed some issues with <code>cabal repl</code> silently failing
(<a href="https://github.com/haskell/cabal/issues/11107"><code>Cabal</code> <span>#11107</span></a>,
<a href="https://github.com/haskell/cabal/pull/11237"><span>#11237</span></a>).</p></li>
</ul>
<h2 id="hls">HLS</h2>
<ul>
<li><p>In collaboration with Zubin and Andreas, Hannes investigated the root cause of
<a href="https://github.com/haskell/haskell-language-server/issues/4674">HLS issue <span>#4674</span></a>,
posting his analysis in <a href="https://github.com/haskell/haskell-language-server/issues/4674#issuecomment-3527937881">this comment</a>.
In short, the problem was that the <code>hlint</code> plugin was using an incompatible
version of <code>ghc-lib-parser</code>, and a version mismatch in this library was causing
segfaults due to changes to the <code>GHC.Data.FastString</code> implementation between
the versions.
Hannes disabled the <code>hlint</code> plugin on GHC 9.10 to work around this issue
(<a href="https://github.com/haskell/haskell-language-server/pull/4767">HLS PR <span>#4767</span></a>).</p></li>
<li><p>Hannes reviewed and assisted with <a href="https://github.com/haskell/haskell-language-server/pull/4856">HLS PR <span>#4856</span></a>
by <code>@vidit-od</code>. This PR makes HLS use the stored server-side diagnostics for
code actions, in order to make them more responsive. This fixes <a href="https://github.com/haskell/haskell-language-server/issues/4805">HLS issue <span>#4805</span></a>.</p></li>
<li><p>Hannes helped land long-running <a href="https://github.com/haskell/haskell-language-server/pull/4445">HLS PR <span>#4445</span></a>
by <code>@soulomoon</code>, which allows files to be loaded concurrently in batches in
order to improve responsiveness of HLS.</p></li>
<li><p>Zubin and Hannes worked together to update HLS to work with GHC 9.14 (<a href="https://github.com/haskell/haskell-language-server/pull/4780">HLS PR <span>#4780</span></a>).</p></li>
<li><p>Hannes worked on general maintenance of the HLS project:</p>
<ul>
<li>Prepared release 2.13.0.0 (<a href="https://github.com/haskell/haskell-language-server/pull/4785">HLS PR <span>#4785</span></a>)</li>
<li>Tackled various CI issues (<a href="https://github.com/haskell/haskell-language-server/pull/4863">HLS PR <span>#4863</span></a>,
<a href="https://github.com/haskell/haskell-language-server/pull/4812">HLS PR <span>#4812</span></a>,
<a href="https://github.com/haskell/haskell-language-server/pull/4811">HLS PR <span>#4811</span></a>)</li>
<li>Updated the advertised range of supported GHC versions (<a href="https://github.com/haskell/haskell-language-server/pull/4801">HLS PR <span>#4801</span></a>, <a href="https://github.com/haskell/haskell-language-server/pull/4799">HLS PR <span>#4799</span></a>)</li>
</ul></li>
<li><p>Hannes and Zubin implemented some fixes to Windows CI (<a href="https://github.com/haskell/haskell-language-server/pull/4800">HLS PR <span>#4800</span></a>, <a href="https://github.com/haskell/haskell-language-server/pull/4768">HLS PR <span>#4768</span></a>).</p></li>
<li><p>Hannes merged the <code>hls-module-name-plugin</code> into <code>hls-rename-plugin</code> in
<a href="https://github.com/haskell/haskell-language-server/pull/4847">HLS PR <span>#4847</span></a>.</p></li>
<li><p>Hannes improved the robustness of the <code>hls-call-hierarchy-plugin-tests</code> in
<a href="https://github.com/haskell/haskell-language-server/pull/4834">HLS PR <span>#4834</span></a>
by using <code>VirtualFileTree</code>.</p></li>
<li><p>Hannes also worked on <code>hie-bios</code>:</p>
<ul>
<li>Preparing release 0.18.0.0 (<a href="https://github.com/haskell/hie-bios/pull/496"><code>hie-bios</code> PR <span>#496</span></a>)</li>
<li>Updated the supported GHC versions (<a href="https://github.com/haskell/hie-bios/pull/495"><code>hie-bios</code> PR <span>#495</span></a>)</li>
<li>Adapted to GHC migrating some parts of its codebase to use <code>OsPath</code> (<a href="https://github.com/haskell/hie-bios/pull/493"><code>hie-bios</code> PR <span>#493</span></a>)</li>
</ul></li>
</ul>
<h2 id="haskell-debugger">Haskell Debugger</h2>
<p>Rodrigo continued work on the new <a href="https://github.com/well-typed/haskell-debugger">Haskell Debugger</a>.</p>
<ul>
<li><p>Matt and Rodrigo introduced a DSL for evaluation on the remote process, which
allows the debuggee to be queried from a custom instance, making it possible
to implement visualisations which rely on e.g. evaluatedness of a term
(<a href="https://github.com/well-typed/haskell-debugger/pull/139"><span>#139</span></a>).</p></li>
<li><p>Matt improved support for exceptions: break-on-exception breakpoints now provide
source locations (<a href="https://github.com/well-typed/haskell-debugger/pull/165"><span>#165</span></a>).</p></li>
<li><p>Rodrigo allowed call stacks to be inspected in the debugger (<a href="https://github.com/well-typed/haskell-debugger/pull/158"><span>#158</span></a>).</p></li>
<li><p>Hannes introduced support for stack decoding and viewing custom stack annotations
(<a href="https://github.com/well-typed/haskell-debugger/pull/172"><span>#172</span></a>).</p></li>
<li><p>Rodrigo made the Haskell Debugger use the external interpreter (<a href="https://github.com/well-typed/haskell-debugger/pull/170"><span>#170</span></a>),
which paves the way for multi-threaded debugging (see also <a href="https://github.com/well-typed/haskell-debugger/pull/140"><span>#140</span></a>).
This change also allowed Rodrigo to implement Windows support (<a href="https://github.com/well-typed/haskell-debugger/pull/184"><span>#184</span></a>)
with the help of Hannes.</p></li>
<li><p>Matt fixed a bug in the handling of data constructors with constraints
(<a href="https://github.com/well-typed/haskell-debugger/pull/175"><span>#175</span></a>).</p></li>
<li><p>Hannes improved caching in the CI (<a href="https://github.com/well-typed/haskell-debugger/pull/173"><span>#173</span></a>).</p></li>
</ul>
<h2 id="ghc-debug"><code>ghc-debug</code></h2>
<ul>
<li><p>Matt and Hannes fixed several issues with <code>AP_STACK</code> closures
(<a href="https://gitlab.haskell.org/ghc/ghc-debug/-/merge_requests/79"><span>!79</span></a>,
<a href="https://gitlab.haskell.org/ghc/ghc-debug/-/merge_requests/80"><span>!80</span></a>,
<a href="https://gitlab.haskell.org/ghc/ghc-debug/-/merge_requests/86"><span>!86</span></a>).</p></li>
<li><p>Hannes implemented asynchronous heap traversal in <code>ghc-debug-brick</code>,
making the interface more responsive
(<a href="https://gitlab.haskell.org/ghc/ghc-debug/-/merge_requests/78"><span>!78</span></a>).</p></li>
<li><p>Hannes added history navigation and search caching to the <code>ghc-debug-brick</code>
interface
(<a href="https://gitlab.haskell.org/ghc/ghc-debug/-/merge_requests/83"><span>!83</span></a>).</p></li>
<li><p>Hannes added a summary row to the string counting table view
(<a href="https://gitlab.haskell.org/ghc/ghc-debug/-/merge_requests/81"><span>!81</span></a>), and
fixed the search limit not being honoured during incremental searches
(<a href="https://gitlab.haskell.org/ghc/ghc-debug/-/merge_requests/76"><span>!76</span></a>).</p></li>
</ul></div>
    </content>
    <updated>2026-03-19T00:00:00Z</updated>
    <published>2026-03-19T00:00:00Z</published>
    <category term="well-typed"/>
    <category term="ghc"/>
    <category term="community"/>
    <category term="haskell-ecosystem-report"/>
    <author>
      <name>adam, andreask, hannes, magnus, matthew, mikolaj, rodrigo, sam, zubin</name>
    </author>
    <source>
      <id>https://well-typed.com/blog/</id>
      <logo>https://well-typed.com/img/wtnlogoplain.svg</logo>
      <link href="https://well-typed.com/blog/rss2.xml" rel="self" type="application/rss+xml"/>
      <link href="https://well-typed.com/blog/" rel="alternate" type="text/html"/>
      <subtitle>Because Well-Typed Programs Don't Go Wrong</subtitle>
      <title>Well-Typed Blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>tag:,2026:/math/egyptian-fractions-2</id>
    <link href="https://blog.plover.com/math/egyptian-fractions-2.html" rel="alternate" type="text/html"/>
    <title>Did Ahmes find the best expansions for 2/n?</title>
    <content type="xhtml" xml:lang="en"><div xmlns="http://www.w3.org/1999/xhtml"><p>A couple of years back <a href="https://blog.plover.com/math/egyptian-fractions.html">I was discussing the Rhind Mathematical Papyrus</a>
(RMP).  It includes a table expressing <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac%202n%24"/> as a sum
$$\frac1{a_1}+\frac1{a_2}+\dots+\frac1{a_k} $$ fractions with
numerator 1 (“unit fractions”).  I said:</p>

<blockquote>
  <p>Getting the table of good-quality representations of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac%202n%24"/> is not
  trivial, and requires searching, number theory, and some trial and
  error. It's not at all clear that <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac2%7b105%7d%3d%5cfrac1%7b90%7d%20%2b%0a%3e%20%5cfrac1%7b126%7d%24"/>.</p>
</blockquote>

<p>Today I wondered: <em>did</em> Ahmes (the author) have the best possible
expansions for all the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac2n%24"/> values, or were there some
improvements the Egyptians had missed?</p>

<p>It turns out, yes!  Or rather, maybe!</p>

<p>In
<a href="https://www.sciencedirect.com/science/article/pii/S0315086007000274?via%3Dihub">On the Egyptian method of decomposing <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%242%2fn%24"/> into unit fractions</a>
the author, Abdulrahman A. Abdulaziz, points out that for
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac2%7b95%7d%24"/> the Rhind Mathematical Papyrus gives the expansion
$$\frac2{95} = \frac1{60} + \frac1{380} + \frac1{570}$$</p>

<p>but <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac1%7b380%7d%20%2b%20%5cfrac1%7b570%7d%20%3d%20%5cfrac1%7b228%7d%24"/> so it could have been
written as $$\frac2{95} = \frac1{60}+\frac1{228}.$$</p>

<p>But wait, maybe that <em>wasn't</em> an error.  The Egyptians, like everyone,
often had to multiply by 10.  (In fact, the RMP itself, right after
its <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac%202n%24"/> table, has a shorter table of expansions of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac%0an%7b10%7d%24"/>.)  And <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac1%7b60%7d%20%2b%20%5cfrac1%7b380%7d%20%2b%20%5cfrac1%7b570%7d%24"/> is trivially
multiplied by 10, whereas <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac1%7b228%7d%24"/>  isn't.  There is some
indication that Ahmes preferred fractions with even denominators,
because they are easier to double, and the usual Egyptian method of
multiplication required repeated doubling.  But the Egyptians also
sometimes decupled while multiplying, and the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac1%7b60%7d%20%2b%0a%5cfrac1%7b380%7d%20%2b%20%5cfrac1%7b570%7d%24"/>  expansion would have made both of those
easy.</p>

<p>The methods by which Ahmes chose the expansions of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24%5cfrac%202n%24"/>, and
the criteria by which he preferred one to another, are still unknown;
he doesn't explain them.  So it's tough to say that any item was or
wasn't “best” from Ahmes' point of view.</p></div>
    </content>
    <updated>2026-03-17T13:28:00Z</updated>
    <published>2026-03-17T13:28:00Z</published>
    <category term="/math"/>
    <author>
      <name>Mark Dominus</name>
      <email>mjd@plover.com</email>
      <uri>http://www.plover.com/</uri>
    </author>
    <source>
      <id>tag:,2005:/</id>
      <icon>http://perl.plover.com/favicon.ico</icon>
      <link href="https://blog.plover.com/index.atom" rel="self" type="application/atom+xml"/>
      <link href="https://blog.plover.com" rel="alternate" type="text/html"/>
      <subtitle>The Universe of Discourse (Mark Dominus Blog)</subtitle>
      <title>The Universe of Discourse</title>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>https://haskellforall.com/2026/03/a-sufficiently-detailed-spec-is-code</id>
    <link href="https://haskellforall.com/2026/03/a-sufficiently-detailed-spec-is-code" rel="alternate" type="text/html"/>
    <title>A sufficiently detailed spec is code&gt;</title>
    <summary>Specifications do not address the limitations of agentic coding</summary>
    <updated>2026-03-17T00:00:00Z</updated>
    <published>2026-03-17T00:00:00Z</published>
    <source>
      <id>https://haskellforall.com</id>
      <author>
        <name>Gabriella Gonzalez</name>
      </author>
      <link href="https://haskellforall.com" rel="alternate" type="text/html"/>
      <link href="https://haskellforall.com/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>A blog about Haskell and functional programming.</subtitle>
      <title>Haskell for all</title>
      <updated>2026-03-17T14:41:11Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://medium.com/p/d4811e66120b</id>
    <link href="https://cdsmithus.medium.com/to-flip-or-not-to-flip-d4811e66120b?source=rss-18bd5acaea78------2" rel="alternate" type="text/html"/>
    <title>To Flip Or Not To Flip</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><em>A fair coin, an unfair offer, and the price of certainty.</em></p><p>I sat down to work out a classic probability problem numerically, and accidentally built a casino.</p><h3>The Problem of Points</h3><p>In 1654, a gambler named Antoine Gombaud posed a question to Blaise Pascal: two players are in a race to win a certain number of points. The game is interrupted. How should they divide the pot?</p><p>Pascal wrote to Fermat, and their correspondence became one of the founding documents of probability theory. The answer is elegant: if you need <em>a</em> more points and your opponent needs <em>b</em> more, you can compute the fair split with a simple recurrence. Let P(a, b) be your probability of winning:</p><ul><li>P(0, b) = 1 — you just won</li><li>P(a, 0) = 0 — your opponent just won</li><li>P(a, b) = ½ · P(a−1, b) + ½ · P(a, b−1)</li></ul><p>Every value in this table is a fraction with a power-of-2 denominator, and the numerators are just Pascal’s triangle. Beautiful math, clean solution, problem solved since the 17th century.</p><p>I built an interactive table to explore it. And then I thought: what if this were a game?</p><h3>The Game</h3><p>You and The House race to a target score. Each round, a fair coin is flipped — heads you score, tails The House scores. First to the target wins a pot of money.</p><p>But before each flip, judges look at the current game state, consult the probability table, and offer you cash to walk away. Accept, and you take the money. Decline, and the coin is flipped.</p><p>The question, every single round, is: <em>to flip or not to flip?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*UNqw2J4ROuCLqAwivdI_kQ.png"/></figure><p>You can play at <a href="https://willowdale.online/flip">willowdale.online/flip</a>.</p><h3>How the Judges Set Their Offers</h3><p>The judges know the exact fair value of your position — they have the same formula Pascal and Fermat computed. If you have a 37.5% chance of winning a $10,000 pot, your fair value is $3,750.</p><p>But they don’t offer fair value. They offer the nearest “clean” fraction of the pot that sits strictly <em>below</em> your true odds.</p><p>“Clean” means small denominators whose only prime factors are 2, 3, and 5 — fractions like 1/3, 3/8, 7/20, nothing with a denominator above 20. These produce dollar amounts that look like something a human came up with: $3,333, $3,750, $3,500. Not $3,077 or $3,846, which look like someone ran the numbers to the last penny.</p><p>So if your fair value is $3,770 (193/512 of the pot), the judges offer $3,750 (3/8). Barely below fair, and a beautifully round number. If your fair value is $1,875 (3/16), they offer $1,666 (1/6). An 89% offer — a real discount, but still a clean, human-sounding number.</p><p>This matters psychologically. Round numbers feel like ballpark estimates — casual, generous, not fully analyzed. Precise numbers feel calculated. When the judges offer $7,500, it sounds reasonable. If they offered $7,517, you’d immediately suspect they did the math and it’s in their favor. The irony is that $7,517 is a <em>better</em> deal for you — but I think you’d be less likely to take it. The round number keeps your guard down.</p><p>The algorithm is deterministic — same game state, same offer every time. Just math dressed up in a game show contract.</p><h3>Why People Sign</h3><p>Since the offers are always strictly below fair value, the play that maximizes your expected winnings is to <em>never accept a deal</em>. The coin is fair, the game has zero house edge, and every offer leaves money on the table. A player who always flips would win 50% of their games and, on average, neither gain nor lose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*bIC-rVKVQv62gnpV455J3A.png"/></figure><p>And yet.</p><p>When you’re ahead 4–3 in a race to 10, and the contract says $6,000, and you’ve already paid $5,000 to enter this game… you hesitate. That’s a guaranteed profit. The alternative is variance — maybe you win $10,000, but you are not that far ahead. Maybe your luck turns and you lose everything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*s4_Dy1tM1WLf2Dbo7DfVVA.png"/></figure><p>You <em>know</em> the offer is below fair. You can peek behind the curtain and see the exact numbers. The judges are shortchanging you by $128. But $128 feels like nothing when the alternative is watching your lead evaporate flip by flip.</p><p>So you sign. And $128 goes into the casino’s pocket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*sZAWF6XdrMOb3dWu0Pxxmg.png"/></figure><p>This is what makes the game unusual. In blackjack or roulette, the house edge is baked into the rules — you can’t avoid it no matter how disciplined you are. Here, the game has no edge at all. The coin is fair. The race is symmetric. The <em>only</em> source of profit is human nature. Every dollar the casino makes is expected value that a player voluntarily left on the table.</p><p>Play for a while and you start to notice specific situations where the offer gets harder to refuse.</p><p><strong>Managing risk.</strong> A guaranteed $7,500 is safer than a coin flip worth $7,734. In real life, you might need that money for rent. Variance has a real cost, and paying a premium for certainty can be entirely rational. There is a sophisticated argument for sometimes making decisions that reduce your expected value: bankroll management, survival probability and duration. Here the stakes are fictional, your bankroll buys nothing except more fair coin flips, and going broke is solved by refreshing your web browser, so that case is weaker — but it doesn’t feel weaker when your bankroll is shrinking and the judges are holding out real-looking money.</p><p><strong>Mis-anchoring.</strong> The rational comparison is always between the offer and the expected value of continuing to flip. But that’s rarely the comparison your brain actually makes. If you were staring at a $0 offer last round and now the judges are offering $500, you’re comparing to the $0 — not to the $625 fair value. If your bankroll started at $10,000 and you’re down to $7,000, and the judges offer $3,200, you’re comparing to $10,000 — because taking the deal would put you above where you started. In both cases, the reference point that feels relevant has nothing to do with the expected value of this game.</p><p><strong>Black and white thinking.</strong> When you’re behind in the race, the most likely single outcome is that you lose. If the judges offer $500 and your odds of winning are 6%, it feels like a choice between $500 and nothing. But expected value accounts for the 6% — the rare wins are big enough to compensate for all the losses across many games. You just don’t experience many games at once. You experience this one, where you’ll probably lose, and where the person who took $500 looks smart 94 times out of 100.</p><p><strong>Imaginary momentum.</strong> You lose three flips in a row and it feels like the coin has turned against you — time to take the deal before things get worse. Or you win three in a row and feel like you’re on a streak that shouldn’t be interrupted. The coin has no memory. Each flip is independent. But the human brain is a pattern-recognition machine, and it will find narratives in random sequences whether they’re there or not.</p><h3>The Optimal Judges</h3><p>The judges in this game are clever, but simple — they mechanically pick the nearest clean fraction below fair value, blind to everything except the current expected value.</p><p>But the <em>optimal</em> offer would be very different. The right objective isn’t just the EV gap (fair value minus offer). It’s:</p><p><strong>EV gap × P(acceptance | entire game trajectory)</strong></p><p>A huge gap with low acceptance is worthless — the player just turns it down. A tiny gap with high acceptance is pennies. The sweet spot is a moderate discount the player <em>almost</em> can’t refuse.</p><p>And that acceptance probability depends on far more than just the current score — it depends on everything described above: the bankroll trajectory, the recent streak, what the last offer was, how long the player has been sitting there.</p><p>A perfect judge would think about all of this, and decide exactly what it can get you — tired, frustrated, scared little you — to accept. The clean-fraction heuristic doesn’t. And yet it still works. I still sign those offers.</p><h3>The Lesson</h3><p>The game is a playable demonstration of why casinos stay in business, maybe even why people accept below-market returns for safety, and why insurance companies are profitable.</p><p>The math is always available — right there behind a curtain. If your goal is to maximize expected dollars, the answer is always to flip the coin. And yet, round after round, the judges offer deals, and I sign them.</p><p><em>Play the game at </em><a href="https://willowdale.online/flip"><em>willowdale.online/flip</em></a><em>. It’s free, the coin is fair, and you will almost certainly take a deal you know you shouldn’t.</em></p><img alt="" height="1" src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=d4811e66120b" width="1"/></div>
    </content>
    <updated>2026-03-16T12:00:03Z</updated>
    <published>2026-03-14T00:48:43Z</published>
    <category term="finance"/>
    <category term="game-theory"/>
    <category term="mathematics"/>
    <category term="probability"/>
    <author>
      <name>Chris Smith</name>
    </author>
    <source>
      <id>https://medium.com/@cdsmithus?source=rss-18bd5acaea78------2</id>
      <logo>https://cdn-images-1.medium.com/fit/c/150/150/1*rU9ddF_bkph8Qg3jip7vfw.jpeg</logo>
      <link href="https://medium.com/@cdsmithus?source=rss-18bd5acaea78------2" rel="alternate" type="text/html"/>
      <link href="https://medium.com/@cdsmithus/feed" rel="self" type="application/rss+xml"/>
      <link href="http://medium.superfeedr.com" rel="hub" type="text/html"/>
      <subtitle>Stories by Chris Smith on Medium</subtitle>
      <title>Stories by Chris Smith on Medium</title>
      <updated>2026-04-12T01:59:24Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://haskell.org/ghc/blog/20260313-ghc-9.12.4-rc1-released.html</id>
    <link href="http://haskell.org/ghc/blog/20260313-ghc-9.12.4-rc1-released.html" rel="alternate" type="text/html"/>
    <title>GHC 9.12.4-rc1 is now available</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><h1>GHC 9.12.4-rc1 is now available</h1>
<h4 class="text-muted">wz1000 - 2026-03-13</h4>

<p>The GHC developers are very pleased to announce the availability
of the release candidate for GHC 9.12.4. Binary distributions, source
distributions, and documentation are available at <a href="https://downloads.haskell.org/ghc/9.12.4-rc1">downloads.haskell.org</a> and
via <a href="https://www.haskell.org/ghcup/">GHCup</a>.</p>
<p>GHC 9.12.4 is a bug-fix release fixing many issues of a variety of
severities and scopes, including:</p>
<ul>
<li>Fixed a critical code generation regression where sub-word division produced
incorrect results (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26711">#26711</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26668">#26668</a>), similar to the bug fixed in 9.12.2</li>
<li>Numerous fixes for register allocation bugs, preventing data corruption
when spilling and reloading registers
(<a href="https://gitlab.haskell.org/ghc/ghc/issues/26411">#26411</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26526">#26526</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26537">#26537</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26542">#26542</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26550">#26550</a>)</li>
<li>Fixes for several compiler crashes, including issues with
CSE (<a href="https://gitlab.haskell.org/ghc/ghc/issues/25468">#25468</a>), SetLevels (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26681">#26681</a>),
implicit parameters (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26451">#26451</a>), and the type-class specialiser (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26682">#26682</a>)</li>
<li>Fixed cast worker/wrapper incorrectly firing on INLINE functions (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26903">#26903</a>)</li>
<li>Fixed LLVM backend miscompilation of bit manipulation operations
(<a href="https://gitlab.haskell.org/ghc/ghc/issues/20645">#20645</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26065">#26065</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26109">#26109</a>)</li>
<li>Fixed associated type family and data family instance changes not triggering
recompilation (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26183">#26183</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26705">#26705</a>)</li>
<li>Fixed negative type literals causing the compiler to hang (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26861">#26861</a>)</li>
<li>Improvements to determinism of compiler output (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26846">#26846</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26858">#26858</a>)</li>
<li>Fixes for eventlog shutdown deadlocks (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26573">#26573</a>)
and lost wakeups in the RTS (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26324">#26324</a>)</li>
<li>Fixed split sections support on Windows (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26696">#26696</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26494">#26494</a>) and the LLVM backend (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26770">#26770</a>)</li>
<li>Fixes for the bytecode compiler, PPC native code generator, and Wasm backend</li>
<li>The runtime linker now supports COMMON symbols (<a href="https://gitlab.haskell.org/ghc/ghc/issues/6107">#6107</a>)</li>
<li>Improved backtrace support: backtraces for <code>error</code> exceptions are now
evaluated at throw time</li>
<li><code>NamedDefaults</code> now correctly requires the class to be standard or have an
in-scope default declaration, and handles poly-kinded classes (<a href="https://gitlab.haskell.org/ghc/ghc/issues/25775">#25775</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/25778">#25778</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/25882">#25882</a>)</li>
<li>… and many more</li>
</ul>
<p>A full accounting of these fixes can be found in the</p>
<p><a href="https://gitlab.haskell.org/ghc/ghc/-/blob/ghc-9.12/docs/users_guide/9.12.4-notes.rst?ref_type=heads&amp;plain=1">release notes</a>. As always, GHC’s release status, including planned future
releases, can be found on the GHC Wiki <a href="https://gitlab.haskell.org/ghc/ghc/-/wikis/GHC-status">status</a>.</p>
<p>This release candidate will have a two-week testing period. If all goes well
the final release will be available the week of 26 March 2026.</p>
<p>GHC development is sponsored by:</p>
<ul>
<li><a href="https://juspay.com/">Juspay</a></li>
<li><a href="https://qbaylogic.com/">QBayLogic</a></li>
<li><a href="https://www.channable.com/">Channable</a></li>
<li><a href="https://haskell.foundation/">Haskell Foundation</a></li>
<li><a href="https://serokell.io/">Serokell</a></li>
<li><a href="https://well-typed.com/">Well-Typed</a></li>
<li><a href="https://www.tweag.io/">Tweag</a></li>
<li><a href="https://www.dotcom-monitor.com/">Dotcom-Monitor</a></li>
<li><a href="https://www.loadview-testing.com/">LoadView</a></li>
<li><a href="https://webhostingbuddy.com/">Web Hosting Buddy</a></li>
<li><a href="https://www.findmyelectric.com/">Find My Electric</a></li>
<li><a href="https://www.sc.com">Standard Chartered</a></li>
<li><a href="https://upcloud.com">UpCloud</a></li>
<li><a href="https://mercury.com">Mercury</a></li>
</ul>
<p>We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.</p>
<p>As always, do give this release a try and open a <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/new">ticket</a> if you see
anything amiss.</p></div>
    </summary>
    <updated>2026-03-13T00:00:00Z</updated>
    <published>2026-03-13T00:00:00Z</published>
    <author>
      <name>ghc-devs</name>
    </author>
    <source>
      <id>http://haskell.org/ghc</id>
      <link href="http://haskell.org/ghc" rel="alternate" type="text/html"/>
      <link href="http://haskell.org/ghc/rss.xml" rel="self" type="application/rss+xml"/>
      <title>GHC Developer blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>tag:blogger.com,1999:blog-8897180777295067814.post-6620848665537137843</id>
    <link href="http://bokesan.blogspot.com/feeds/6620848665537137843/comments/default" rel="replies" title="Post Comments" type="application/atom+xml"/>
    <link href="http://bokesan.blogspot.com/2026/03/functional-valhalla.html#comment-form" rel="replies" title="1 Comments" type="text/html"/>
    <link href="http://www.blogger.com/feeds/8897180777295067814/posts/default/6620848665537137843" rel="edit" type="application/atom+xml"/>
    <link href="http://www.blogger.com/feeds/8897180777295067814/posts/default/6620848665537137843" rel="self" type="application/atom+xml"/>
    <link href="http://bokesan.blogspot.com/2026/03/functional-valhalla.html" rel="alternate" title="Functional Valhalla?" type="text/html"/>
    <title>Functional Valhalla?</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><span style="background-color: #fdfdfd; color: #1a1a1a;">Pointer-rich data layouts lead to suboptimal performance on modern hardware. For an excellent introduction to this, see the article</span><span style="background-color: #fdfdfd; color: #1a1a1a;"> </span><a href="https://openjdk.org/projects/valhalla/design-notes/state-of-valhalla/01-background" style="color: #1a1a1a;">The Road to Valhalla</a><span style="background-color: #fdfdfd; color: #1a1a1a;">. While it is specifically about Java, many parts of the article also apply to other languages. To summarize some of the key points of the article:</span></p><ul style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1em; padding-left: 1.7em;"><li>In 1990, a main memory fetch was about as expensive as an arithmetic operation. Now, it might be a hundred times slower.</li><li>A pointer-rich data layout involving indirections between data at different locations is not ideal for today’s hardware.</li><li>A language should make flat (cache-efficient) and dense (memory-efficient) memory layouts possible without compromising abstraction or type safety.</li></ul><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Consider a vector of records (or tuples, structures, product types - I’ll stay with “record” in this article). A pointer-rich layout has each record allocated separately in the heap, with a vector containing pointers to the records. For example, given a “Point” record of two numbers:</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;"><img alt="pikchr diagram" height="336" width="491"/></p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">The flat and dense layout has the records directly in the array:</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;"><img alt="pikchr diagram" height="88" width="647"/></p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">(Note that there is another flat layout, namely, using one vector per field of the record. This is better suited to instruction-level parallelism or specialized hardware (e.g., GPUs), especially when the record fields have different sizes. But it is less suited for general-purpose computing, as reading a single vector element requires one memory access per field, whereas the “vector of records” layout above requires only one access per record. Such a layout can be easily implemented in any language that has arrays of native types, whether in the language itself or in a library (e.g., OCaml’s Owl library). Thus, in this article, I will only consider the “array of records” layout above.)</p><h2 id="functional-language-considerations" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Functional language considerations</h2><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Things should be much easier in functional languages than in Java: we have purity, referential transparency, and everything is a value. So it should be simple enough to store these values in memory in their native representation. But there are reasons that that is often not the case in practice:</p><ul style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1em; padding-left: 1.7em;"><li>Lazyness: a value can be a computation that produces a value only when needed.</li><li>Layout polymorphism: unless we replicate the code for every type (as, for example, Rust does), we need to be able to store every possible value in the same kind of slot.</li><li>Dynamically typed languages require type information at runtime.</li><li>Functional languages often have automatic memory management, which may require runtime type information.</li><li>Many of our languages are not purely functional, but contain impure features.</li><li>Pure languages often lack traditional vectors or arrays, since making them perform well in immutable code is not easy.</li><li>Historical reasons: Graph reduction was a common implementation technique for lazy languages, and graphs involve pointers.</li><li>Implementation restrictions: not being mainstream, fewer resources are devoted to implementation and optimization.</li></ul><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Many implementations can not even lay out native types flat in records, so a Point record of IEEE 754 double-precision numbers may actually look like this in memory:</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;"><img alt="pikchr diagram" height="232" width="330"/></p><h2 id="the-very-short-list" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">The (very short) List</h2><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">So, given a record type, which functional languages allow a collection of values of that type to have a flat, linear memory layout? The number of programming languages that claim to be “functional” is huge, so the ones listed here are just a selection based on my preferences - mainly languages that allow that layout, and some I have some experience with and can speculate on how easy or hard it would be to add that as a library or extension.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Since the Point record can be misleading in its simplicity when it comes to the question of whether the functionality could be implemented as a library, I’ll point out that there are records where the layout is a bit more interesting:</p><ul style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1em; padding-left: 1.7em;"><li>Records containing different types with different storage sizes, for example, one 64-bit float and one 32-bit integer. On most architectures, this will require 4 bytes of padding between elements.</li><li>Records containing native values along with something that has to be represented as a pointer, for example, a reference-type or a lazy value. In a flat layout, this means that every nth element will be a pointer, requiring special support from the memory management system, either by providing layout information or by using a conservative GC that treats everything as a potential pointer.</li></ul><h3 id="pure-languages" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Pure languages:</h3><h4 id="clean" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Clean</h4><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Yes: Clean has unboxed arrays of records in the base language.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Caveat: it does not have integer types of specific sizes and only one floating-point type, making it harder to reduce memory usage by using the smallest type just large enough to support the required value range. It seems possible to implement such types in a library (the <a href="http://bokesan.blogspot.de/feeds/posts/default/-/gitlab.com/mtask/" style="color: #1a1a1a;">mTask system</a> does that).</p><h4 id="futhark" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Futhark</h4><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">No. Futhark does not intend to be a general-purpose language, so this is not surprising.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">I mention it here because it <em>does have arrays of records,</em> but, since it targets GPUs and related hardware, it uses the “record of arrays” layout mentioned above.</p><h4 id="haskell" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Haskell</h4><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Yes. Not in the base language, but there is library support via <a href="https://hackage-content.haskell.org/package/vector-0.13.2.0/docs/Data-Vector-Unboxed.html" style="color: #1a1a1a;">Data.Vector.Unboxed</a>. Types that implement the <code>Unbox</code> type class can be used in these vectors. Many basic types and tuples have an <code>Unbox</code> instance. However, when you care about efficiency, you probably do not want to use tuples but rather a data type with strict fields, i.e., not:</p><div class="sourceCode" id="cb1" style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px; overflow: auto;"><pre class="sourceCode haskell" style="background-color: transparent; margin: 0px; overflow: visible;"><code class="sourceCode haskell"><span id="cb1-1" style="color: inherit; display: inline-block; line-height: 1.25; text-decoration: inherit;"><a href="http://bokesan.blogspot.de/feeds/posts/default/-/Haskell#cb1-1" style="color: #1a1a1a;" tabindex="-1"/><span class="kw" style="color: #007020; font-weight: bold;">type</span> <span class="dt" style="color: #902000;">Point</span> <span class="ot" style="color: #007020;">=</span> (<span class="dt" style="color: #902000;">Double</span>, <span class="dt" style="color: #902000;">Double</span>)</span></code></pre></div><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">but:</p><div class="sourceCode" id="cb2" style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px; overflow: auto;"><pre class="sourceCode haskell" style="background-color: transparent; margin: 0px; overflow: visible;"><code class="sourceCode haskell"><span id="cb2-1" style="color: inherit; display: inline-block; line-height: 1.25; text-decoration: inherit;"><a href="http://bokesan.blogspot.de/feeds/posts/default/-/Haskell#cb2-1" style="color: #1a1a1a;" tabindex="-1"/><span class="kw" style="color: #007020; font-weight: bold;">data</span> <span class="dt" style="color: #902000;">Point</span> <span class="ot" style="color: #007020;">=</span> <span class="dt" style="color: #902000;">Point</span> <span class="op" style="color: #666666;">!</span><span class="dt" style="color: #902000;">Double</span> <span class="op" style="color: #666666;">!</span><span class="dt" style="color: #902000;">Double</span></span></code></pre></div><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Writing an <code>Unbox</code> instance for such a type is not trivial. The <a href="https://hackage.haskell.org/package/vector-th-unbox-0.2.2/docs/Data-Vector-Unboxed-Deriving.html" style="color: #1a1a1a;">vector-th-unbox library</a> makes it easier, but requires Template Haskell. Unboxed vectors are implemented by marshalling the values to byte arrays, so records with pointer fields are not supported.</p><h3 id="impure-languages" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Impure Languages</h3><h4 id="f" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">F#</h4><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Yes, even records with pointer fields. Records have structural equality, and you can use structs or the <code>[&lt;Struct&gt;]</code> attribute to get a flat layout.</p><hr style="background-color: #fdfdfd; border-bottom: none; border-left: none; border-right: none; border-top-color: rgb(26, 26, 26); border-top-style: solid; height: 1px; margin: 1em 0px;"/><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">And that’s all I could find. Unless I follow <a href="https://en.wikipedia.org/wiki/List_of_programming_languages_by_type#Functional_languages" style="color: #1a1a1a;">Wikipedia's list of functional programming languages</a>, which contains languages such as C++, C#, Rust, or Swift, that allow the flat layout, but don’t really fit my idea of a functional language. But SML, OCaml, Erlang (Elixir, Gleam), Scala? Not that I could see (but please correct me if I’m wrong).</p><h3 id="rolling-your-own" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Rolling your own</h3><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Since there is a library implementation for Haskell, maybe that’s a possibility for other languages?</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">You should be able to implement flat layouts in any language that supports byte vectors. More interesting is how well such a library fits into the language, and whether a user of the library has to write code or annotations for user-defined record types, or whether the library can handle part or all of that automagically.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">I’ll only mention my beloved Lisp/Scheme here. Lisp’s uniform syntax and macro system are a bonus here, but the lack of static typing makes things harder.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">In Scheme, R6RS (and R7RS with the help of some SRFIs) has byte-vectors and marshalling to/from them in the standard library. But Scheme does not have type annotations, so you either need to offer a macro to define records with typed fields or to define how to marshal the fields of a regular (sealed) record. Since you can shadow standard procedures in a library, you can write code that looks like regular Scheme code, but, perhaps surprisingly, loses identity when storing/retrieving values from records:</p><div class="sourceCode" id="cb3" style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px; overflow: auto;"><pre class="sourceCode scheme" style="background-color: transparent; margin: 0px; overflow: visible;"><code class="sourceCode scheme"><span id="cb3-1" style="color: inherit; display: inline-block; line-height: 1.25; text-decoration: inherit;"><a href="http://bokesan.blogspot.de/feeds/posts/default/-/Haskell#cb3-1" style="color: #1a1a1a;" tabindex="-1"/>(<span class="kw" style="color: #007020; font-weight: bold;">let</span> ((vec (make-typed-vector 'point <span class="dv" style="color: #40a070;">1000</span>))</span>
<span id="cb3-2" style="color: inherit; display: inline-block; line-height: 1.25; text-decoration: inherit;"><a href="http://bokesan.blogspot.de/feeds/posts/default/-/Haskell#cb3-2" style="color: #1a1a1a;" tabindex="-1"/>      (pt (make-point x y)))</span>
<span id="cb3-3" style="color: inherit; display: inline-block; line-height: 1.25; text-decoration: inherit;"><a href="http://bokesan.blogspot.de/feeds/posts/default/-/Haskell#cb3-3" style="color: #1a1a1a;" tabindex="-1"/>  (<span class="kw" style="color: #007020; font-weight: bold;">vector-set!</span> vec <span class="dv" style="color: #40a070;">0</span> pt)</span>
<span id="cb3-4" style="color: inherit; display: inline-block; line-height: 1.25; text-decoration: inherit;"><a href="http://bokesan.blogspot.de/feeds/posts/default/-/Haskell#cb3-4" style="color: #1a1a1a;" tabindex="-1"/>  (<span class="kw" style="color: #007020; font-weight: bold;">eq?</span> (<span class="kw" style="color: #007020; font-weight: bold;">vector-ref</span> vec <span class="dv" style="color: #40a070;">0</span>) pt))</span>
<span id="cb3-5" style="color: inherit; display: inline-block; line-height: 1.25; text-decoration: inherit;"><a href="http://bokesan.blogspot.de/feeds/posts/default/-/Haskell#cb3-5" style="color: #1a1a1a;" tabindex="-1"/> ⇒ <span class="dv" style="color: #40a070;">#f</span></span></code></pre></div><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">(But then, you probably shouldn’t be using <code>eq?</code> when doing functional programming in Scheme).</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">The same approach is possible in Common Lisp. In contrast to Scheme, it does have optional type annotations, and, together with a helper library for accessing the innards of floats and either the meta-object protocol to get type information or (probably better) a macro to define typed records, an implementation should be reasonably straightforward. Making it play nice with inheritance and the dynamic nature of Common Lisp (e.g., adding slots to classes or even changing an object's class at runtime) would be a much harder undertaking.</p><h2 id="conclusion" style="background-color: #fdfdfd; color: #1a1a1a; margin-top: 1.4em;">Conclusion</h2><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Of the functional languages I looked at, only F# fully supports flat and dense memory layouts. Among the pure languages, Haskell and Clean come close.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">The question is how important this really is. There’s a good argument to be made for turning to more specialized languages like Futhark if you mainly care about performance. On the other hand, having a uniform codebase in one language also has advantages.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Then, the performance story has changed, too. While the points Project Valhalla raises remain true in principle, processor designers are aware of this as well. They are doing their best to hide memory latency with techniques such as out-of-order execution or humongous caches. Thus, on a modern CPU, the effects of a pointer-rich layout are often only observable with large working set sizes.</p><p style="background-color: #fdfdfd; color: #1a1a1a; margin: 1em 0px;">Still, given the plethora of imperative language that can get you to Valhalla, support for this in the functional landscape seems lacking. In the future, I hope to see more languages or libraries that will make this possible.</p></div>
    </content>
    <updated>2026-03-12T11:17:02Z</updated>
    <published>2026-03-12T11:17:00Z</published>
    <category scheme="http://www.blogger.com/atom/ns#" term="Haskell"/>
    <category scheme="http://www.blogger.com/atom/ns#" term="Lisp"/>
    <category scheme="http://www.blogger.com/atom/ns#" term="Performance"/>
    <category scheme="http://www.blogger.com/atom/ns#" term="programming"/>
    <author>
      <name>bokesan</name>
      <email>noreply@blogger.com</email>
      <uri>http://www.blogger.com/profile/09457977231270485255</uri>
    </author>
    <source>
      <id>tag:blogger.com,1999:blog-8897180777295067814</id>
      <category term="programming"/>
      <category term="Photography"/>
      <category term="Haskell"/>
      <category term="Nikon"/>
      <category term="Scala"/>
      <category term="Benchmark"/>
      <category term="Hasselblad"/>
      <category term="Nikkor"/>
      <category term="Performance"/>
      <category term="Sony"/>
      <category term="ICFPC"/>
      <category term="Java"/>
      <category term="Lenses"/>
      <category term="Lisp"/>
      <category term="Sony Alpha"/>
      <category term="Focus shift"/>
      <category term="Lens test"/>
      <category term="LoCA"/>
      <category term="Adapter"/>
      <category term="Android"/>
      <category term="CMOS"/>
      <category term="Catalyst"/>
      <category term="Closures"/>
      <category term="CoffeeScript"/>
      <category term="D600"/>
      <category term="Extension"/>
      <category term="Hannover"/>
      <category term="JavaScript"/>
      <category term="Kaveri"/>
      <category term="Leaf Shutter"/>
      <category term="Lens"/>
      <category term="Linux"/>
      <category term="Loongson"/>
      <category term="MTF"/>
      <category term="Macro"/>
      <category term="MapExplorer"/>
      <category term="Megapixels"/>
      <category term="Mirrorless"/>
      <category term="Olympus"/>
      <category term="Oracle"/>
      <category term="PSF"/>
      <category term="Samsung"/>
      <category term="Sensor"/>
      <category term="Steamroller"/>
      <category term="Tripod"/>
      <category term="Tubes"/>
      <category term="Windows"/>
      <category term="bokeh"/>
      <category term="bugs"/>
      <category term="chromatic aberration"/>
      <category term="cooling"/>
      <category term="field curvature"/>
      <category term="mainboard"/>
      <category term="mini-itx"/>
      <category term="multithreaded"/>
      <category term="testing"/>
      <author>
        <name>bokesan</name>
        <email>noreply@blogger.com</email>
        <uri>http://www.blogger.com/profile/09457977231270485255</uri>
      </author>
      <link href="http://bokesan.blogspot.com/feeds/posts/default" rel="http://schemas.google.com/g/2005#feed" type="application/atom+xml"/>
      <link href="http://www.blogger.com/feeds/8897180777295067814/posts/default/-/Haskell" rel="self" type="application/atom+xml"/>
      <link href="http://bokesan.blogspot.com/search/label/Haskell" rel="alternate" type="text/html"/>
      <link href="http://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <subtitle>Thoughts about programming, photography, and sometimes life</subtitle>
      <title>about:random</title>
      <updated>2026-03-17T14:13:03Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://www.parsonsmatt.org/2026/03/10/teaching_claude_to_be_lazy.html</id>
    <link href="https://www.parsonsmatt.org/2026/03/10/teaching_claude_to_be_lazy.html" rel="alternate" type="text/html"/>
    <title>Teaching Claude to Be Lazy</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>I’ve been watching AI development for a long time.
I found LessWrong around 2012-2013, and managed to get myself worked up about the oncoming singularity.
I managed to chill out about it, but interest and excitement for AI remained.
The initial Deep Dream image generation, Alpha Go, etc, were all so exciting.
And then GPT-2 came out.</p>

<p>Over the last five years, people have been making wild claims about the utility of <em>present</em> AI.
Not “the AI that you’ll have <em>soon</em>,” but the current generation stuff.
And the results, frankly, had been garbage.
A sea of garbage coating the internet.
I’d try using the tools, and when checking them against my own expertise or knowledge, they always fell short.</p>

<p>I heard the noise on Twitter after Opus 4.5 was released in November of 2025.
Seemed like a step change- people were much more impressed with it than prior versions.
In December, I decided to give it a try.
Opus 4.5, with significant guidance, properly diagnosed and fixed some Template Haskell code generation issues.
It knew how to <code class="language-plaintext highlighter-rouge">-ddump-splices</code>, it knew how to read those splices and diagnose the issue.
Given a small, highly mechanical problem, plenty of examples, and a ton of tests, it took about 6 hours to do what I felt would have taken me 3 or 4 hours.</p>

<p>This is pretty incredible, because my productivity has always been limited by two things:</p>

<ol>
  <li>Effort. Literally whacking my keyboard and staring at computer and waiting on a compile/test loop to tell me what to do next.</li>
  <li>Attention. Where I’m focusing my effort. My editor? Slack? Meetings? A bike ride? Cello? Which OSS project?</li>
</ol>

<p>Now, with Opus 4.5, I can set a robot going and do <em>something else</em> with my effort.
While Claude Code was spinning on the Template Haskell code, I was doing another project in a different repository.
Sure, Claude took 6 hours instead of my 3, but I was able to fill those 6 hours with <em>effort</em> and attention placed elsewhere - not a full 6, as Claude required supervision and input, but call it 5.
This is a positive investment, and my personal “break even” moment.</p>

<h1 id="using-claude-code-effectively">Using Claude Code Effectively</h1>

<p>In mid February, I got access to an API token and unlimited usage.
I’ve been trying to figure out how to leverage this tool to improve my productivity, and the results have been pretty strongly positive.</p>

<p>The brief tl;dr:</p>

<blockquote>
  <p>It’s the same shit that makes humans good at software development</p>
</blockquote>

<h2 id="haskell-is-awesome-for-llms">Haskell is Awesome for LLMs</h2>

<p>This was true with Opus 4.5 and is much more true with Opus 4.6.
Prior versions of LLM coding agents produced utter garbage with Haskell, most likely due to the relatively low quantity of examples.
It <em>seems</em> like the AI labs have figured out how to do higher quality training with less data, and the relatively high average quality of Haskell code helps the LLMs generate relatively high quality Haskell.</p>

<p>Haskell’s type safety, purity, and library design opportunities make it a fantastic choice for LLM generated code.
The human developer can easily specify a solution and let Claude fill in a <em>surprising</em> amount of the boring details.</p>

<p>Haskell’s terse nature benefits LLMs - you can simply fit more tokens into the context window when the tokens are more semantically dense.</p>

<p>Funny enough, all of Haskell’s benefits “for LLMs” are also benefits of Haskell for humans.
I do earnestly believe that if all devs knew Haskell, we would consider switching to other languages only very rarely.
And Claude <em>knows Haskell</em>.</p>

<h2 id="software-engineering-matters">Software Engineering Matters</h2>

<p>Claude Code works <em>really well</em> with tightly scoped issues, lots of tests and examples, and good safety guardrails.
I asked it to make <code class="language-plaintext highlighter-rouge">cabal</code> faster, and taught it how to run <code class="language-plaintext highlighter-rouge">cabal</code> with debug logs, timings, and then to build a profiled version of it.
Then it looped for a bit, collected timing information on our codebase, and figured out the critical path and hot spot - the solver.
Then it <a href="https://github.com/haskell/cabal/pull/11566">made several fixes to optimize the solver</a>.
These changes resulted in a 30% improvement in solver times, which shaved 2 seconds off every <code class="language-plaintext highlighter-rouge">cabal repl</code> invocation- a pretty nice benefit, since that happens virtually anytime you want to do anything in our codebase.</p>

<p>But this only worked because the <code class="language-plaintext highlighter-rouge">cabal</code> library had timing logs, and I gave it a quick feedback loop and target.
I’ve had Claude Code totally fall over when trying to do bigger or more undirected work.</p>

<p>Fortunately, Claude can do this pretty well.
I’ve had Claude do some exploratory research (generally pretty highly supervised), then generate some plans for improvement (then edited and clarified), and it can then do a good job of writing up a ticket- certainly better than almost all human written tickets I’ve seen.</p>

<h2 id="build-workflows-iteratively">Build Workflows Iteratively</h2>

<p>LLMs can do anything.
But they are expensive, slow, and non-deterministic (and often incorrect).
So get the LLM to help with <em>replacing themselves</em> - build a tool or skill to do the thing faster and deterministically.</p>

<p>My Claude sessions generally progress from “highly supervised, exploratory work” to “mostly unsupervised, automated work.”
Early sessions in a project often involve having Claude build tools - CLI scripts, libraries, interfaces - that it can use in later work to make the job easier.
A surprisingly effective prompt here is “What tools would help you do this job better next time?”
At the end of a session, I’ll also have Claude review and update its skill documentation with everything I told it to do differently.</p>

<p>So each work session with Claude produces:</p>

<ol>
  <li>An artifact (the work itself)</li>
  <li>Often, updates to the skill to improve efficiency on further work</li>
  <li>Sometimes, a tool to deterministically do some chunk of the work.</li>
</ol>

<p>This process ends up reducing the highly non-deterministic LLM tool with a much more deterministic tool.</p>

<h2 id="mock-reviews-and-refactoring">Mock Reviews and Refactoring</h2>

<p>You can ask Claude to review code, and that works OK.
But Claude works <em>much better</em> if you ask it to assume someone else’s perspective.
I’ve asked it to mimic myself and it did Alright.
I asked it to mimic Edward Kmett, Alexis King, and Michael Snoyman, and it did Alright - it noticed different things with each perspective and suggested improvements in line with those perspectives.</p>

<p>I’ve generally found that the initial output is of poor to middling quality.
But you can get decently far with “now make it more legible/faster/more correct” or “apply ‘Parse, Don’t Validate’ here” etc.
After several rounds of refactoring, it makes stuff that I’m reasonably happy with putting my name on.</p>

<h1 id="what-doesnt-work-well">What Doesn’t Work Well</h1>

<p>Claude isn’t a replacement for human engineering (“yet” i guess).
It lacks qualities like taste, judgement, and vision, that are generally required in subjective work like software and product design.
So when I let Claude run totally loose on something, it <em>produces</em>, but it produces poor quality code and poorly thought out features.</p>

<p>I haven’t had great luck with getting Claude to iterate on this itself.
When given the very large picture, it sort of flounders.
It can do some analysis and subdivision, but the divisions are often somewhat unnatural and don’t feel right to me.</p>

<h2 id="defined-by-our-vice">Defined by our Vice</h2>

<p>If the above complaint is about Claude’s lack of virtue, let me also complain about Claude’s lack of vice.
Claude is infinitely patient and willing to work very hard.
However, “infinitely patient” means that Claude has no problem at all waiting an hour for a build to finish.
You have to teach it to use faster tools and feedback loops.</p>

<p>Likewise, “hardworking” is a virtue when you’re paying a human by the month and trusting in their laziness to be efficient, but when you’re paying per unit of thought, “more work” means “more cost” and often not “more output.”
You have to tell Claude to <em>stop doing stuff</em> or to do stuff more efficiently.</p>

<p>Fortunately, Claude is relatively teachable - but Claude very often will start a skill and then do a lot of “research and understanding” before running the one-shot script to generate the compile-errors to track down and fix.</p>

<p>Humans are impatient and lazy, so we build fast and efficient systems.
Without pain to guide us, we make little progress in <em>reducing</em> that pain.</p>

<h1 id="am-i-still-a-skeptic">Am I still a skeptic?</h1>

<p>I’ve been using AI to write 95% of my code for the last month.
And yet, I still feel like I’m more on the skeptic side of things.
AI is clearly a <em>useful</em> tool - my own productivity has doubled or more while maintaining my personal quality bar.
But it’s not a do-it-all miracle - yet?</p>

<p>AI-first companies are experiencing massive reliability issues.
Vibe coding projects start, enjoy some success, and then go down in flames.</p>

<p>Humans are clearly still necessary at key points in the software lifecycle.
The bottlenecks have shifted, though, and the easiest parts of my job have been mostly automated.
What’s coming next?</p>

<p>I’m excited to wait and find out.</p></div>
    </summary>
    <updated>2026-03-10T00:00:00Z</updated>
    <published>2026-03-10T00:00:00Z</published>
    <source>
      <id>https://www.parsonsmatt.org</id>
      <author>
        <name>Matt Parsons</name>
      </author>
      <link href="https://www.parsonsmatt.org" rel="alternate" type="text/html"/>
      <link href="https://www.parsonsmatt.org/feed.xml" rel="self" type="application/rss+xml"/>
      <title>parsonsmatt.org</title>
      <updated>2026-03-10T19:05:50Z</updated>
    </source>
  </entry>

  <entry>
    <id>tag:,2026:/tech/gpt/documentation-wins-2</id>
    <link href="https://blog.plover.com/tech/gpt/documentation-wins-2.html" rel="alternate" type="text/html"/>
    <title>Programmers will document for Claude, but not for each other</title>
    <content type="xhtml" xml:lang="en"><div xmlns="http://www.w3.org/1999/xhtml"><p><a href="https://blog.plover.com/tech/gpt/documentation-wins.html">A couple of days ago I recounted a common complaint</a>:</p>

<blockquote>
  <p>I keep seeing programmers say how angry it makes them that people
  are willing to write detailed <code>CLAUDE.md</code> and <code>PROJECT.md</code> files for
  Claude to use, but they weren't willing to write them for their
  coworkers.</p>
</blockquote>

<p>For larger projects, I've taken to having Claude maintain a handoff
document that I can have the next Claude read, saying what we planned
to do, what has been done, and other pertinent information.  Then when
I shut down one Claude I can have the next one read the file to get up
to speed.  Then I have the Claude <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24n%2b1%24"/> update it for Claude
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&amp;cht=tx&amp;chl=%24n%2b2%24"/>.</p>

<p>After seeing the common complaint enough times I had a happy
inspiration.  I'd been throwing away Claude's handoff documents at the
end of each project.  Why do that?  It's no trouble to copy the file
into the repository and commit it.  Someone in the future, wondering
what was going on, might luckily find the right document with <code>git
grep</code> and learn something useful.</p>

<p>I'm a little slow so it took me until this week to think of a better
version of this: at the end of the project I now ask Claude to write
up from scratch a detailed but high-level explanation of what problem
we were solving and what changes we made, and I commit <em>that</em>.  Not
just running notes, but a structured overview of the whole thing.</p>

<p>I review these overviews carefully and make edits as necessary before
I check them in. It's my signature on the commit, and my bank account
receiving the paycheck, so nothing goes into the repository that I
haven't read carefully and understood, same as if Claude were a human
programmer under my supervision.</p>

<p>But Claude's explanations haven't required much editing.  Claude's
most recent project summary was around as good as what I could have
written myself, maybe a little worse and maybe a little better.  But
it took ten seconds to write instead of an hour, and it didn't take
anything like an hour to review.</p>

<p>The serious thing I had to fix the last time around was that Claude
had used a previous, related report as a model, and the previous
report had had a paragraph I had added at the end that said:</p>

<blockquote>
  <p># Approved-by</p>
  
  <p>Claude abstracted these notes from our discussions of the issue. Mark
  Dominus has read, reviewed, edited, and approved these notes.</p>
</blockquote>

<p>Claude's new document had an identical section at the end.  Oops!
Fortunately, by the time I saw it, it was true, so I didn't have to
delete it.  I had Claude add a sentence to <code>CLAUDE.md</code> to tell it not
to do this again.</p>

<p>My advice for the day:</p>

<ol>
<li><p>If you have Claude write down notes, check them into the repo when
you're done.  It probably can't hurt and it might help.</p></li>
<li><p>Have Claude write a project summary, and then check it into the repo.</p></li>
</ol>

<p>Maybe this is obvious?  But it wasn't obvious to me.  I'm still
getting used to this new world.</p></div>
    </content>
    <updated>2026-03-09T08:04:00Z</updated>
    <published>2026-03-09T07:40:00Z</published>
    <category term="/tech/gpt"/>
    <author>
      <name>Mark Dominus</name>
      <email>mjd@plover.com</email>
      <uri>http://www.plover.com/</uri>
    </author>
    <source>
      <id>tag:,2005:/</id>
      <icon>http://perl.plover.com/favicon.ico</icon>
      <link href="https://blog.plover.com/index.atom" rel="self" type="application/atom+xml"/>
      <link href="https://blog.plover.com" rel="alternate" type="text/html"/>
      <subtitle>The Universe of Discourse (Mark Dominus Blog)</subtitle>
      <title>The Universe of Discourse</title>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>Buzzsprout-18809011</id>
    <link href="https://www.buzzsprout.com/1817535/episodes/18809011-78-jamie-willis.mp3" length="31284259" rel="enclosure" type="audio/mpeg"/>
    <title>78: Jamie Willis</title>
    <summary>In this episode, we focus on a particular part of Haskell: teaching it. To help us, we are joined by Jamie Willis who is a Teaching Fellow at Imperial College London. The episode explores the benefits of live coding, and why Haskell is the best language for teaching programming.</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><b>In this episode, we focus on a particular part of Haskell: teaching it. To help us, we are joined by Jamie Willis who is a Teaching Fellow at Imperial College London. The episode explores the benefits of live coding, and why Haskell is the best language for teaching programming.</b></p><p><br/></p></div>
    </content>
    <updated>2026-03-08T11:00:00Z</updated>
    <published>2026-03-08T11:00:00Z</published>
    <author>
      <name>Haskell Podcast</name>
    </author>
    <source>
      <id>https://haskell.foundation/podcast/</id>
      <logo>https://storage.buzzsprout.com/tnk1ztokn5vmeiufqmr4kkp37mw2?.jpg</logo>
      <category scheme="http://www.itunes.com/" term="Technology"/>
      <author>
        <name>Haskell Podcast</name>
      </author>
      <link href="https://rss.buzzsprout.com/1817535.rss" rel="self" type="application/rss+xml"/>
      <link href="https://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <link href="https://haskell.foundation/podcast/" rel="alternate" type="text/html"/>
      <rights>Â© 2026 The Haskell Interlude</rights>
      <subtitle>This is the Haskell Interlude, where the five co-hosts (Wouter Swierstra, Andres LÃ¶h, Alejandro Serrano, Niki Vazou, and Joachim Breitner) chat with Haskell guests!</subtitle>
      <title>The Haskell Interlude</title>
      <updated>2026-04-10T16:12:47Z</updated>
    </source>
  </entry>

  <entry>
    <id>tag:,2026:/movie/john_waters</id>
    <link href="https://blog.plover.com/movie/john_waters.html" rel="alternate" type="text/html"/>
    <title>How are John Waters movies like James Bond movies?</title>
    <content type="xhtml" xml:lang="en"><div xmlns="http://www.w3.org/1999/xhtml"><p>A number of years ago I wondered how many movies I had seen. The only
way I could think of finding out was just to make a list.  This I did
as best I could.  (It turned out to be around 700.)</p>

<p>I found, though, that I could not include all the James Bond movies I
had seen, because I couldn't tell them apart from the descriptions.
I'd read a plot summary for a James Bond movie, and ask myself “Did I
see that?  I don't know, it sounds like every other James Bond movie.”</p>

<p>Today I discovered that John Waters movies are like that also.  I was
trying to remember if I had seen <em>A Dirty Shame</em>:</p>

<blockquote>
  <p>The people of Harford Road are firmly divided into two camps: the
  neuters, the puritanical residents who despise anything even
  remotely carnal; and the perverts, a group of sex addicts whose
  unique fetishes have all been brought to the fore by accidental
  concussions. Repressed Sylvia Stickles finds herself firmly
  entrenched in the former camp.</p>
</blockquote>

<p>You'd think that would be something I would remember decisively, or
not.  But I'm really not sure. All I can do is shrug and say “I don't
know, it sounds like a John Waters movie I have seen, but maybe it
wasn't that one.”</p>

<p>Looking into it further I discovered that I also wasn't sure if I had
seen <em>Multiple Maniacs</em>.  In it, Divine's character is raped by a
giant lobster.  On the one hand, that seems like the sort of thing I
would remember.  And I think maybe I do?  But again I'm not sure I'm
not just imagining what it would be like!</p></div>
    </content>
    <updated>2026-03-08T09:50:00Z</updated>
    <published>2026-03-08T09:50:00Z</published>
    <category term="/movie"/>
    <author>
      <name>Mark Dominus</name>
      <email>mjd@plover.com</email>
      <uri>http://www.plover.com/</uri>
    </author>
    <source>
      <id>tag:,2005:/</id>
      <icon>http://perl.plover.com/favicon.ico</icon>
      <link href="https://blog.plover.com/index.atom" rel="self" type="application/atom+xml"/>
      <link href="https://blog.plover.com" rel="alternate" type="text/html"/>
      <subtitle>The Universe of Discourse (Mark Dominus Blog)</subtitle>
      <title>The Universe of Discourse</title>
    </source>
  </entry>

  <entry>
    <id>tag:,2026:/tech/gpt/documentation-wins</id>
    <link href="https://blog.plover.com/tech/gpt/documentation-wins.html" rel="alternate" type="text/html"/>
    <title>Documentation is a message in a bottle</title>
    <content type="xhtml" xml:lang="en"><div xmlns="http://www.w3.org/1999/xhtml"><p>Our company is
going to a convention later this month, and they will have a booth with
big TV screens showing statistics that update in real time.  My job is
to write the backend server that delivers the statistics.</p>

<p>I read over the documents that the product people had written up about
what was wanted, asked questions, got answers, and then turned the
original two-line ticket into a three-page ticket that said what
should be done and how.  I intended to do the ticket myself, but it's
good practice to write all this stuff down, for many reasons:</p>

<ul>
<li><p>Writing things down forces me to think them through carefully and
realize what doesn't make sense or what I still don't understand.</p></li>
<li><p>I forget things easily and this will keep the plan where I can find it.</p></li>
<li><p>I might get sick, and if someone else has to pick up the project
this might help them understand what I was doing.</p></li>
<li><p>If my boss gets worried that all I do is post on 4chan all day, this
is tangible work product that proves I did something else that might
have enhanced shareholder value.</p></li>
<li><p>If I'm tempted to spend the day posting on 4chan, and then to later
<em>claim</em> I spent the time planning the project, I might fool my boss.
But without that tangible work product, I won't be able to fool
<em>myself</em>, and that's more important.</p></li>
<li><p>Conversely if I later think back and ask “What was I doing the week
of March 2?” I might be tempted to imagine that all I did was post
on 4chan.  But the three pages of ticket description will prove to
me that I am not just a lazy slacker.  <a href="https://blog.plover.com/misc/evaluation.html">This is a real problem for
me</a>.</p></li>
<li><p>In principle, a future person going back to extend the work might
find this helpful documentation of what was done and why.  Does this
ever really happen?  I don't know, but it might.</p></li>
<li><p>I like writing because writing is fun.</p></li>
</ul>

<p>A few days after I wrote the ticket, something unexpected happened.
It transpired that person who was to build the front-end consumer of
my statistics would not be a professional programmer.  It would be the
company's Head of Product, a very smart woman named Amanda. The actual
code would be written by Claude, under her supervision.</p>

<p>I have never done anything like this before, and I would not have
wanted to try it on a short deadline, but there is some slack in the
schedule and it seemed a worthwhile and exciting experiment.</p>

<p>Amanda shared some screencaps of her chats with Claude about the
project, and I suggested:</p>

<blockquote>
  <p>When you get a chance, please ask Claude to write out a Markdown
  file memorializing all this.  Tell it that you're going to give it
  to the backend programmer for discussion, so more detail is better.
  When it's ready, send it over.</p>
</blockquote>

<p>Claude immediately produced a nine-page, 14-part memo and a half-page
overview.  I spent a couple of hours reviewing it and marking it up.</p>

<p>It became immediately clear that Claude and I had very similar ideas
about how the project should go and how the front and back ends would
hook up.  So similar that I asked Angela:</p>

<blockquote>
  <p>It looks like maybe you started it off by feeding it my ticket
  description.  Is that right? </p>
</blockquote>

<p>She said yes, she had.  She had also fed it the original product
documents I had read.</p>

<p>I was delighted.  I had had many reasons for writing detailed ticket
descriptions before, but the most plausible ones were aimed back at
myself.</p>

<p>The external consumers of the documentation all seemed somewhat
unlikely.  The person who would extend the project in the future
probably didn't exist, and if they did they probably wouldn't have
thought to look at my notes.  Same for the hypothetical person who
would take over when I got sick.  My boss probably isn't checking up
on me by looking at my ticketing history.  Still, I like to document
these things for my own benefit, and also just in case.</p>

<p>But now, because I had written the project plan, it was available for
consumption when an unexpected consumer turned up!  Claude and I were
able to rapidly converge on the design of the system, because Amanda
had found my notes and cleverly handed them to Claude.  Suddenly one
of those unlikely-seeming external reasons materialized!</p>

<p>On Mastodon I keep seeing programmers say how angry it makes
them that people are willing to write detailed <code>CLAUDE.md</code> and
<code>PROJECT.md</code> files for Claude to use, but they weren't willing to
write them for their coworkers.  (They complain about this as if this
is somehow the fault of the AI, rather than of the people who failed
in the past to write documentation for their coworkers.)</p>

<p>The obvious answer to the question of why people are willing to write
documentation for Claude but not for their coworkers is that the
author can count on Claude to <em>read</em> the documentation, whereas it's a
rare coworker who will look at it attentively.</p>

<p>Rik Signes points out there's a less obvious but more likely answer:
your coworkers will remember things if you just tell them, but Claude
forgets everything every time.  If you want Claude to remember
something, you have to write it down.  So people using Claude do write
things down, because otherwise they have to say them over and over.</p>

<p>And there's a happy converse to the complaint that most programmers
don't bother to write documentation.  It means that people like me,
professionals who <em>have</em> always written meticulous documentation, are
now reaping new benefits from that always valuable practice.</p>

<p>Not everything is going to get worse.  Some things will get better.</p>

<h3>Addendum 20260208</h3>

<p><a href="https://blog.plover.com/tech/gpt/documentation-wins-2.html">A corollary</a>: You don't
have to write the rocumentation yourself.  You can have Claude write a
detailed summary based on your ongoing chats about the work, and then
you can edit it and check it in.</p>

<p>If you're good at editing, anyway.  I wonder if part of the reason
Claude is working so well for me is that I'm really good at editing
and at code review?</p></div>
    </content>
    <updated>2026-03-05T16:07:00Z</updated>
    <published>2026-03-05T16:07:00Z</published>
    <category term="/tech/gpt"/>
    <author>
      <name>Mark Dominus</name>
      <email>mjd@plover.com</email>
      <uri>http://www.plover.com/</uri>
    </author>
    <source>
      <id>tag:,2005:/</id>
      <icon>http://perl.plover.com/favicon.ico</icon>
      <link href="https://blog.plover.com/index.atom" rel="self" type="application/atom+xml"/>
      <link href="https://blog.plover.com" rel="alternate" type="text/html"/>
      <subtitle>The Universe of Discourse (Mark Dominus Blog)</subtitle>
      <title>The Universe of Discourse</title>
    </source>
  </entry>

  <entry>
    <id>tag:,2026:/lang/etym/Bo-Diddley</id>
    <link href="https://blog.plover.com/lang/etym/Bo-Diddley.html" rel="alternate" type="text/html"/>
    <title>Bo Diddley</title>
    <content type="xhtml" xml:lang="en"><div xmlns="http://www.w3.org/1999/xhtml"><p>Bo Diddley's cover of "Sixteen Tons" sounds very much like one of my
favorites, "Can't Judge A Book By Its Cover".  It's interesting to
compare.</p>

<p>Thinking on that it suddenly occured to me that his name might have
been a play on “diddley bow”, which is a sort of homemade one-stringed
zither.  The player uses a bottle as a bridge for the string, and
changes the pitch by sliding the bottle up and down.  When you hear
about blues artists whose first guitars were homemade, this is often
what was meant: it wasn't a six-string guitar, it was a diddley bow.</p>

<p>But it's not clear that Bo Diddley <em>did</em> play his name on the diddley
bow.  "Diddly" also means something insignificant or of little value,
and might have been a disparaging nickname he received in his
youth. (It also appears in the phrase "diddly squat").  Maybe that's
also the source of the name of the diddley bow.</p></div>
    </content>
    <updated>2026-03-03T16:39:00Z</updated>
    <published>2026-03-03T16:39:00Z</published>
    <category term="/lang/etym"/>
    <author>
      <name>Mark Dominus</name>
      <email>mjd@plover.com</email>
      <uri>http://www.plover.com/</uri>
    </author>
    <source>
      <id>tag:,2005:/</id>
      <icon>http://perl.plover.com/favicon.ico</icon>
      <link href="https://blog.plover.com/index.atom" rel="self" type="application/atom+xml"/>
      <link href="https://blog.plover.com" rel="alternate" type="text/html"/>
      <subtitle>The Universe of Discourse (Mark Dominus Blog)</subtitle>
      <title>The Universe of Discourse</title>
    </source>
  </entry>

  <entry>
    <id>https://doisinkidney.com/posts/2026-03-03-monus-heaps.html</id>
    <link href="https://doisinkidney.com/posts/2026-03-03-monus-heaps.html" rel="alternate" type="text/html"/>
    <title>Monuses and Heaps</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><div class="info">
    Posted on March  3, 2026
</div>
<div class="info">
    
</div>
<div class="info">
    
        Tags: <a href="https://doisinkidney.com/tags/Haskell.html" rel="tag" title="All pages tagged 'Haskell'.">Haskell</a>
    
</div>

<p>This post is about a simple algebraic structure that I have found
useful for algorithms that involve searching or sorting based on some
ordered weight. I used it a bit in a pair of papers on graph search
<span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-kidney_algebras_2021">2021</a>; <a href="https://doisinkidney.com/rss.xml#ref-kidney_formalising_2025">2025</a>)</span>, and more recently I used it to
implement a version of the <code class="sourceCode haskell"><span class="dt">Phases</span></code> type
<span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-easterly_functions_2019">Easterly
2019</a>)</span> that supported arbitrary keys, inspired by some work by
BlÃ¶ndal <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-blondal_generalized_2025">2025a</a>; <a href="https://doisinkidney.com/rss.xml#ref-blondal_phases_2025">2025b</a>)</span>
and <span class="citation">Visscher
(<a href="https://doisinkidney.com/rss.xml#ref-visscher_phases_2025">2025</a>)</span>.</p>
<p>The algebraic structure in question is a <em>monus</em>, which is a
kind of monoid that supports a partial subtraction operation (that
subtraction operation, denoted by the symbol âˆ¸, is itself often called a
â€œmonusâ€�). However, before giving the full definition of the structure,
let me first try to motivate its use. The context here is heap-based
algorithms. For the purposes of this post, a heap is a tree that obeys
the â€œheap propertyâ€�; i.e.Â every node in the tree has some â€œweightâ€�
attached to it, and every parent node has a weight less than or equal to
the weight of each of its children. So, for a tree like the
following:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="https://doisinkidney.com/rss.xml#cb1-1" tabindex="-1"/>   â”Œd</span>
<span id="cb1-2"><a href="https://doisinkidney.com/rss.xml#cb1-2" tabindex="-1"/> â”Œbâ”¤</span>
<span id="cb1-3"><a href="https://doisinkidney.com/rss.xml#cb1-3" tabindex="-1"/>aâ”¤ â””e</span>
<span id="cb1-4"><a href="https://doisinkidney.com/rss.xml#cb1-4" tabindex="-1"/> â””c</span></code></pre></div>
<p>The heap property is satisfied when
<math display="inline">&lt;semantics&gt;<mrow><mi>a</mi><mo>â‰¤</mo><mi>b</mi></mrow>&lt;annotation encoding="application/x-tex"&gt;a \leq b&lt;/annotation&gt;&lt;/semantics&gt;</math>,
<math display="inline">&lt;semantics&gt;<mrow><mi>a</mi><mo>â‰¤</mo><mi>c</mi></mrow>&lt;annotation encoding="application/x-tex"&gt;a \leq c&lt;/annotation&gt;&lt;/semantics&gt;</math>,
<math display="inline">&lt;semantics&gt;<mrow><mi>b</mi><mo>â‰¤</mo><mi>d</mi></mrow>&lt;annotation encoding="application/x-tex"&gt;b \leq d&lt;/annotation&gt;&lt;/semantics&gt;</math>,
and
<math display="inline">&lt;semantics&gt;<mrow><mi>b</mi><mo>â‰¤</mo><mi>e</mi></mrow>&lt;annotation encoding="application/x-tex"&gt;b \leq e&lt;/annotation&gt;&lt;/semantics&gt;</math>.</p>
<p>Usually, we also want our heap structure to have an operation like
<code class="sourceCode haskell"><span class="ot">popMin ::</span> <span class="dt">Heap</span> k v <span class="ot">-&gt;</span> <span class="dt">Maybe</span> (v, <span class="dt">Heap</span> k v)</code>
that returns the least-weight value in the heap paired with the rest of
the heap. If this operation is efficient, we can use the heap to
efficiently implement sorting algorithms, graph search, etc. In fact,
let me give the whole basic interface for a heap here:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb2-1"><a href="https://doisinkidney.com/rss.xml#cb2-1" tabindex="-1"/><span class="ot">popMin ::</span> <span class="dt">Heap</span> k v <span class="ot">-&gt;</span> <span class="dt">Maybe</span> (v, <span class="dt">Heap</span> k v)</span>
<span id="cb2-2"><a href="https://doisinkidney.com/rss.xml#cb2-2" tabindex="-1"/><span class="ot">insert ::</span> k <span class="ot">-&gt;</span> v <span class="ot">-&gt;</span> <span class="dt">Heap</span> k v <span class="ot">-&gt;</span> <span class="dt">Heap</span> k v</span>
<span id="cb2-3"><a href="https://doisinkidney.com/rss.xml#cb2-3" tabindex="-1"/><span class="ot">empty  ::</span> <span class="dt">Heap</span> k v</span></code></pre></div>
<p>Using these functions itâ€™s not hard to see how we can implement a
sorting algorithm:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb3-1"><a href="https://doisinkidney.com/rss.xml#cb3-1" tabindex="-1"/><span class="ot">sortOn ::</span> (a <span class="ot">-&gt;</span> k) <span class="ot">-&gt;</span> [a] <span class="ot">-&gt;</span> [a]</span>
<span id="cb3-2"><a href="https://doisinkidney.com/rss.xml#cb3-2" tabindex="-1"/>sortOn k <span class="ot">=</span> unfoldr popMin <span class="op">.</span> <span class="fu">foldr</span> (\x <span class="ot">-&gt;</span> insert (k x) x) empty</span></code></pre></div>
<p>The monus becomes relevant when the weight involved is some kind of
<em>monoid</em>. This is quite a common situation: if we were using the
heap for graph search (least-cost paths or something), we would expect
the weight to correspond to path costs, and we would expect that we can
add the costs of paths in a kind of monoidal way. Furthermore, we would
probably expect the monoidal operations to relate to the order in some
coherent way. A monus <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-amer_equationally_1984">Amer
1984</a>)</span> is an ordered monoid where the order itself can be
defined <em>in terms</em> of the monoidal operations<a class="footnote-ref" href="https://doisinkidney.com/rss.xml#fn1" id="fnref1"><sup>1</sup></a>:</p>
<p><math display="block">&lt;semantics&gt;<mrow><mi>x</mi><mo>â‰¤</mo><mi>y</mi><mo>â‡”</mo><mo>âˆƒ</mo><mi>z</mi><mi>.</mi><mspace width="0.278em"/><mi>y</mi><mo>=</mo><mi>x</mi><mo>â€¢</mo><mi>z</mi></mrow>&lt;annotation encoding="application/x-tex"&gt; x \leq y \iff \exists z. \; y = x \bullet z &lt;/annotation&gt;&lt;/semantics&gt;</math></p>
<p>I read this definition as saying
â€œ<math display="inline">&lt;semantics&gt;<mi>x</mi>&lt;annotation encoding="application/x-tex"&gt;x&lt;/annotation&gt;&lt;/semantics&gt;</math>
is less than
<math display="inline">&lt;semantics&gt;<mi>y</mi>&lt;annotation encoding="application/x-tex"&gt;y&lt;/annotation&gt;&lt;/semantics&gt;</math>
iff there is some
<math display="inline">&lt;semantics&gt;<mi>z</mi>&lt;annotation encoding="application/x-tex"&gt;z&lt;/annotation&gt;&lt;/semantics&gt;</math>
that <em>fits between</em>
<math display="inline">&lt;semantics&gt;<mi>x</mi>&lt;annotation encoding="application/x-tex"&gt;x&lt;/annotation&gt;&lt;/semantics&gt;</math>
and
<math display="inline">&lt;semantics&gt;<mi>y</mi>&lt;annotation encoding="application/x-tex"&gt;y&lt;/annotation&gt;&lt;/semantics&gt;</math>â€�.
In other words, the <em>gap</em> between
<math display="inline">&lt;semantics&gt;<mi>x</mi>&lt;annotation encoding="application/x-tex"&gt;x&lt;/annotation&gt;&lt;/semantics&gt;</math>
and
<math display="inline">&lt;semantics&gt;<mi>y</mi>&lt;annotation encoding="application/x-tex"&gt;y&lt;/annotation&gt;&lt;/semantics&gt;</math>
has to exist, and it is equal to
<math display="inline">&lt;semantics&gt;<mi>z</mi>&lt;annotation encoding="application/x-tex"&gt;z&lt;/annotation&gt;&lt;/semantics&gt;</math>.</p>
<p>Notice that this order definition wonâ€™t work for groups like
<math display="inline">&lt;semantics&gt;<mrow><mo form="prefix" stretchy="false">(</mo><mi>â„¤</mi><mo>,</mo><mi>+</mi><mo>,</mo><mn>0</mn><mo form="postfix" stretchy="false">)</mo></mrow>&lt;annotation encoding="application/x-tex"&gt;(\mathbb{Z},+,0)&lt;/annotation&gt;&lt;/semantics&gt;</math>.
For a group, we can <em>always</em> find some
<math display="inline">&lt;semantics&gt;<mi>z</mi>&lt;annotation encoding="application/x-tex"&gt;z&lt;/annotation&gt;&lt;/semantics&gt;</math>
that will fit the existential (specifically,
<math display="inline">&lt;semantics&gt;<mrow><mi>z</mi><mo>=</mo><mo form="prefix" stretchy="false">(</mo><mi>âˆ’</mi><mi>x</mi><mo form="postfix" stretchy="false">)</mo><mo>â€¢</mo><mi>y</mi></mrow>&lt;annotation encoding="application/x-tex"&gt;z = (- x) \bullet y&lt;/annotation&gt;&lt;/semantics&gt;</math>).
Monuses, then, tend to be positive monoids: in fact, many monuses are
the positive cones of some group
(<math display="inline">&lt;semantics&gt;<mrow><mo form="prefix" stretchy="false">(</mo><mi>â„•</mi><mo>,</mo><mi>+</mi><mo>,</mo><mn>0</mn><mo form="postfix" stretchy="false">)</mo></mrow>&lt;annotation encoding="application/x-tex"&gt;(\mathbb{N},+,0)&lt;/annotation&gt;&lt;/semantics&gt;</math>
is the positive cone of
<math display="inline">&lt;semantics&gt;<mrow><mo form="prefix" stretchy="false">(</mo><mi>â„¤</mi><mo>,</mo><mi>+</mi><mo>,</mo><mn>0</mn><mo form="postfix" stretchy="false">)</mo></mrow>&lt;annotation encoding="application/x-tex"&gt;(\mathbb{Z},+,0)&lt;/annotation&gt;&lt;/semantics&gt;</math>).</p>
<p>We can derive a lot of useful properties from this basic structure.
For example, if the order above is total, then we can derive the binary
subtraction operator mentioned above:</p>
<p><math display="block">&lt;semantics&gt;<mrow><mi>x</mi><mo>âˆ¸</mo><mi>y</mi><mo>=</mo><mrow><mo form="prefix" stretchy="true">{</mo><mtable><mtr><mtd columnalign="left" style="text-align: left;"><mi>z</mi><mo>,</mo></mtd><mtd columnalign="left" style="text-align: left;"><mrow><mtext mathvariant="normal">if </mtext><mspace width="0.333em"/></mrow><mi>y</mi><mo>â‰¤</mo><mi>x</mi><mrow><mspace width="0.333em"/><mtext mathvariant="normal"> and </mtext><mspace width="0.333em"/></mrow><mi>x</mi><mo>=</mo><mi>y</mi><mo>â€¢</mo><mi>z</mi></mtd></mtr><mtr><mtd columnalign="left" style="text-align: left;"><mn>0</mn><mo>,</mo></mtd><mtd columnalign="left" style="text-align: left;"><mtext mathvariant="normal">otherwise.</mtext></mtd></mtr></mtable></mrow></mrow>&lt;annotation encoding="application/x-tex"&gt; x âˆ¸ y =
\begin{cases}
z, &amp; \text{if } y \leq x \text{ and } x = y \bullet z  \\
0, &amp; \text{otherwise.}
\end{cases}
&lt;/annotation&gt;&lt;/semantics&gt;</math></p>
<p>If we require the underlying monoid to be commutative, and we further
require the derived order to be total and antisymmetric, we get the
particular flavour of monus I worked with in a pair of papers on graph
search <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-kidney_algebras_2021">2021</a>; <a href="https://doisinkidney.com/rss.xml#ref-kidney_formalising_2025">2025</a>)</span>. In this post I will actually be
working with a weakened form of the algebra that I will define
shortly.</p>
<p>Getting back to our heap from above, with this new order defined, we
can see that the heap property actually tells us something about the
makeup of the weights in the tree. Instead of every child just having a
weight equal to some arbitrary quantity, the heap property tells us that
each child weight has to be made up of the combination of its parentâ€™s
weight and some difference.</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb4-1"><a href="https://doisinkidney.com/rss.xml#cb4-1" tabindex="-1"/>   â”Œd              â”Œbâ€¢(dâˆ¸b)</span>
<span id="cb4-2"><a href="https://doisinkidney.com/rss.xml#cb4-2" tabindex="-1"/> â”Œbâ”¤       â”Œaâ€¢(bâˆ¸a)â”¤</span>
<span id="cb4-3"><a href="https://doisinkidney.com/rss.xml#cb4-3" tabindex="-1"/>aâ”¤ â””e  <span class="ot">=</span>  aâ”¤       â””bâ€¢(eâˆ¸b)</span>
<span id="cb4-4"><a href="https://doisinkidney.com/rss.xml#cb4-4" tabindex="-1"/> â””c        â””aâ€¢(câˆ¸a)</span></code></pre></div>
<p>This observation gives us an opportunity for a different
representation: instead of storing the full weight at each node, we
could instead just store the difference.</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb5-1"><a href="https://doisinkidney.com/rss.xml#cb5-1" tabindex="-1"/>     â”Œdâˆ¸b</span>
<span id="cb5-2"><a href="https://doisinkidney.com/rss.xml#cb5-2" tabindex="-1"/> â”Œbâˆ¸aâ”¤</span>
<span id="cb5-3"><a href="https://doisinkidney.com/rss.xml#cb5-3" tabindex="-1"/>aâ”¤   â””eâˆ¸b</span>
<span id="cb5-4"><a href="https://doisinkidney.com/rss.xml#cb5-4" tabindex="-1"/> â””câˆ¸a</span></code></pre></div>
<p>Just in terms of data structure design, I prefer this version: if we
wanted to write down a type of heaps using the previous design, we would
first define the type of trees, and then separately write a predicate
corresponding to the heap property. With this design, it is impossible
to write down a tree that <em>doesnâ€™t</em> satisfy the heap
property.</p>
<p>More practically, though, using this algebraic structure when working
with heaps enables some optimisations that might be difficult to
implement otherwise. The strength of this representation is that it
allows for efficient relative and global computation: now, if we wanted
to add some quantity to every weight in the tree, we can do it just by
adding the weight to the root node.</p>
<h1 id="monuses-in-haskell">Monuses in Haskell</h1>
<p>To see some examples of how to use this pattern, letâ€™s first write a
class for Haskell monuses:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb6-1"><a href="https://doisinkidney.com/rss.xml#cb6-1" tabindex="-1"/><span class="kw">class</span> (<span class="dt">Semigroup</span> a, <span class="dt">Ord</span> a) <span class="ot">=&gt;</span> <span class="dt">Monus</span> a <span class="kw">where</span></span>
<span id="cb6-2"><a href="https://doisinkidney.com/rss.xml#cb6-2" tabindex="-1"/>  (âˆ¸)<span class="ot"> ::</span> a <span class="ot">-&gt;</span> a <span class="ot">-&gt;</span> a</span></code></pre></div>
<p>Youâ€™ll notice that weâ€™re requiring semigroup here, not monoid. Thatâ€™s
because one of the nice uses of this pattern actually works with a
weakening of the usual monus algebra; this weakening only requires
semigroup, and the following two laws.</p>
<p><math display="block">&lt;semantics&gt;<mrow><mi>x</mi><mo>â‰¤</mo><mi>y</mi><mo>âŸ¹</mo><mi>x</mi><mo>â€¢</mo><mo form="prefix" stretchy="false">(</mo><mi>y</mi><mo>âˆ¸</mo><mi>x</mi><mo form="postfix" stretchy="false">)</mo><mo>=</mo><mi>y</mi><mspace width="1.0em"/><mspace width="1.0em"/><mspace width="1.0em"/><mspace width="1.0em"/><mspace width="1.0em"/><mi>x</mi><mo>â‰¤</mo><mi>y</mi><mo>âŸ¹</mo><mi>z</mi><mo>â€¢</mo><mi>x</mi><mo>â‰¤</mo><mi>z</mi><mo>â€¢</mo><mi>y</mi></mrow>&lt;annotation encoding="application/x-tex"&gt; x \leq y \implies x \bullet (y âˆ¸ x) = y
\quad \quad \quad \quad \quad
x \leq y \implies z \bullet x \leq z \bullet y &lt;/annotation&gt;&lt;/semantics&gt;</math></p>
<p>A straightforward monus instance is the following:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb7-1"><a href="https://doisinkidney.com/rss.xml#cb7-1" tabindex="-1"/><span class="kw">instance</span> (<span class="dt">Num</span> a, <span class="dt">Ord</span> a) <span class="ot">=&gt;</span> <span class="dt">Monus</span> (<span class="dt">Sum</span> a) <span class="kw">where</span></span>
<span id="cb7-2"><a href="https://doisinkidney.com/rss.xml#cb7-2" tabindex="-1"/>  (âˆ¸) <span class="ot">=</span> (<span class="op">-</span>)</span></code></pre></div>
<h1 id="pairing-heaps-in-haskell">Pairing Heaps in Haskell</h1>
<p>Next, letâ€™s look at a simple heap implementation. I will always go
for pairing heaps <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-fredman_pairing_1986">Fredman et al. 1986</a>)</span> in Haskell; they
are extremely simple to implement, and (as long as you donâ€™t have
significant persistence requirements) their performance seems to be the
best of the available pointer-based heaps <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-larkin_backtobasics_2013">Larkin, Sen,
and Tarjan 2013</a>)</span>. Here is the type definition:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb8-1"><a href="https://doisinkidney.com/rss.xml#cb8-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Root</span> k v <span class="ot">=</span> <span class="dt">Root</span> <span class="op">!</span>k v [<span class="dt">Root</span> k v]</span>
<span id="cb8-2"><a href="https://doisinkidney.com/rss.xml#cb8-2" tabindex="-1"/><span class="kw">type</span> <span class="dt">Heap</span> k v <span class="ot">=</span> <span class="dt">Maybe</span> (<span class="dt">Root</span> k v)</span></code></pre></div>
<p>A <code class="sourceCode haskell"><span class="dt">Root</span></code> is a
non-empty pairing heap; the <code class="sourceCode haskell"><span class="dt">Heap</span></code> type
represents possibly-empty heaps. The key function to implement is the
merging of two heaps; we can accomplish this as an implementation of the
semigroup <code class="sourceCode haskell"><span class="op">&lt;&gt;</span></code>.</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb9-1"><a href="https://doisinkidney.com/rss.xml#cb9-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> <span class="dt">Semigroup</span> (<span class="dt">Root</span> k v) <span class="kw">where</span></span>
<span id="cb9-2"><a href="https://doisinkidney.com/rss.xml#cb9-2" tabindex="-1"/>  <span class="dt">Root</span> xk xv xs <span class="op">&lt;&gt;</span> <span class="dt">Root</span> yk yv ys</span>
<span id="cb9-3"><a href="https://doisinkidney.com/rss.xml#cb9-3" tabindex="-1"/>    <span class="op">|</span> xk <span class="op">&lt;=</span> yk  <span class="ot">=</span> <span class="dt">Root</span> xk xv (<span class="dt">Root</span> (yk âˆ¸ xk) yv ys <span class="op">:</span> xs)</span>
<span id="cb9-4"><a href="https://doisinkidney.com/rss.xml#cb9-4" tabindex="-1"/>    <span class="op">|</span> <span class="fu">otherwise</span> <span class="ot">=</span> <span class="dt">Root</span> yk yv (<span class="dt">Root</span> (xk âˆ¸ yk) xv xs <span class="op">:</span> ys)</span></code></pre></div>
<p>The only difference between this and a normal pairing heap merge is
the use of <code class="sourceCode haskell">âˆ¸</code> in the key of the
child node (<code class="sourceCode haskell">yk âˆ¸ xk</code> and <code class="sourceCode haskell">xk âˆ¸ yk</code>). This difference ensures that
each child only holds the difference of the weight between itself and
its parent.</p>
<p>Itâ€™s worth working out why the weakened monus laws above are all we
need in order to maintain the heap property on this structure.</p>
<p>The rest of the methods are implemented the same as their
implementations on a normal pairing heap. First, we have the pairing
merge of a list of heaps, here given as an implementation of the
semigroup method <code class="sourceCode haskell">sconcat</code>:</p>
<div class="sourceCode" id="cb10"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb10-1"><a href="https://doisinkidney.com/rss.xml#cb10-1" tabindex="-1"/>  sconcat (x1 <span class="op">:|</span> []) <span class="ot">=</span> x1</span>
<span id="cb10-2"><a href="https://doisinkidney.com/rss.xml#cb10-2" tabindex="-1"/>  sconcat (x1 <span class="op">:|</span> [x2]) <span class="ot">=</span> x1 <span class="op">&lt;&gt;</span> x2</span>
<span id="cb10-3"><a href="https://doisinkidney.com/rss.xml#cb10-3" tabindex="-1"/>  sconcat (x1 <span class="op">:|</span> x2 <span class="op">:</span> x3 <span class="op">:</span> xs) <span class="ot">=</span> (x1 <span class="op">&lt;&gt;</span> x2) <span class="op">&lt;&gt;</span> sconcat (x3 <span class="op">:|</span> xs)</span>
<span id="cb10-4"><a href="https://doisinkidney.com/rss.xml#cb10-4" tabindex="-1"/></span>
<span id="cb10-5"><a href="https://doisinkidney.com/rss.xml#cb10-5" tabindex="-1"/><span class="ot">merges ::</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> [<span class="dt">Root</span> k v] <span class="ot">-&gt;</span> <span class="dt">Heap</span> k v</span>
<span id="cb10-6"><a href="https://doisinkidney.com/rss.xml#cb10-6" tabindex="-1"/>merges <span class="ot">=</span> <span class="fu">fmap</span> sconcat <span class="op">.</span> nonEmpty</span></code></pre></div>
<p>The pattern of this two-level merge is what gives the pairing heap
its excellent performance.</p>
<p>The <code class="sourceCode haskell"><span class="dt">Heap</span></code> type
derives its monoid instance from the monoid instance on <code class="sourceCode haskell"><span class="dt">Maybe</span></code> (<code class="sourceCode haskell"><span class="kw">instance</span> <span class="dt">Semigroup</span> a <span class="ot">=&gt;</span> <span class="dt">Monoid</span> (<span class="dt">Maybe</span> a)</code>),
so we can implement <code class="sourceCode haskell">insert</code> like
so:</p>
<div class="sourceCode" id="cb11"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb11-1"><a href="https://doisinkidney.com/rss.xml#cb11-1" tabindex="-1"/><span class="ot">insert ::</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> k <span class="ot">-&gt;</span> v <span class="ot">-&gt;</span> <span class="dt">Heap</span> k v <span class="ot">-&gt;</span> <span class="dt">Heap</span> k v</span>
<span id="cb11-2"><a href="https://doisinkidney.com/rss.xml#cb11-2" tabindex="-1"/>insert k v hp <span class="ot">=</span> <span class="dt">Just</span> (<span class="dt">Root</span> k v []) <span class="op">&lt;&gt;</span> hp</span></code></pre></div>
<p>And <code class="sourceCode haskell">popMin</code> is also relatively
simple:</p>
<div class="sourceCode" id="cb12"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb12-1"><a href="https://doisinkidney.com/rss.xml#cb12-1" tabindex="-1"/><span class="ot">delay ::</span> <span class="dt">Semigroup</span> k <span class="ot">=&gt;</span> k <span class="ot">-&gt;</span> <span class="dt">Heap</span> k v <span class="ot">-&gt;</span> <span class="dt">Heap</span> k v</span>
<span id="cb12-2"><a href="https://doisinkidney.com/rss.xml#cb12-2" tabindex="-1"/>delay by <span class="ot">=</span> <span class="fu">fmap</span> (\(<span class="dt">Root</span> k v xs) <span class="ot">-&gt;</span> <span class="dt">Root</span> (by <span class="op">&lt;&gt;</span> k) v xs)</span>
<span id="cb12-3"><a href="https://doisinkidney.com/rss.xml#cb12-3" tabindex="-1"/></span>
<span id="cb12-4"><a href="https://doisinkidney.com/rss.xml#cb12-4" tabindex="-1"/><span class="ot">popMin ::</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> <span class="dt">Heap</span> k v <span class="ot">-&gt;</span> <span class="dt">Maybe</span> (v,<span class="dt">Heap</span> k v)</span>
<span id="cb12-5"><a href="https://doisinkidney.com/rss.xml#cb12-5" tabindex="-1"/>popMin <span class="ot">=</span> <span class="fu">fmap</span> (\(<span class="dt">Root</span> k v xs) <span class="ot">-&gt;</span> (v, k <span class="ot">`delay`</span> merges xs))</span></code></pre></div>
<p>Notice that we <code class="sourceCode haskell">delay</code> the rest
of the heap, because all of its entries need to be offset by the weight
of the previous root node. Thankfully, because weâ€™re only storing the
differences, we can â€œmodifyâ€� every weight by just increasing the weight
of the root, making this an
<math display="inline">&lt;semantics&gt;<mrow><mi>ğ�’ª</mi><mo form="prefix" stretchy="false">(</mo><mn>1</mn><mo form="postfix" stretchy="false">)</mo></mrow>&lt;annotation encoding="application/x-tex"&gt;\mathcal{O}(1)&lt;/annotation&gt;&lt;/semantics&gt;</math>
operation.</p>
<p>Finally, we can implement heap sort like so:</p>
<div class="sourceCode" id="cb13"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb13-1"><a href="https://doisinkidney.com/rss.xml#cb13-1" tabindex="-1"/><span class="ot">sortOn ::</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> (a <span class="ot">-&gt;</span> k) <span class="ot">-&gt;</span> [a] <span class="ot">-&gt;</span> [a]</span>
<span id="cb13-2"><a href="https://doisinkidney.com/rss.xml#cb13-2" tabindex="-1"/>sortOn k <span class="ot">=</span> unfoldr popMin <span class="op">.</span> foldl' (\xs x <span class="ot">-&gt;</span> insert (k x) x xs) <span class="dt">Nothing</span></span></code></pre></div>
<p>And it does indeed work:</p>
<div class="sourceCode" id="cb14"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb14-1"><a href="https://doisinkidney.com/rss.xml#cb14-1" tabindex="-1"/><span class="op">&gt;&gt;&gt;</span> sortOn <span class="dt">Sum</span> [<span class="dv">3</span>,<span class="dv">4</span>,<span class="dv">2</span>,<span class="dv">5</span>,<span class="dv">1</span>]</span>
<span id="cb14-2"><a href="https://doisinkidney.com/rss.xml#cb14-2" tabindex="-1"/>[<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">3</span>,<span class="dv">4</span>,<span class="dv">5</span>]</span></code></pre></div>
<p>Here is a trace of the output:</p>
<details>

Trace

<pre class="text"><code>Input           Heap:
list:

[3,4,2,5,1]


[4,2,5,1]       3


[2,5,1]         3â”€1


[5,1]           2â”€1â”€1


[1]              â”Œ3
                2â”¤
                 â””1â”€1


[]                 â”Œ3
                1â”€1â”¤
                   â””1â”€1


Output          Heap:
list:

[]                 â”Œ3
                1â”€1â”¤
                   â””1â”€1


[1]              â”Œ3
                2â”¤
                 â””1â”€1


[1,2]            â”Œ2
                3â”¤
                 â””1


[1,2,3]         4â”€1


[1,2,3,4]       5


[1,2,3,4,5]</code></pre>
</details>
<p>While the heap implementation presented here is pretty efficient,
note that we could significantly improve its performance with a few
optimisations: first, we could unpack all of the constructors, using a
custom list definition in <code class="sourceCode haskell"><span class="dt">Root</span></code> instead
of Haskellâ€™s built-in lists; second, in <code class="sourceCode haskell">foldl'</code> we could avoid the <code class="sourceCode haskell"><span class="dt">Maybe</span></code> wrapper
by building a non-empty heap. There are probably more small
optimisations available as well.</p>
<h1 id="retrieving-a-normal-heap">Retrieving a Normal Heap</h1>
<p>A problem with the definition of <code class="sourceCode haskell">sortOn</code> above is that it requires a
<code class="sourceCode haskell"><span class="dt">Monus</span></code>
instance on the keys, but it only really needs <code class="sourceCode haskell"><span class="dt">Ord</span></code>. It seems
that by switching to the <code class="sourceCode haskell"><span class="dt">Monus</span></code>-powered
heap we have lost some generality.</p>
<p>Luckily, there are two monuses we can use to solve this problem:</p>
<div class="sourceCode" id="cb16"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb16-1"><a href="https://doisinkidney.com/rss.xml#cb16-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Ord</span> a <span class="ot">=&gt;</span> <span class="dt">Monus</span> (<span class="dt">Max</span> a) <span class="kw">where</span></span>
<span id="cb16-2"><a href="https://doisinkidney.com/rss.xml#cb16-2" tabindex="-1"/>  x âˆ¸ y <span class="ot">=</span> x</span>
<span id="cb16-3"><a href="https://doisinkidney.com/rss.xml#cb16-3" tabindex="-1"/></span>
<span id="cb16-4"><a href="https://doisinkidney.com/rss.xml#cb16-4" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Ord</span> a <span class="ot">=&gt;</span> <span class="dt">Monus</span> (<span class="dt">Last</span> a) <span class="kw">where</span></span>
<span id="cb16-5"><a href="https://doisinkidney.com/rss.xml#cb16-5" tabindex="-1"/>  x âˆ¸ y <span class="ot">=</span> x</span></code></pre></div>
<p>The <code class="sourceCode haskell"><span class="dt">Max</span></code> semigroup
uses the <code class="sourceCode haskell"><span class="fu">max</span></code> operation,
and the <code class="sourceCode haskell"><span class="dt">Last</span></code> semigroup
returns its second operand.</p>
<div class="sourceCode" id="cb17"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb17-1"><a href="https://doisinkidney.com/rss.xml#cb17-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Ord</span> a <span class="ot">=&gt;</span> <span class="dt">Semigroup</span> (<span class="dt">Max</span> a) <span class="kw">where</span></span>
<span id="cb17-2"><a href="https://doisinkidney.com/rss.xml#cb17-2" tabindex="-1"/>  <span class="dt">Max</span> x <span class="op">&lt;&gt;</span> <span class="dt">Max</span> y <span class="ot">=</span> <span class="dt">Max</span> (<span class="fu">max</span> x y)</span>
<span id="cb17-3"><a href="https://doisinkidney.com/rss.xml#cb17-3" tabindex="-1"/></span>
<span id="cb17-4"><a href="https://doisinkidney.com/rss.xml#cb17-4" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Semigroup</span> (<span class="dt">Last</span> a) <span class="kw">where</span></span>
<span id="cb17-5"><a href="https://doisinkidney.com/rss.xml#cb17-5" tabindex="-1"/>  x <span class="op">&lt;&gt;</span> y <span class="ot">=</span> y</span></code></pre></div>
<p>While the <code class="sourceCode haskell"><span class="dt">Monus</span></code>
instances here might seem degenerate, they do actually satisfy the <code class="sourceCode haskell"><span class="dt">Monus</span></code> laws as
given above.</p>
<details>

<code class="sourceCode haskell"><span class="dt">Max</span></code> and
<code class="sourceCode haskell"><span class="dt">Last</span></code>
Monus laws

<p>Max:</p>
<div class="sourceCode" id="cb18"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb18-1"><a href="https://doisinkidney.com/rss.xml#cb18-1" tabindex="-1"/>x <span class="op">&lt;=</span> y <span class="op">==&gt;</span> x <span class="op">&lt;&gt;</span> (y âˆ¸ x) <span class="ot">=</span> y</span>
<span id="cb18-2"><a href="https://doisinkidney.com/rss.xml#cb18-2" tabindex="-1"/><span class="dt">Max</span> x <span class="op">&lt;=</span> <span class="dt">Max</span> y <span class="op">==&gt;</span> <span class="dt">Max</span> x <span class="op">&lt;&gt;</span> (<span class="dt">Max</span> y âˆ¸ <span class="dt">Max</span> x) <span class="ot">=</span> <span class="dt">Max</span> y</span>
<span id="cb18-3"><a href="https://doisinkidney.com/rss.xml#cb18-3" tabindex="-1"/><span class="dt">Max</span> x <span class="op">&lt;=</span> <span class="dt">Max</span> y <span class="op">==&gt;</span> <span class="dt">Max</span> x <span class="op">&lt;&gt;</span> <span class="dt">Max</span> y <span class="ot">=</span> <span class="dt">Max</span> y</span>
<span id="cb18-4"><a href="https://doisinkidney.com/rss.xml#cb18-4" tabindex="-1"/><span class="dt">Max</span> x <span class="op">&lt;=</span> <span class="dt">Max</span> y <span class="op">==&gt;</span> <span class="dt">Max</span> (<span class="fu">max</span> x y) <span class="ot">=</span> <span class="dt">Max</span> y</span>
<span id="cb18-5"><a href="https://doisinkidney.com/rss.xml#cb18-5" tabindex="-1"/>x <span class="op">&lt;=</span> y <span class="op">==&gt;</span> <span class="fu">max</span> x y <span class="ot">=</span> y</span>
<span id="cb18-6"><a href="https://doisinkidney.com/rss.xml#cb18-6" tabindex="-1"/></span>
<span id="cb18-7"><a href="https://doisinkidney.com/rss.xml#cb18-7" tabindex="-1"/>x <span class="op">&lt;=</span> y <span class="op">==&gt;</span> z <span class="op">&lt;&gt;</span> x <span class="op">&lt;=</span> z <span class="op">&lt;&gt;</span> y</span>
<span id="cb18-8"><a href="https://doisinkidney.com/rss.xml#cb18-8" tabindex="-1"/><span class="dt">Max</span> x <span class="op">&lt;=</span> <span class="dt">Max</span> y <span class="op">==&gt;</span> <span class="dt">Max</span> z <span class="op">&lt;&gt;</span> <span class="dt">Max</span> x <span class="op">&lt;=</span> <span class="dt">Max</span> z <span class="op">&lt;&gt;</span> <span class="dt">Max</span> y</span>
<span id="cb18-9"><a href="https://doisinkidney.com/rss.xml#cb18-9" tabindex="-1"/><span class="dt">Max</span> x <span class="op">&lt;=</span> <span class="dt">Max</span> y <span class="op">==&gt;</span> <span class="dt">Max</span> (<span class="fu">max</span> z x) <span class="op">&lt;=</span> <span class="dt">Max</span> (<span class="fu">max</span> z y)</span>
<span id="cb18-10"><a href="https://doisinkidney.com/rss.xml#cb18-10" tabindex="-1"/>x <span class="op">&lt;=</span> y <span class="op">==&gt;</span> <span class="fu">max</span> z x <span class="op">&lt;=</span> <span class="fu">max</span> z y</span></code></pre></div>
<p>Last:</p>
<div class="sourceCode" id="cb19"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb19-1"><a href="https://doisinkidney.com/rss.xml#cb19-1" tabindex="-1"/>x <span class="op">&lt;=</span> y <span class="op">==&gt;</span> x <span class="op">&lt;&gt;</span> (y âˆ¸ x) <span class="ot">=</span> y</span>
<span id="cb19-2"><a href="https://doisinkidney.com/rss.xml#cb19-2" tabindex="-1"/><span class="dt">Last</span> x <span class="op">&lt;=</span> <span class="dt">Last</span> y <span class="op">==&gt;</span> <span class="dt">Last</span> x <span class="op">&lt;&gt;</span> (<span class="dt">Last</span> y âˆ¸ <span class="dt">Last</span> x) <span class="ot">=</span> <span class="dt">Last</span> y</span>
<span id="cb19-3"><a href="https://doisinkidney.com/rss.xml#cb19-3" tabindex="-1"/><span class="dt">Last</span> x <span class="op">&lt;=</span> <span class="dt">Last</span> y <span class="op">==&gt;</span> <span class="dt">Last</span> x <span class="op">&lt;&gt;</span> <span class="dt">Last</span> y <span class="ot">=</span> <span class="dt">Last</span> y</span>
<span id="cb19-4"><a href="https://doisinkidney.com/rss.xml#cb19-4" tabindex="-1"/><span class="dt">Last</span> x <span class="op">&lt;=</span> <span class="dt">Last</span> y <span class="op">==&gt;</span> <span class="dt">Last</span> y <span class="ot">=</span> <span class="dt">Last</span> y</span>
<span id="cb19-5"><a href="https://doisinkidney.com/rss.xml#cb19-5" tabindex="-1"/></span>
<span id="cb19-6"><a href="https://doisinkidney.com/rss.xml#cb19-6" tabindex="-1"/>x <span class="op">&lt;=</span> y <span class="op">==&gt;</span> z <span class="op">&lt;&gt;</span> x <span class="op">&lt;=</span> z <span class="op">&lt;&gt;</span> y</span>
<span id="cb19-7"><a href="https://doisinkidney.com/rss.xml#cb19-7" tabindex="-1"/><span class="dt">Last</span> x <span class="op">&lt;=</span> <span class="dt">Last</span> y <span class="op">==&gt;</span> <span class="dt">Last</span> z <span class="op">&lt;&gt;</span> <span class="dt">Last</span> x <span class="op">&lt;=</span> <span class="dt">Last</span> z <span class="op">&lt;&gt;</span> <span class="dt">Last</span> y</span>
<span id="cb19-8"><a href="https://doisinkidney.com/rss.xml#cb19-8" tabindex="-1"/><span class="dt">Last</span> x <span class="op">&lt;=</span> <span class="dt">Last</span> y <span class="op">==&gt;</span> <span class="dt">Last</span> x <span class="op">&lt;=</span> <span class="dt">Last</span> y</span></code></pre></div>
</details>
<p>Either <code class="sourceCode haskell"><span class="dt">Max</span></code> or <code class="sourceCode haskell"><span class="dt">Last</span></code> will
work; semantically, thereâ€™s no real difference. <code class="sourceCode haskell"><span class="dt">Last</span></code> avoids
some comparisons, so we can use that:</p>
<div class="sourceCode" id="cb20"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb20-1"><a href="https://doisinkidney.com/rss.xml#cb20-1" tabindex="-1"/><span class="ot">sortOn' ::</span> <span class="dt">Ord</span> k <span class="ot">=&gt;</span> (a <span class="ot">-&gt;</span> k) <span class="ot">-&gt;</span> [a] <span class="ot">-&gt;</span> [a]</span>
<span id="cb20-2"><a href="https://doisinkidney.com/rss.xml#cb20-2" tabindex="-1"/>sortOn' k <span class="ot">=</span> sortOn (<span class="dt">Last</span> <span class="op">.</span> k)</span></code></pre></div>
<h1 id="phases-as-a-pairing-heap">Phases as a Pairing Heap</h1>
<p>The <code class="sourceCode haskell"><span class="dt">Phases</span></code>
applicative <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-easterly_functions_2019">Easterly
2019</a>)</span> is an <code class="sourceCode haskell"><span class="dt">Applicative</span></code>
transformer that allows reordering of <code class="sourceCode haskell"><span class="dt">Applicative</span></code>
effects in an easy-to-use, high-level way. The interface looks like
this:</p>
<div class="sourceCode" id="cb21"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb21-1"><a href="https://doisinkidney.com/rss.xml#cb21-1" tabindex="-1"/><span class="ot">phase     ::</span> <span class="dt">Natural</span> <span class="ot">-&gt;</span> f a <span class="ot">-&gt;</span> <span class="dt">Phases</span> f a</span>
<span id="cb21-2"><a href="https://doisinkidney.com/rss.xml#cb21-2" tabindex="-1"/><span class="ot">runPhases ::</span> <span class="dt">Applicative</span> f <span class="ot">=&gt;</span> <span class="dt">Phases</span> f a <span class="ot">-&gt;</span> f a</span>
<span id="cb21-3"><a href="https://doisinkidney.com/rss.xml#cb21-3" tabindex="-1"/></span>
<span id="cb21-4"><a href="https://doisinkidney.com/rss.xml#cb21-4" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Applicative</span> f <span class="ot">=&gt;</span> <span class="dt">Applicative</span> (<span class="dt">Phases</span> f)</span></code></pre></div>
<p>And we can use it like this:</p>
<div class="sourceCode" id="cb22"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb22-1"><a href="https://doisinkidney.com/rss.xml#cb22-1" tabindex="-1"/><span class="ot">phased ::</span> <span class="dt">IO</span> <span class="dt">String</span></span>
<span id="cb22-2"><a href="https://doisinkidney.com/rss.xml#cb22-2" tabindex="-1"/>phased <span class="ot">=</span> runPhases <span class="op">$</span> <span class="fu">sequenceA</span></span>
<span id="cb22-3"><a href="https://doisinkidney.com/rss.xml#cb22-3" tabindex="-1"/>  [ phase <span class="dv">3</span> <span class="op">$</span> emit <span class="ch">'a'</span></span>
<span id="cb22-4"><a href="https://doisinkidney.com/rss.xml#cb22-4" tabindex="-1"/>  , phase <span class="dv">2</span> <span class="op">$</span> emit <span class="ch">'b'</span></span>
<span id="cb22-5"><a href="https://doisinkidney.com/rss.xml#cb22-5" tabindex="-1"/>  , phase <span class="dv">1</span> <span class="op">$</span> emit <span class="ch">'c'</span></span>
<span id="cb22-6"><a href="https://doisinkidney.com/rss.xml#cb22-6" tabindex="-1"/>  , phase <span class="dv">2</span> <span class="op">$</span> emit <span class="ch">'d'</span></span>
<span id="cb22-7"><a href="https://doisinkidney.com/rss.xml#cb22-7" tabindex="-1"/>  , phase <span class="dv">3</span> <span class="op">$</span> emit <span class="ch">'e'</span> ]</span>
<span id="cb22-8"><a href="https://doisinkidney.com/rss.xml#cb22-8" tabindex="-1"/>  <span class="kw">where</span> emit c <span class="ot">=</span> <span class="fu">putChar</span> c <span class="op">&gt;&gt;</span> <span class="fu">return</span> c</span>
<span id="cb22-9"><a href="https://doisinkidney.com/rss.xml#cb22-9" tabindex="-1"/></span>
<span id="cb22-10"><a href="https://doisinkidney.com/rss.xml#cb22-10" tabindex="-1"/><span class="op">&gt;&gt;&gt;</span> phased</span>
<span id="cb22-11"><a href="https://doisinkidney.com/rss.xml#cb22-11" tabindex="-1"/>cbdae</span>
<span id="cb22-12"><a href="https://doisinkidney.com/rss.xml#cb22-12" tabindex="-1"/><span class="st">"abcde"</span></span></code></pre></div>
<p>The above computation performs the <em>effects</em> in the order
dictated by their phases (this is why the characters are printed out in
the order <code class="sourceCode haskell">cbdae</code>), but the pure
value (the returned string) has its order unaffected.</p>
<p>I have written about this type <a href="https://doisinkidney.com/2019-05-28-linear-phases.html">before</a>, and in a handful of
papers <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-kidney_algebras_2021">2021</a>; <a href="https://doisinkidney.com/rss.xml#ref-gibbons_breadthfirst_2022">Gibbons et
al. 2022</a>; <a href="https://doisinkidney.com/rss.xml#ref-gibbons_phases_2023">Gibbons et al. 2023</a>)</span>, but more recently
<span class="citation">BlÃ¶ndal (<a href="https://doisinkidney.com/rss.xml#ref-blondal_generalized_2025">2025a</a>)</span> started looking into trying to
use the <code class="sourceCode haskell"><span class="dt">Phases</span></code> pattern
with arbitrary ordered keys <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-visscher_phases_2025">Visscher 2025</a>;
<a href="https://doisinkidney.com/rss.xml#ref-blondal_phases_2025">BlÃ¶ndal
2025b</a>)</span>. There are a lot of different directions you can go
from the <code class="sourceCode haskell"><span class="dt">Phases</span></code> type;
what interested me most immediately was the idea of implementing the
type efficiently using standard data-structure representations. If our
core goal here is to order some values according to a key, then that is
clearly a problem that a heap should solve: enter the free applicative
pairing heap.</p>
<p>Here is the typeâ€™s definition:</p>
<div class="sourceCode" id="cb23"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb23-1"><a href="https://doisinkidney.com/rss.xml#cb23-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Heap</span> k f a <span class="kw">where</span></span>
<span id="cb23-2"><a href="https://doisinkidney.com/rss.xml#cb23-2" tabindex="-1"/>  <span class="dt">Pure</span><span class="ot"> ::</span> a <span class="ot">-&gt;</span> <span class="dt">Heap</span> k f a</span>
<span id="cb23-3"><a href="https://doisinkidney.com/rss.xml#cb23-3" tabindex="-1"/>  <span class="dt">Root</span><span class="ot"> ::</span> <span class="op">!</span>k <span class="ot">-&gt;</span> (x <span class="ot">-&gt;</span> y <span class="ot">-&gt;</span> a) <span class="ot">-&gt;</span> f x <span class="ot">-&gt;</span> <span class="dt">Heaps</span> k f y <span class="ot">-&gt;</span> <span class="dt">Heap</span> k f a</span>
<span id="cb23-4"><a href="https://doisinkidney.com/rss.xml#cb23-4" tabindex="-1"/></span>
<span id="cb23-5"><a href="https://doisinkidney.com/rss.xml#cb23-5" tabindex="-1"/><span class="kw">data</span> <span class="dt">Heaps</span> k f a <span class="kw">where</span></span>
<span id="cb23-6"><a href="https://doisinkidney.com/rss.xml#cb23-6" tabindex="-1"/>  <span class="dt">Nil</span><span class="ot"> ::</span> <span class="dt">Heaps</span> k f ()</span>
<span id="cb23-7"><a href="https://doisinkidney.com/rss.xml#cb23-7" tabindex="-1"/>  <span class="dt">App</span><span class="ot"> ::</span> <span class="op">!</span>k <span class="ot">-&gt;</span> f x <span class="ot">-&gt;</span> <span class="dt">Heaps</span> k f y <span class="ot">-&gt;</span> <span class="dt">Heaps</span> k f z <span class="ot">-&gt;</span> <span class="dt">Heaps</span> k f (x,y,z)</span></code></pre></div>
<p>We have had to change a few aspects of the original pairing heap, but
the overall structure remains. The entries in this heap are now
effectful computations: the <code class="sourceCode haskell">f</code>s.
The data structure also contains some scaffolding to reconstruct the
pure values â€œinsideâ€� each effect when we actually run the heap.</p>
<p>The root-level structure is the <code class="sourceCode haskell"><span class="dt">Heap</span></code>: this can
either be <code class="sourceCode haskell"><span class="dt">Pure</span></code>
(corresponding to an empty heap: notice that, though this constructor
has some contents (the <code class="sourceCode haskell">a</code>), it is
still regarded as â€œemptyâ€� because it contains no effects (<code class="sourceCode haskell">f</code>)); or a <code class="sourceCode haskell"><span class="dt">Root</span></code>, which is
a singleton value, paired with the list of sub-heaps represented by the
<code class="sourceCode haskell"><span class="dt">Heaps</span></code>
type. Weâ€™re using the usual Yoneda-ish trick here to allow the top-level
data type to be parametric and a <code class="sourceCode haskell"><span class="dt">Functor</span></code>, by
storing the function <code class="sourceCode haskell">x <span class="ot">-&gt;</span> y <span class="ot">-&gt;</span> a</code>.</p>
<p>The <code class="sourceCode haskell"><span class="dt">Heaps</span></code> type
then plays the role of <code class="sourceCode haskell">[<span class="dt">Root</span> k v]</code> in
the previous pairing heap implementation; here, we have inlined all of
the constructors so that we can get all of the types to line up.
Remember, this is a heap of <em>effects</em>, not of pure values: the
pure values need to be able to be reconstructed to one single top-level
<code class="sourceCode haskell">a</code> when we run the heap at the
end.</p>
<p>Merging two heaps happens in the <code class="sourceCode haskell"><span class="dt">Applicative</span></code>
instance itself:</p>
<div class="sourceCode" id="cb24"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb24-1"><a href="https://doisinkidney.com/rss.xml#cb24-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Functor</span> (<span class="dt">Heap</span> k f) <span class="kw">where</span></span>
<span id="cb24-2"><a href="https://doisinkidney.com/rss.xml#cb24-2" tabindex="-1"/>  <span class="fu">fmap</span> f (<span class="dt">Pure</span> x) <span class="ot">=</span> <span class="dt">Pure</span> (f x)</span>
<span id="cb24-3"><a href="https://doisinkidney.com/rss.xml#cb24-3" tabindex="-1"/>  <span class="fu">fmap</span> f (<span class="dt">Root</span> k c x xs) <span class="ot">=</span> <span class="dt">Root</span> k (\a b <span class="ot">-&gt;</span> f (c a b)) x xs</span>
<span id="cb24-4"><a href="https://doisinkidney.com/rss.xml#cb24-4" tabindex="-1"/></span>
<span id="cb24-5"><a href="https://doisinkidney.com/rss.xml#cb24-5" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> <span class="dt">Applicative</span> (<span class="dt">Heap</span> k f) <span class="kw">where</span></span>
<span id="cb24-6"><a href="https://doisinkidney.com/rss.xml#cb24-6" tabindex="-1"/>  <span class="fu">pure</span> <span class="ot">=</span> <span class="dt">Pure</span></span>
<span id="cb24-7"><a href="https://doisinkidney.com/rss.xml#cb24-7" tabindex="-1"/>  <span class="dt">Pure</span> f <span class="op">&lt;*&gt;</span> xs <span class="ot">=</span> <span class="fu">fmap</span> f xs</span>
<span id="cb24-8"><a href="https://doisinkidney.com/rss.xml#cb24-8" tabindex="-1"/>  xs <span class="op">&lt;*&gt;</span> <span class="dt">Pure</span> f <span class="ot">=</span> <span class="fu">fmap</span> (<span class="op">$</span> f) xs</span>
<span id="cb24-9"><a href="https://doisinkidney.com/rss.xml#cb24-9" tabindex="-1"/></span>
<span id="cb24-10"><a href="https://doisinkidney.com/rss.xml#cb24-10" tabindex="-1"/>  <span class="dt">Root</span> xk xc xs xss <span class="op">&lt;*&gt;</span> <span class="dt">Root</span> yk yc ys yss</span>
<span id="cb24-11"><a href="https://doisinkidney.com/rss.xml#cb24-11" tabindex="-1"/>    <span class="op">|</span> xk <span class="op">&lt;=</span> yk  <span class="ot">=</span> <span class="dt">Root</span> xk (\a (b,c,d) <span class="ot">-&gt;</span> xc a d (yc b c)) xs (<span class="dt">App</span> (yk âˆ¸ xk) ys yss xss)</span>
<span id="cb24-12"><a href="https://doisinkidney.com/rss.xml#cb24-12" tabindex="-1"/>    <span class="op">|</span> <span class="fu">otherwise</span> <span class="ot">=</span> <span class="dt">Root</span> yk (\a (b,c,d) <span class="ot">-&gt;</span> xc b c (yc a d)) ys (<span class="dt">App</span> (xk âˆ¸ yk) xs xss yss)</span></code></pre></div>
<p>To actually run the heap we will use the following two functions:</p>
<div class="sourceCode" id="cb25"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb25-1"><a href="https://doisinkidney.com/rss.xml#cb25-1" tabindex="-1"/><span class="ot">merges ::</span> (<span class="dt">Monus</span> k, <span class="dt">Applicative</span> f) <span class="ot">=&gt;</span> <span class="dt">Heaps</span> k f a <span class="ot">-&gt;</span> <span class="dt">Heap</span> k f a</span>
<span id="cb25-2"><a href="https://doisinkidney.com/rss.xml#cb25-2" tabindex="-1"/>merges <span class="dt">Nil</span> <span class="ot">=</span> <span class="dt">Pure</span> ()</span>
<span id="cb25-3"><a href="https://doisinkidney.com/rss.xml#cb25-3" tabindex="-1"/>merges (<span class="dt">App</span> k1 e1 t1 <span class="dt">Nil</span>) <span class="ot">=</span> <span class="dt">Root</span> k1 (,,()) e1 t1</span>
<span id="cb25-4"><a href="https://doisinkidney.com/rss.xml#cb25-4" tabindex="-1"/>merges (<span class="dt">App</span> k1 e1 t1 (<span class="dt">App</span> k2 e2 t2 xs)) <span class="ot">=</span></span>
<span id="cb25-5"><a href="https://doisinkidney.com/rss.xml#cb25-5" tabindex="-1"/>   (<span class="dt">Root</span> k1 (\a b cd es <span class="ot">-&gt;</span> (a,b, cd es)) e1 t1 <span class="op">&lt;*&gt;</span> <span class="dt">Root</span> k2 (,,) e2 t2) <span class="op">&lt;*&gt;</span> merges xs</span>
<span id="cb25-6"><a href="https://doisinkidney.com/rss.xml#cb25-6" tabindex="-1"/></span>
<span id="cb25-7"><a href="https://doisinkidney.com/rss.xml#cb25-7" tabindex="-1"/><span class="ot">runHeap ::</span> (<span class="dt">Monus</span> k, <span class="dt">Applicative</span> f) <span class="ot">=&gt;</span> <span class="dt">Heap</span> k f a <span class="ot">-&gt;</span> f a</span>
<span id="cb25-8"><a href="https://doisinkidney.com/rss.xml#cb25-8" tabindex="-1"/>runHeap (<span class="dt">Pure</span> x) <span class="ot">=</span> <span class="fu">pure</span> x</span>
<span id="cb25-9"><a href="https://doisinkidney.com/rss.xml#cb25-9" tabindex="-1"/>runHeap (<span class="dt">Root</span> _ c x xs) <span class="ot">=</span> liftA2 c x (runHeap (merges xs))</span></code></pre></div>
<p>And we can lift a computation into <code class="sourceCode haskell"><span class="dt">Phases</span></code> like
so:</p>
<div class="sourceCode" id="cb26"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb26-1"><a href="https://doisinkidney.com/rss.xml#cb26-1" tabindex="-1"/><span class="ot">phase ::</span> k <span class="ot">-&gt;</span> f a <span class="ot">-&gt;</span> <span class="dt">Heap</span> k f a</span>
<span id="cb26-2"><a href="https://doisinkidney.com/rss.xml#cb26-2" tabindex="-1"/>phase k xs <span class="ot">=</span> <span class="dt">Root</span> k <span class="fu">const</span> xs <span class="dt">Nil</span></span></code></pre></div>
<h1 id="stabilising-phases">Stabilising Phases</h1>
<p>Thereâ€™s a problem. A heap sort based on a pairing heap isnâ€™t
<em>stable</em>. That means that the order of effects here can vary for
two effects in the same phase. If we look back to the example with the
strings we saw above, that means that outputs like <code class="sourceCode haskell">cdbea</code> would be possible (in actual
fact, we donâ€™t get any reordering in this particular example, but thatâ€™s
just an accident of the way the applicative operators are associated
under the hood).</p>
<p>This is problematic because we would expect effects in the same phase
to behave as if they were normal applicative effects, sequenced
according to their syntactic order. It also means that the applicative
transformer breaks the applicative laws, because effects might be
reordered according to the association of the applicative operators,
which should lawfully be associative.</p>
<p>To make the sort stable, we could layer the heap effect with some
state effect that would tag each effect with its order. However, that
would hurt efficiency and composability: it would force us to linearise
the whole heap sort procedure, where currently different branches of the
tree can compute completely independently of each other. The solution
comes in the form of another monus: the key monus.</p>
<div class="sourceCode" id="cb27"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb27-1"><a href="https://doisinkidney.com/rss.xml#cb27-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Key</span> k <span class="ot">=</span> <span class="op">!</span>k <span class="op">:*</span> <span class="ot">{-# UNPACK #-}</span> <span class="op">!</span><span class="dt">Int</span> <span class="kw">deriving</span> (<span class="dt">Eq</span>, <span class="dt">Ord</span>)</span></code></pre></div>
<p>A <code class="sourceCode haskell"><span class="dt">Key</span> k</code> is some
ordered key <code class="sourceCode haskell">k</code> coupled with an
<code class="sourceCode haskell"><span class="dt">Int</span></code> that
represents the offset between the original position and the current
position of the key. In this way, when two keys compare as equal, we can
cascade on to compare their original positions, thereby maintaining
their original order when there is ambiguity caused by a key collision.
However, in contrast to the approach of walking over the data once and
tagging it all with positions, this approach keeps the location
information completely local: we never need to know that some key is in
the
<math display="inline">&lt;semantics&gt;<mi>n</mi>&lt;annotation encoding="application/x-tex"&gt;n&lt;/annotation&gt;&lt;/semantics&gt;</math>th
position in the original sequence, only that it has moved
<math display="inline">&lt;semantics&gt;<mi>n</mi>&lt;annotation encoding="application/x-tex"&gt;n&lt;/annotation&gt;&lt;/semantics&gt;</math>
steps from its original position.</p>
<p>The instances are as follows:</p>
<div class="sourceCode" id="cb28"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb28-1"><a href="https://doisinkidney.com/rss.xml#cb28-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Semigroup</span> (<span class="dt">Key</span> k) <span class="kw">where</span></span>
<span id="cb28-2"><a href="https://doisinkidney.com/rss.xml#cb28-2" tabindex="-1"/>  (xk <span class="op">:*</span> xi) <span class="op">&lt;&gt;</span> (yk <span class="op">:*</span> yi) <span class="ot">=</span> yk <span class="op">:*</span> (xi <span class="op">+</span> yi)</span>
<span id="cb28-3"><a href="https://doisinkidney.com/rss.xml#cb28-3" tabindex="-1"/></span>
<span id="cb28-4"><a href="https://doisinkidney.com/rss.xml#cb28-4" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Ord</span> k <span class="ot">=&gt;</span> <span class="dt">Monus</span> (<span class="dt">Key</span> k) <span class="kw">where</span></span>
<span id="cb28-5"><a href="https://doisinkidney.com/rss.xml#cb28-5" tabindex="-1"/>  (xk <span class="op">:*</span> xi) âˆ¸ (yk <span class="op">:*</span> yi) <span class="ot">=</span> xk <span class="op">:*</span> (xi <span class="op">-</span> yi)</span></code></pre></div>
<p>This instance is basically a combination of the <code class="sourceCode haskell"><span class="dt">Last</span></code> semigroup
and the
<math display="inline">&lt;semantics&gt;<mrow><mo form="prefix" stretchy="false">(</mo><mi>â„¤</mi><mo>,</mo><mi>+</mi><mo>,</mo><mn>0</mn><mo form="postfix" stretchy="false">)</mo></mrow>&lt;annotation encoding="application/x-tex"&gt;(\mathbb{Z}, +, 0)&lt;/annotation&gt;&lt;/semantics&gt;</math>
group. We could make a slightly more generalised version of <code class="sourceCode haskell"><span class="dt">Key</span></code> that is
the combination of any monus and
<math display="inline">&lt;semantics&gt;<mi>â„¤</mi>&lt;annotation encoding="application/x-tex"&gt;\mathbb{Z}&lt;/annotation&gt;&lt;/semantics&gt;</math>,
but since Iâ€™m only going to be using this type for simple sorting-like
algorithms I will leave that generalisation for another time.</p>
<p>The stable heap type is as follows:</p>
<div class="sourceCode" id="cb29"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb29-1"><a href="https://doisinkidney.com/rss.xml#cb29-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Stable</span> k f a</span>
<span id="cb29-2"><a href="https://doisinkidney.com/rss.xml#cb29-2" tabindex="-1"/>  <span class="ot">=</span> <span class="dt">Stable</span> {<span class="ot"> size ::</span> <span class="ot">{-# UNPACK #-}</span> <span class="op">!</span><span class="dt">Int</span></span>
<span id="cb29-3"><a href="https://doisinkidney.com/rss.xml#cb29-3" tabindex="-1"/>           ,<span class="ot"> heap ::</span> <span class="op">!</span>(<span class="dt">Heap</span> (<span class="dt">Key</span> k) f a) }</span></code></pre></div>
<p>We need to track the size of the heap so that we can supply the
right-hand operand with their offsets. Because weâ€™re storing
differences, we can add an offset to every entry in a heap in
<math display="inline">&lt;semantics&gt;<mrow><mi>ğ�’ª</mi><mo form="prefix" stretchy="false">(</mo><mn>1</mn><mo form="postfix" stretchy="false">)</mo></mrow>&lt;annotation encoding="application/x-tex"&gt;\mathcal{O}(1)&lt;/annotation&gt;&lt;/semantics&gt;</math>
time by simply adding to the root:</p>
<div class="sourceCode" id="cb30"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb30-1"><a href="https://doisinkidney.com/rss.xml#cb30-1" tabindex="-1"/><span class="ot">delayKey ::</span> <span class="dt">Int</span> <span class="ot">-&gt;</span> <span class="dt">Heap</span> (<span class="dt">Key</span> k) f a <span class="ot">-&gt;</span> <span class="dt">Heap</span> (<span class="dt">Key</span> k) f a</span>
<span id="cb30-2"><a href="https://doisinkidney.com/rss.xml#cb30-2" tabindex="-1"/>delayKey _ hp<span class="op">@</span>(<span class="dt">Pure</span> _) <span class="ot">=</span> hp</span>
<span id="cb30-3"><a href="https://doisinkidney.com/rss.xml#cb30-3" tabindex="-1"/>delayKey n (<span class="dt">Root</span> (k <span class="op">:*</span> m) c x xs) <span class="ot">=</span> <span class="dt">Root</span> (k <span class="op">:*</span> (n <span class="op">+</span> m)) c x xs</span></code></pre></div>
<p>Finally, using this we can implement the <code class="sourceCode haskell"><span class="dt">Applicative</span></code>
instance and the rest of the interface:</p>
<div class="sourceCode" id="cb31"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb31-1"><a href="https://doisinkidney.com/rss.xml#cb31-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Ord</span> k <span class="ot">=&gt;</span> <span class="dt">Applicative</span> (<span class="dt">Stable</span> k f) <span class="kw">where</span></span>
<span id="cb31-2"><a href="https://doisinkidney.com/rss.xml#cb31-2" tabindex="-1"/>  <span class="fu">pure</span> <span class="ot">=</span> <span class="dt">Stable</span> <span class="dv">0</span> <span class="op">.</span> <span class="fu">pure</span></span>
<span id="cb31-3"><a href="https://doisinkidney.com/rss.xml#cb31-3" tabindex="-1"/>  <span class="dt">Stable</span> n xs <span class="op">&lt;*&gt;</span> <span class="dt">Stable</span> m ys <span class="ot">=</span> <span class="dt">Stable</span> (n<span class="op">+</span>m) (xs <span class="op">&lt;*&gt;</span> delayKey n ys)</span>
<span id="cb31-4"><a href="https://doisinkidney.com/rss.xml#cb31-4" tabindex="-1"/></span>
<span id="cb31-5"><a href="https://doisinkidney.com/rss.xml#cb31-5" tabindex="-1"/><span class="ot">runStable ::</span> (<span class="dt">Applicative</span> f, <span class="dt">Ord</span> k) <span class="ot">=&gt;</span> <span class="dt">Stable</span> k f a <span class="ot">-&gt;</span> f a</span>
<span id="cb31-6"><a href="https://doisinkidney.com/rss.xml#cb31-6" tabindex="-1"/>runStable <span class="ot">=</span> runHeap <span class="op">.</span> heap</span>
<span id="cb31-7"><a href="https://doisinkidney.com/rss.xml#cb31-7" tabindex="-1"/></span>
<span id="cb31-8"><a href="https://doisinkidney.com/rss.xml#cb31-8" tabindex="-1"/><span class="ot">stable ::</span> <span class="dt">Ord</span> k <span class="ot">=&gt;</span> k <span class="ot">-&gt;</span> f a <span class="ot">-&gt;</span> <span class="dt">Stable</span> k f a</span>
<span id="cb31-9"><a href="https://doisinkidney.com/rss.xml#cb31-9" tabindex="-1"/>stable k fa <span class="ot">=</span> <span class="dt">Stable</span> <span class="dv">1</span> (phase (k <span class="op">:*</span> <span class="dv">0</span>) fa)</span></code></pre></div>
<p>This is a pure, optimally efficient implementation of <code class="sourceCode haskell"><span class="dt">Phases</span></code> ordered
by an arbitrary total-ordered key.</p>
<h1 id="local-computation-in-a-monadic-heap">Local Computation in a
Monadic Heap</h1>
<p>In <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-kidney_algebras_2021">2021</a>)</span>,
I developed a monadic heap based on the free monad transformer.</p>
<div class="sourceCode" id="cb32"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb32-1"><a href="https://doisinkidney.com/rss.xml#cb32-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">Search</span> k a <span class="ot">=</span> <span class="dt">Search</span> {<span class="ot"> runSearch ::</span> [<span class="dt">Either</span> a (k, <span class="dt">Search</span> k a)] }</span></code></pre></div>
<p>This type is equivalent to the <a href="https://hackage.haskell.org/package/free-5.2/docs/Control-Monad-Trans-Free.html#t:FreeT">free
monad transformer</a> over the list monad and <code class="sourceCode haskell">(,) k</code> functor (i.e.Â the writer
monad).</p>
<div class="sourceCode" id="cb33"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb33-1"><a href="https://doisinkidney.com/rss.xml#cb33-1" tabindex="-1"/><span class="dt">Search</span> k a â‰… <span class="dt">FreeT</span> ((,) k) [] a</span></code></pre></div>
<p>In the paper <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-kidney_algebras_2021">2021</a>)</span> we extended the type to become a
full monad transformer, replacing lists with <code class="sourceCode haskell"><span class="dt">ListT</span></code>. This
let us order the effects according to the weight <code class="sourceCode haskell">k</code>; however, for this example we only
need the simplified type, which lets us order the values according to
<code class="sourceCode haskell">k</code>.</p>
<p>This <code class="sourceCode haskell"><span class="dt">Search</span></code> type
follows the structure of a pairing heap (although not as closely as the
version above). However, this type is interesting because
<em>semantically</em> it needs the weights to be stored as differences,
rather than absolute weights. As a free monad transformer, the <code class="sourceCode haskell"><span class="dt">Search</span></code> type
layers effects on top of each other; we can later interpret those layers
by collapsing them together using the monadic <code class="sourceCode haskell">join</code>. In the case of <code class="sourceCode haskell"><span class="dt">Search</span></code>, those
layers are drawn from the list monad and the <code class="sourceCode haskell">(,) k</code> functor (writer monad). That
means that if we have some heap representing the tree from above:</p>
<div class="sourceCode" id="cb34"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb34-1"><a href="https://doisinkidney.com/rss.xml#cb34-1" tabindex="-1"/><span class="dt">Search</span> [ <span class="dt">Right</span> (a, <span class="dt">Search</span> [ <span class="dt">Right</span> (b, <span class="dt">Search</span> [ <span class="dt">Right</span> (d, <span class="dt">Search</span> [<span class="dt">Left</span> x])</span>
<span id="cb34-2"><a href="https://doisinkidney.com/rss.xml#cb34-2" tabindex="-1"/>                                             , <span class="dt">Right</span> (e, <span class="dt">Search</span> [<span class="dt">Left</span> y])])</span>
<span id="cb34-3"><a href="https://doisinkidney.com/rss.xml#cb34-3" tabindex="-1"/>                          , <span class="dt">Right</span> (c, <span class="dt">Search</span> [<span class="dt">Left</span> z])])]</span></code></pre></div>
<p>When we collapse this computation down to the leaves, the weights we
will get are the following:</p>
<div class="sourceCode" id="cb35"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb35-1"><a href="https://doisinkidney.com/rss.xml#cb35-1" tabindex="-1"/>[(a <span class="op">&lt;&gt;</span> b <span class="op">&lt;&gt;</span> d, x), (a <span class="op">&lt;&gt;</span> b <span class="op">&lt;&gt;</span> e, y), (a <span class="op">&lt;&gt;</span> c, z)]</span></code></pre></div>
<p>So, if we want the weights to line up properly, we need to store the
differences.</p>
<div class="sourceCode" id="cb36"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb36-1"><a href="https://doisinkidney.com/rss.xml#cb36-1" tabindex="-1"/><span class="ot">mergeS ::</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> [(k, <span class="dt">Search</span> k a)] <span class="ot">-&gt;</span> <span class="dt">Maybe</span> (k, <span class="dt">Search</span> k a)</span>
<span id="cb36-2"><a href="https://doisinkidney.com/rss.xml#cb36-2" tabindex="-1"/>mergeS [] <span class="ot">=</span> <span class="dt">Nothing</span></span>
<span id="cb36-3"><a href="https://doisinkidney.com/rss.xml#cb36-3" tabindex="-1"/>mergeS (x<span class="op">:</span>xs) <span class="ot">=</span> <span class="dt">Just</span> (mergeS' x xs)</span>
<span id="cb36-4"><a href="https://doisinkidney.com/rss.xml#cb36-4" tabindex="-1"/>  <span class="kw">where</span></span>
<span id="cb36-5"><a href="https://doisinkidney.com/rss.xml#cb36-5" tabindex="-1"/>    mergeS' x1 [] <span class="ot">=</span> x1</span>
<span id="cb36-6"><a href="https://doisinkidney.com/rss.xml#cb36-6" tabindex="-1"/>    mergeS' x1 [x2] <span class="ot">=</span> x1 <span class="op">&lt;+&gt;</span> x2</span>
<span id="cb36-7"><a href="https://doisinkidney.com/rss.xml#cb36-7" tabindex="-1"/>    mergeS' x1 (x2<span class="op">:</span>x3<span class="op">:</span>xs) <span class="ot">=</span> (x1 <span class="op">&lt;+&gt;</span> x2) <span class="op">&lt;+&gt;</span> mergeS' x3 xs</span>
<span id="cb36-8"><a href="https://doisinkidney.com/rss.xml#cb36-8" tabindex="-1"/></span>
<span id="cb36-9"><a href="https://doisinkidney.com/rss.xml#cb36-9" tabindex="-1"/>    (xw, <span class="dt">Search</span> xs) <span class="op">&lt;+&gt;</span> (yw, <span class="dt">Search</span> ys)</span>
<span id="cb36-10"><a href="https://doisinkidney.com/rss.xml#cb36-10" tabindex="-1"/>      <span class="op">|</span> xw <span class="op">&lt;=</span> yw  <span class="ot">=</span> (xw, <span class="dt">Search</span> (<span class="dt">Right</span> (yw âˆ¸ xw, <span class="dt">Search</span> ys) <span class="op">:</span> xs))</span>
<span id="cb36-11"><a href="https://doisinkidney.com/rss.xml#cb36-11" tabindex="-1"/>      <span class="op">|</span> <span class="fu">otherwise</span> <span class="ot">=</span> (yw, <span class="dt">Search</span> (<span class="dt">Right</span> (xw âˆ¸ yw, <span class="dt">Search</span> xs) <span class="op">:</span> ys))</span>
<span id="cb36-12"><a href="https://doisinkidney.com/rss.xml#cb36-12" tabindex="-1"/></span>
<span id="cb36-13"><a href="https://doisinkidney.com/rss.xml#cb36-13" tabindex="-1"/><span class="ot">popMins ::</span> <span class="dt">Monus</span> k <span class="ot">=&gt;</span> <span class="dt">Search</span> k a <span class="ot">-&gt;</span> ([a], <span class="dt">Maybe</span> (k, <span class="dt">Search</span> k a))</span>
<span id="cb36-14"><a href="https://doisinkidney.com/rss.xml#cb36-14" tabindex="-1"/>popMins <span class="ot">=</span> <span class="fu">fmap</span> mergeS <span class="op">.</span> partitionEithers <span class="op">.</span> runSearch</span></code></pre></div>
<h1 id="conclusion">Conclusion</h1>
<p>The technique of â€œdonâ€™t store the absolute value, store the
differenceâ€� seems to be generally quite useful; I think that monuses are
a handy algebra to keep in mind whenever that technique looks like it
might be needed. The <code class="sourceCode haskell"><span class="dt">Key</span></code> monus
above is closely related to the factorial numbers, and the trick I used
in <a href="https://doisinkidney.com/2019-03-24-permutations-by-sorting.html">this</a> post.</p>
<hr/>
<h2 class="unnumbered" id="references">References</h2>
<div class="references csl-bib-body hanging-indent" id="refs">
<div class="csl-entry" id="ref-amer_equationally_1984">
Amer, K. 1984. <span>â€œEquationally complete classes of commutative
monoids with monus.â€�</span> <em>algebra universalis</em> 18 (1)
(February): 129â€“131. doi:<a href="https://doi.org/10.1007/BF01182254">10.1007/BF01182254</a>.
</div>
<div class="csl-entry" id="ref-blondal_generalized_2025">
BlÃ¶ndal, Baldur. 2025a. <span>â€œGeneralized multi-phase
compiler/concurrency.â€�</span> <em>reddit</em>. <a href="https://www.reddit.com/r/haskell/comments/1m25fw8/generalized_multiphase_compilerconcurrency/">https://www.reddit.com/r/haskell/comments/1m25fw8/generalized_multiphase_compilerconcurrency/</a>.
</div>
<div class="csl-entry" id="ref-blondal_phases_2025">
â€”â€”â€”. 2025b. <span>â€œPhases using <span>Vault</span>.â€�</span>
<em>reddit</em>. <a href="https://www.reddit.com/r/haskell/comments/1msvwzd/phases_using_vault/">https://www.reddit.com/r/haskell/comments/1msvwzd/phases_using_vault/</a>.
</div>
<div class="csl-entry" id="ref-easterly_functions_2019">
Easterly, Noah. 2019. <span>â€œFunctions and newtype wrappers for
traversing <span>Trees</span>: Rampion/tree-traversals.â€�</span> <a href="https://github.com/rampion/tree-traversals">https://github.com/rampion/tree-traversals</a>.
</div>
<div class="csl-entry" id="ref-fredman_pairing_1986">
Fredman, Michael L., Robert Sedgewick, Daniel D. Sleator, and Robert E.
Tarjan. 1986. <span>â€œThe pairing heap: <span>A</span> new form of
self-adjusting heap.â€�</span> <em>Algorithmica</em> 1 (1-4) (January):
111â€“129. doi:<a href="https://doi.org/10.1007/BF01840439">10.1007/BF01840439</a>.
</div>
<div class="csl-entry" id="ref-gibbons_breadthfirst_2022">
Gibbons, Jeremy, Donnacha OisÃ­n Kidney, Tom Schrijvers, and Nicolas Wu.
2022. <span>â€œBreadth-<span>First Traversal</span>
viaÂ <span>Staging</span>.â€�</span> In <em>Mathematics of <span>Program
Construction</span></em>, ed by. Ekaterina Komendantskaya, 1â€“33. Cham:
Springer International Publishing. doi:<a href="https://doi.org/10.1007/978-3-031-16912-0_1">10.1007/978-3-031-16912-0_1</a>.
</div>
<div class="csl-entry" id="ref-gibbons_phases_2023">
â€”â€”â€”. 2023. <span>â€œPhases in <span>Software Architecture</span>.â€�</span>
In <em>Proceedings of the 1st <span>ACM SIGPLAN International
Workshop</span> on <span>Functional Software Architecture</span></em>,
29â€“33. <span>FUNARCH</span> 2023. New York, NY, USA: Association for
Computing Machinery. doi:<a href="https://doi.org/10.1145/3609025.3609479">10.1145/3609025.3609479</a>.
</div>
<div class="csl-entry" id="ref-kidney_algebras_2021">
Kidney, Donnacha OisÃ­n, and Nicolas Wu. 2021. <span>â€œAlgebras for
weighted search.â€�</span> <em>Proceedings of the ACM on Programming
Languages</em> 5 (ICFP) (August): 72:1â€“72:30. doi:<a href="https://doi.org/10.1145/3473577">10.1145/3473577</a>.
</div>
<div class="csl-entry" id="ref-kidney_formalising_2025">
â€”â€”â€”. 2025. <span>â€œFormalising <span>Graph Algorithms</span> with
<span>Coinduction</span>.â€�</span> <em>Proc. ACM Program. Lang.</em> 9
(POPL) (January): 56:1657â€“56:1686. doi:<a href="https://doi.org/10.1145/3704892">10.1145/3704892</a>.
</div>
<div class="csl-entry" id="ref-larkin_backtobasics_2013">
Larkin, Daniel H., Siddhartha Sen, and Robert E. Tarjan. 2013. <span>â€œA
<span class="nocase">Back-to-Basics Empirical Study</span> of
<span>Priority Queues</span>.â€�</span> In <em>2014
<span>Proceedings</span> of the <span>Meeting</span> on <span>Algorithm
Engineering</span> and <span>Experiments</span>
(<span>ALENEX</span>)</em>, 61â€“72. Proceedings. <span>Society for
Industrial and Applied Mathematics</span>. doi:<a href="https://doi.org/10.1137/1.9781611973198.7">10.1137/1.9781611973198.7</a>.
</div>
<div class="csl-entry" id="ref-visscher_phases_2025">
Visscher, Sjoerd. 2025. <span>â€œPhases with any <span>Ord</span> key
type.â€�</span> <a href="https://gist.github.com/sjoerdvisscher/bf282a050f0681e2f737908e254c4061">https://gist.github.com/sjoerdvisscher/bf282a050f0681e2f737908e254c4061</a>.
</div>
</div>
<section class="footnotes footnotes-end-of-document" id="footnotes">
<hr/>
<ol>
<li id="fn1"><p>Note that there are many related structures that all
fall under the umbrella notion of â€œmonusâ€�; the structure that I am
defining here is the same structure I worked with in <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-kidney_algebras_2021">2021</a>)</span>
and <span class="citation">(<a href="https://doisinkidney.com/rss.xml#ref-kidney_formalising_2025">2025</a>)</span>.<a class="footnote-back" href="https://doisinkidney.com/rss.xml#fnref1">â†©ï¸�</a></p></li>
</ol>
</section></div>
    </summary>
    <updated>2026-03-03T00:00:00Z</updated>
    <published>2026-03-03T00:00:00Z</published>
    <author>
      <name>Donnacha Oisín Kidney</name>
    </author>
    <source>
      <id>https://doisinkidney.com</id>
      <link href="https://doisinkidney.com" rel="alternate" type="text/html"/>
      <link href="https://doisinkidney.com/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Mainly writing about programming</subtitle>
      <title>Donnacha Oisín Kidney's Blog</title>
      <updated>2026-03-03T00:00:00Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>http://readerunner.wordpress.com/?p=251</id>
    <link href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/" rel="alternate" type="text/html"/>
    <link href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#comments" rel="replies" type="text/html"/>
    <link href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/feed/atom/" rel="replies" type="application/atom+xml"/>
    <title xml:lang="en">PenroseKiteDart User Guide</title>
    <summary xml:lang="en">Introduction (Updated February 2026 for PenroseKiteDart version 1.6.1) PenroseKiteDart is a Haskell package with tools to experiment with finite tilings of Penrose’s Kites and Darts. It uses the Haskell Diagrams package for drawing tilings. As well as providing drawing tools, this package introduces tile graphs (Tgraphs) for describing finite tilings. (I would like to thank […]</summary>
    <content type="xhtml" xml:lang="en"><div xmlns="http://www.w3.org/1999/xhtml"><h2 id="introduction">Introduction</h2>
<p><strong>(Updated February 2026 for PenroseKiteDart version 1.6.1)</strong></p>
<p>PenroseKiteDart is a Haskell package with tools to experiment with finite tilings of Penrose’s Kites and Darts. It uses the <a href="https://diagrams.github.io">Haskell Diagrams</a> package for drawing tilings. As well as providing drawing tools, this package introduces tile graphs (<code>Tgraphs</code>) for describing finite tilings. (I would like to thank Stephen Huggett for suggesting planar graphs as a way to reperesent the tilings).</p>
<p>This document summarises the design and use of the PenroseKiteDart package.</p>
<p>PenroseKiteDart package is now available on <a href="https://hackage.haskell.org">Hackage</a>.</p>
<p>The source files are available on GitHub at <a href="https://github.com/chrisreade/PenroseKiteDart">https://github.com/chrisreade/PenroseKiteDart</a>.</p>
<p>There is a small art gallery of examples created with PenroseKiteDart <a href="https://github.com/chrisreade/PenroseKiteDart/tree/master/ArtGallery">here</a>.</p>
<p><strong>Index</strong></p>
<ol type="1">
<li><a href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#1">About Penrose’s Kites and Darts</a></li>
<li><a href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#2">Using the PenroseKiteDart Package</a> (initial set up).</li>
<li><a href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#3">Overview of Types and Operations</a></li>
<li><a href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#4">Drawing in more detail</a></li>
<li><a href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#5">Forcing in more detail</a></li>
<li><a href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#6">Advanced Operations</a></li>
<li><a href="https://readerunner.wordpress.com/2024/04/08/penrosekitedart-user-guides/#7">Other Reading</a></li>
</ol>
<p><a name="1"> </a></p>
<h2 id="about-penroses-kites-and-darts">1. About Penroseâ€™s Kites and Darts</h2>
<h3 id="the-tiles">The Tiles</h3>
<p>In figure 1 we show a dart and a kite. All angles are multiples of <img alt="36^{\circ}" class="latex" src="https://s0.wp.com/latex.php?latex=36%5E%7B%5Ccirc%7D&amp;bg=ffffff&amp;fg=444444&amp;s=0&amp;c=20201002"/> (a tenth of a full turn). If the shorter edges are of length 1, then the longer edges are of length <img alt="\phi" class="latex" src="https://s0.wp.com/latex.php?latex=%5Cphi&amp;bg=ffffff&amp;fg=444444&amp;s=0&amp;c=20201002"/>, where <img alt="\phi = (1+ \sqrt{5})/ 2" class="latex" src="https://s0.wp.com/latex.php?latex=%5Cphi+%3D+%281%2B+%5Csqrt%7B5%7D%29%2F+2&amp;bg=ffffff&amp;fg=444444&amp;s=0&amp;c=20201002"/> is the golden ratio.</p>
<figure>
<img alt="Figure 1: The Dart and Kite Tiles" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/geomtiles.png?w=625"/><figcaption>Figure 1: The Dart and Kite Tiles</figcaption></figure>
<h3 id="aperiodic-infinite-tilings">Aperiodic Infinite Tilings</h3>
<p>What is interesting about these tiles is:</p>
<p><em>It is possible to tile the entire plane with kites and darts in an aperiodic way.</em></p>
<p>Such a tiling is non-periodic and does not contain arbitrarily large periodic regions or patches.</p>
<p>The possibility of aperiodic tilings with kites and darts was discovered by Sir Roger Penrose in 1974. There are other shapes with this property, including a chiral aperiodic monotile discovered in 2023 by Smith, Myers, Kaplan, Goodman-Strauss. (See the Penrose Tiling <a href="https://en.wikipedia.org/wiki/Penrose_tiling">Wikipedia page</a> for the history of aperiodic tilings)</p>
<p>This package is entirely concerned with Penrose’s kite and dart tilings also known as P2 tilings.</p>
<h3 id="legal-tilings">Legal Tilings</h3>
<p>In figure 2 we add a temporary green line marking purely to illustrate a rule for making <em>legal tilings</em>. The purpose of the rule is to exclude the possibility of periodic tilings.</p>
<p>If all tiles are marked as shown, then whenever tiles come together at a point, they must all be marked or must all be unmarked at that meeting point. So, for example, each long edge of a kite can be placed legally on only <em>one</em> of the two long edges of a dart. The kite wing vertex (which is marked) has to go next to the dart tip vertex (which is marked) and cannot go next to the dart wing vertex (which is unmarked) for a legal tiling.</p>
<figure>
<img alt="Figure 2: Marked Dart and Kite" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/markedtiles2.png?w=625"/><figcaption>Figure 2: Marked Dart and Kite</figcaption></figure>
<h3 id="correct-tilings">Correct Tilings</h3>
<p>Unfortunately, having a finite legal tiling is not enough to guarantee you can continue the tiling without getting stuck. Finite legal tilings which can be continued to cover the entire plane are called <em>correct</em> and the others (which are doomed to get stuck) are called <em>incorrect</em>. This means that decomposition and forcing (described later) become important tools for constructing correct finite tilings.</p>
<p><a name="2"> </a></p>
<h2 id="using-the-penrosekitedart-package">2. Using the PenroseKiteDart Package</h2>
<p>You will need the Haskell Diagrams package (See <a href="https://diagrams.github.io">Haskell Diagrams</a>) as well as this package (PenroseKiteDart). When these are installed, you can produce diagrams with a Main.hs module. This should import a chosen backend for diagrams such as the default (SVG) along with <code>Diagrams.Prelude</code>.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">module</span> <span>Main</span> <span style="color: red;">(</span><span>main</span><span style="color: red;">)</span> <span style="color: blue; font-weight: bold;">where</span>
    
    <span style="color: blue; font-weight: bold;">import</span> <span>Diagrams.Backend.SVG.CmdLine</span>
    <span style="color: blue; font-weight: bold;">import</span> <span>Diagrams.Prelude</span></code></pre>
<p>For Penrose’s Kite and Dart tilings, you also need to import the <code>PKD</code> module and (optionally) the <code>TgraphExamples</code> module.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">import</span> <span>PKD</span>
    <span style="color: blue; font-weight: bold;">import</span> <span>TgraphExamples</span></code></pre>
<p>Then to ouput <code>someExample</code> figure</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>fig</span><span style="color: red;">::</span><span>Diagram</span> <span>B</span>
    <span>fig</span> <span style="color: red;">=</span> <span>someExample</span>

    <span>main</span> <span style="color: red;">::</span> <span>IO</span> <span>()</span>
    <span>main</span> <span style="color: red;">=</span> <span>mainWith</span> <span>fig</span></code></pre>
<p>Note that the token <code>B</code> is used in the diagrams package to represent the chosen backend for output. So a diagram has type <code>Diagram B</code>. In this case <code>B</code> is bound to SVG by the import of the SVG backend. When the compiled module is executed it will generate an SVG file. (See <a href="https://diagrams.github.io">Haskell Diagrams</a> for more details on producing diagrams and using alternative backends).</p>
<p><a name="3"> </a></p>
<h2 id="overview-of-types-and-operations">3. Overview of Types and Operations</h2>
<h3 id="half-tiles">Half-Tiles</h3>
<p>In order to implement operations on tilings (<code>decompose</code> in particular), we work with half-tiles. These are illustrated in figure 3 and labelled <code>RD</code> (right dart), <code>LD</code> (left dart), <code>LK</code> (left kite), <code>RK</code> (right kite). The <em>join</em> edges where left and right halves come together are shown with dotted lines, leaving one short edge and one long edge on each half-tile (excluding the join edge). We have shown a red dot at the vertex we regard as the origin of each half-tile (the tip of a half-dart and the base of a half-kite).</p>
<figure>
<img alt="Figure 3: Half-Tile pieces showing join edges (dashed) and origin vertices (red dots)" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/newpiecesfig.png?w=625"/><figcaption>Figure 3: Half-Tile pieces showing join edges (dashed) and origin vertices (red dots)</figcaption></figure>
<p>The labels are actually data constructors introduced with type operator <code>HalfTile</code> which has an argument type (<code>rep</code>) to allow for more than one representation of the half-tiles.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">data</span> <span>HalfTile</span> <span>rep</span> 
      <span style="color: red;">=</span> <span>LD</span> <span>rep</span> <span style="color: green;">-- Left Dart</span>
      <span style="color: red;">|</span> <span>RD</span> <span>rep</span> <span style="color: green;">-- Right Dart</span>
      <span style="color: red;">|</span> <span>LK</span> <span>rep</span> <span style="color: green;">-- Left Kite</span>
      <span style="color: red;">|</span> <span>RK</span> <span>rep</span> <span style="color: green;">-- Right Kite</span>
      <span style="color: blue; font-weight: bold;">deriving</span> <span style="color: red;">(</span><span>Show</span><span style="color: red;">,</span><span>Eq</span><span style="color: red;">)</span></code></pre>
<h3 id="tgraphs">Tgraphs</h3>
<p>We introduce tile graphs (<code>Tgraph</code>s) which provide a simple planar graph representation for finite patches of tiles. For <code>Tgraph</code>s we first specialise <code>HalfTile</code> with a triple of vertices (positive integers) to make a <code>TileFace</code> such as <code>RD(1,2,3)</code>, where the vertices go clockwise round the half-tile triangle starting with the origin.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">type</span> <span>TileFace</span>  <span style="color: red;">=</span> <span>HalfTile</span> <span style="color: red;">(</span><span>Vertex</span><span style="color: red;">,</span><span>Vertex</span><span style="color: red;">,</span><span>Vertex</span><span style="color: red;">)</span>
    <span style="color: blue; font-weight: bold;">type</span> <span>Vertex</span>    <span style="color: red;">=</span> <span>Int</span>  <span style="color: green;">-- must be positive</span></code></pre>
<p>The function</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>makeTgraph</span> <span style="color: red;">::</span> <span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span></code></pre>
<p>then constructs a <code>Tgraph</code> from a <code>TileFace</code> list after checking the <code>TileFace</code>s satisfy certain properties (described below). We also have</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>faces</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span></code></pre>
<p>to retrieve the <code>TileFace</code> list from a <code>Tgraph</code>.</p>
<p>As an example, the <code>fool</code> (short for <em>fool’s kite</em> and also called an <em>ace</em> in the literature) consists of two kites and a dart (= 4 half-kites and 2 half-darts):</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>fool</span> <span style="color: red;">::</span> <span>Tgraph</span>
    <span>fool</span> <span style="color: red;">=</span> <span>makeTgraph</span> <span style="color: red;">[</span><span>RD</span> <span style="color: red;">(</span><span class="hs-num">1</span><span style="color: red;">,</span><span class="hs-num">2</span><span style="color: red;">,</span><span class="hs-num">3</span><span style="color: red;">)</span><span style="color: red;">,</span> <span>LD</span> <span style="color: red;">(</span><span class="hs-num">1</span><span style="color: red;">,</span><span class="hs-num">3</span><span style="color: red;">,</span><span class="hs-num">4</span><span style="color: red;">)</span>   <span style="color: green;">-- right and left dart</span>
                      <span style="color: red;">,</span><span>LK</span> <span style="color: red;">(</span><span class="hs-num">5</span><span style="color: red;">,</span><span class="hs-num">3</span><span style="color: red;">,</span><span class="hs-num">2</span><span style="color: red;">)</span><span style="color: red;">,</span> <span>RK</span> <span style="color: red;">(</span><span class="hs-num">5</span><span style="color: red;">,</span><span class="hs-num">2</span><span style="color: red;">,</span><span class="hs-num">7</span><span style="color: red;">)</span>   <span style="color: green;">-- left and right kite</span>
                      <span style="color: red;">,</span><span>RK</span> <span style="color: red;">(</span><span class="hs-num">5</span><span style="color: red;">,</span><span class="hs-num">4</span><span style="color: red;">,</span><span class="hs-num">3</span><span style="color: red;">)</span><span style="color: red;">,</span> <span>LK</span> <span style="color: red;">(</span><span class="hs-num">5</span><span style="color: red;">,</span><span class="hs-num">6</span><span style="color: red;">,</span><span class="hs-num">4</span><span style="color: red;">)</span>   <span style="color: green;">-- right and left kite</span>
                      <span style="color: red;">]</span></code></pre>
<p>To produce a diagram, we simply <code>draw</code> the <code>Tgraph</code></p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>foolFigure</span> <span style="color: red;">::</span> <span>Diagram</span> <span>B</span>
    <span>foolFigure</span> <span style="color: red;">=</span> <span>draw</span> <span>fool</span></code></pre>
<p>which will produce the diagram on the left in figure 4.</p>
<p>Alternatively,</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>foolFigure</span> <span style="color: red;">::</span> <span>Diagram</span> <span>B</span>
    <span>foolFigure</span> <span style="color: red;">=</span> <span>labelled</span> <span>drawj</span> <span>fool</span></code></pre>
<p>will produce the diagram on the right in figure 4 (showing vertex labels and dashed join edges).</p>
<figure>
<img alt="Figure 4: Diagram of fool without labels and join edges (left), and with (right)" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/newfool.png?w=625"/><figcaption>Figure 4: Diagram of <code>fool</code> without labels and join edges (left), and with (right)</figcaption></figure>
<p>When any (non-empty) <code>Tgraph</code> is drawn, a default orientation and scale are chosen based on the lowest numbered join edge. This is aligned on the positive x-axis with length 1 (for darts) or length <img alt="\phi" class="latex" src="https://s0.wp.com/latex.php?latex=%5Cphi&amp;bg=ffffff&amp;fg=444444&amp;s=0&amp;c=20201002"/> (for kites).</p>
<h3 id="tgraph-properties">Tgraph Properties</h3>
<p>Tgraphs are actually implemented as</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">newtype</span> <span>Tgraph</span> <span style="color: red;">=</span> <span>Tgraph</span> <span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span>
                     <span style="color: blue; font-weight: bold;">deriving</span> <span style="color: red;">(</span><span>Show</span><span style="color: red;">)</span></code></pre>
<p>but the data constructor <code>Tgraph</code> is not exported to avoid accidentally by-passing checks for the required properties. The properties checked by <code>makeTgraph</code> ensure the <code>Tgraph</code> represents a legal tiling as a planar graph with positive vertex numbers, and that the collection of half-tile faces are both connected and have no crossing boundaries (see note below). Finally, there is a check to ensure two or more distinct vertex numbers are not used to represent the same vertex of the graph (a <em>touching vertex</em> check). An error is raised if there is a problem.</p>
<p>Note: If the <code>TileFace</code>s are faces of a planar graph there will also be exterior (untiled) regions, and in graph theory these would also be called faces of the graph. To avoid confusion, we will refer to these only as <em>exterior regions</em>, and unless otherwise stated, <em>face</em> will mean a <code>TileFace</code>. We can then define the boundary of a list of <code>TileFace</code>s as the edges of the exterior regions. There is a <em>crossing boundary</em> if the boundary crosses itself at a vertex. We exclude crossing boundaries from <code>Tgraph</code>s because they prevent us from calculating relative positions of tiles locally and create touching vertex problems.</p>
<p>For convenience, in addition to <code>makeTgraph</code>, we also have</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>makeUncheckedTgraph</span> <span style="color: red;">::</span> <span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span>
    <span>checkedTgraph</span>   <span style="color: red;">::</span> <span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span></code></pre>
<p>The first of these (performing no checks) is useful when you know the required properties hold. The second performs the same checks as <code>makeTgraph</code> except that it omits the touching vertex check. This could be used, for example, when making a <code>Tgraph</code> from a sub-collection of <code>TileFace</code>s of another <code>Tgraph</code>.</p>
<h3 id="main-tiling-operations">Main Tiling Operations</h3>
<p>There are three key operations on finite tilings, namely</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>decompose</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span>
    <span>force</span>     <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span>
    <span>compose</span>   <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span></code></pre>
<h4 id="decompose">Decompose</h4>
<p>Decomposition (also called <em>deflation</em>) works by splitting each half-tile into either 2 or 3 new (<em>smaller scale</em>) half-tiles, to produce a new tiling. The fact that this is possible, is used to establish the existence of infinite aperiodic tilings with kites and darts. Since our <code>Tgraph</code>s have abstracted away from scale, the result of decomposing a <code>Tgraph</code> is just another <code>Tgraph</code>. However if we wish to compare before and after with a drawing, the latter should be scaled by a factor <img alt="1/{\phi} = \phi - 1" class="latex" src="https://s0.wp.com/latex.php?latex=1%2F%7B%5Cphi%7D+%3D+%5Cphi+-+1&amp;bg=ffffff&amp;fg=444444&amp;s=0&amp;c=20201002"/> times the scale of the former, to reflect the change in scale.</p>
<figure>
<img alt="Figure 5: fool (left) and decompose fool (right)" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/foolandfoold.png?w=625"/><figcaption>Figure 5: <code>fool</code> (left) and <code>decompose fool</code> (right)</figcaption></figure>
<p>We can, of course, iterate <code>decompose</code> to produce an infinite list of finer and finer decompositions of a <code>Tgraph</code></p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>decompositions</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Tgraph</span><span style="color: red;">]</span>
    <span>decompositions</span> <span style="color: red;">=</span> <span>iterate</span> <span>decompose</span></code></pre>
<h4 id="force">Force</h4>
<p>Force works by adding any <code>TileFace</code>s on the boundary edges of a <code>Tgraph</code> which are <em>forced</em>. That is, where there is only one legal choice of <code>TileFace</code> addition consistent with the seven possible vertex types. Such additions are continued until either (i) there are no more forced cases, in which case a final (forced) <code>Tgraph</code> is returned, or (ii) the process finds the tiling is stuck, in which case an error is raised indicating an incorrect tiling. [In the latter case, the argument to <code>force</code> must have been an incorrect tiling, because the forced additions cannot produce an incorrect tiling starting from a correct tiling.]</p>
<p>An example is shown in figure 6. When forced, the <code>Tgraph</code> on the left produces the result on the right. The original is highlighted in red in the result to show what has been added.</p>
<figure>
<img alt="Figure 6: A Tgraph (left) and its forced result (right) with the original shown red" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/fooldandforce.png?w=625"/><figcaption>Figure 6: A Tgraph (left) and its forced result (right) with the original shown red</figcaption></figure>
<h4 id="compose">Compose</h4>
<p>Composition (also called <em>inflation</em>) is an opposite to <code>decompose</code> but this has complications for finite tilings, so it is not simply an inverse. (See <a href="https://readerunner.wordpress.com/2023/09/12/graphs-kites-and-darts-and-theorems/">Graphs,Kites and Darts and Theorems</a> for more discussion of the problems). Figure 7 shows a <code>Tgraph</code> (left) with the result of composing (right) where we have also shown (in pale green) the faces of the original that are not included in the composition – the remainder faces.</p>
<figure>
<img alt="Figure 7: A Tgraph (left) and its (part) composed result (right) with the remainder faces shown pale green" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/pcomposeexample.png?w=625"/><figcaption>Figure 7: A Tgraph (left) and its (part) composed result (right) with the remainder faces shown pale green</figcaption></figure>
<p>Under some circumstances composing can fail to produce a <code>Tgraph</code> because there are crossing boundaries in the resulting <code>TileFaces</code>. However, we have established that</p>
<ul>
<li>If <code>g</code> is a forced <code>Tgraph</code>, then <code>compose g</code> is defined and it is also a forced <code>Tgraph</code>.</li>
</ul>
<h3 id="try-results">Try Results</h3>
<p>It is convenient to use types of the form <code>Try a</code> for results where we know there can be a failure. For example, <code>compose</code> can fail if the result does not pass the connected and no crossing boundary check, and <code>force</code> can fail if its argument is an incorrect <code>Tgraph</code>. In situations when you would like to continue some computation rather than raise an error when there is a failure, use a <em>try</em> version of a function.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>tryCompose</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>Tgraph</span>
    <span>tryForce</span>   <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>Tgraph</span></code></pre>
<p>We define <code>Try</code> as a synonym for <code>Either ShowS</code> (which is a monad) in module <code>Tgraph.Try</code>.</p>
<pre><code>type Try a = Either ShowS a</code></pre>
<p>(Note <code>ShowS</code> is <code>String -&gt; String</code>). Successful results have the form <code>Right r</code> (for some correct result <code>r</code>) and failure results have the form <code>Left (s&lt;&gt;)</code> (where <code>s</code> is a <code>String</code> describing the problem as a failure report).</p>
<p>The function</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>runTry</span><span style="color: red;">::</span> <span>Try</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>runTry</span> <span style="color: red;">=</span> <span>either</span> <span>error</span> <span>id</span></code></pre>
<p>will retrieve a correct result but raise an error for failure cases. This means we can always derive an error raising version from a try version of a function by composing with <code>runTry</code>.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>force</span> <span style="color: red;">=</span> <span>runTry</span> <span>.</span> <span>tryForce</span>
    <span>compose</span> <span style="color: red;">=</span> <span>runTry</span> <span>.</span> <span>tryCompose</span></code></pre>
<h3 id="elementary-tgraph-and-tileface-operations">Elementary Tgraph and TileFace Operations</h3>
<p>The module <code>Tgraph.Prelude</code> defines elementary operations on <code>Tgraph</code>s relating vertices, directed edges, and faces. We describe a few of them here.</p>
<p>When we need to refer to particular vertices of a <code>TileFace</code> we use</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>originV</span> <span style="color: red;">::</span> <span>TileFace</span> <span style="color: red;">-&gt;</span> <span>Vertex</span> <span style="color: green;">-- the first vertex - red dot in figure 2</span>
    <span>oppV</span>    <span style="color: red;">::</span> <span>TileFace</span> <span style="color: red;">-&gt;</span> <span>Vertex</span> <span style="color: green;">-- the vertex at the opposite end of the join edge from the origin</span>
    <span>wingV</span>   <span style="color: red;">::</span> <span>TileFace</span> <span style="color: red;">-&gt;</span> <span>Vertex</span> <span style="color: green;">-- the vertex not on the join edge</span></code></pre>
<p>A directed edge is represented as a pair of vertices.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">type</span> <span>Dedge</span> <span style="color: red;">=</span> <span style="color: red;">(</span><span>Vertex</span><span style="color: red;">,</span><span>Vertex</span><span style="color: red;">)</span></code></pre>
<p>So <code>(a,b)</code> is regarded as a directed edge from a to b.</p>
<p>When we need to refer to particular edges of a <code>TileFace</code> we use</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>joinE</span>  <span style="color: red;">::</span> <span>TileFace</span> <span style="color: red;">-&gt;</span> <span>Dedge</span>  <span style="color: green;">-- shown dotted in figure 2</span>
    <span>shortE</span> <span style="color: red;">::</span> <span>TileFace</span> <span style="color: red;">-&gt;</span> <span>Dedge</span>  <span style="color: green;">-- the non-join short edge</span>
    <span>longE</span>  <span style="color: red;">::</span> <span>TileFace</span> <span style="color: red;">-&gt;</span> <span>Dedge</span>  <span style="color: green;">-- the non-join long edge</span></code></pre>
<p>which are all directed clockwise round the <code>TileFace</code>. In contrast, <code>joinOfTile</code> is always directed away from the origin vertex, so is not clockwise for right darts or for left kites:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>joinOfTile</span><span style="color: red;">::</span> <span>TileFace</span> <span style="color: red;">-&gt;</span> <span>Dedge</span>
    <span>joinOfTile</span> <span>face</span> <span style="color: red;">=</span> <span style="color: red;">(</span><span>originV</span> <span>face</span><span style="color: red;">,</span> <span>oppV</span> <span>face</span><span style="color: red;">)</span></code></pre>
<p>In the special case that a list of directed edges is symmetrically closed [(b,a) is in the list whenever (a,b) is in the list] we can think of this as an edge list rather than just a directed edge list.</p>
<p>For example,</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>internalEdges</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Dedge</span><span style="color: red;">]</span></code></pre>
<p>produces an edge list, whereas</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>boundary</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Dedge</span><span style="color: red;">]</span></code></pre>
<p>produces single directions. Each directed edge in the resulting boundary will have a <code>TileFace</code> on the left and an exterior region on the right. The function</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>dedges</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Dedge</span><span style="color: red;">]</span></code></pre>
<p>produces all the directed edges obtained by going clockwise round each <code>TileFace</code> so not every edge in the list has an inverse in the list.</p>
<p><strong>Note:</strong> There is now a class <code>HasFaces</code> (introduced in version 1.4) which includes instances for both <code>Tgraph</code> and <code>[TileFace]</code> and others. This allows some generalisations. For example</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>faces</span>         <span style="color: red;">::</span> <span>HasFaces</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span>
    <span>internalEdges</span> <span style="color: red;">::</span> <span>HasFaces</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Dedge</span><span style="color: red;">]</span>
    <span>boundary</span>      <span style="color: red;">::</span> <span>HasFaces</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Dedge</span><span style="color: red;">]</span> 
    <span>dedges</span>        <span style="color: red;">::</span> <span>HasFaces</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Dedge</span><span style="color: red;">]</span> 
    <span>nullFaces</span>     <span style="color: red;">::</span> <span>HasFaces</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Bool</span></code></pre>
<h3 id="patches-scaled-and-positioned-tilings">Patches (Scaled and Positioned Tilings)</h3>
<p>Behind the scenes, when a <code>Tgraph</code> is drawn, each <code>TileFace</code> is converted to a <code>Piece</code>. A <code>Piece</code> is another specialisation of <code>HalfTile</code> using a two dimensional vector to indicate the length and direction of the join edge of the half-tile (from the <code>originV</code> to the <code>oppV</code>), thus fixing its scale and orientation. The whole <code>Tgraph</code> then becomes a list of located <code>Piece</code>s called a <code>Patch</code>.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">type</span> <span>Piece</span> <span style="color: red;">=</span> <span>HalfTile</span> <span style="color: red;">(</span><span>V2</span> <span>Double</span><span style="color: red;">)</span>
    <span style="color: blue; font-weight: bold;">type</span> <span>Patch</span> <span style="color: red;">=</span> <span style="color: red;">[</span><span>Located</span> <span>Piece</span><span style="color: red;">]</span></code></pre>
<p><code>Piece</code> drawing functions derive vectors for other edges of a half-tile piece from its join edge vector. In particular (in the <code>TileLib</code> module) we have</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>drawPiece</span> <span style="color: red;">::</span> <span>Piece</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>
    <span>darawjPiece</span> <span style="color: red;">::</span> <span>Piece</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>
    <span>fillPieceDK</span> <span style="color: red;">::</span> <span>Colour</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span>Colour</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span>Piece</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>where the first draws the non-join edges of a <code>Piece</code>, the second does the same but adds a faint dashed line for the join edge, and the third takes two colours – one for darts and one for kites, which are used to fill the piece as well as using <code>drawPiece</code>.</p>
<p><code>Patch</code> is an instance of class <code>Transformable</code> so a <code>Patch</code> can be scaled, rotated, and translated.</p>
<h3 id="vertex-patches">Vertex Patches</h3>
<p>It is useful to have an intermediate form between <code>Tgraph</code>s and <code>Patch</code>es, that contains information about both the location of vertices (as 2D points), and the abstract <code>TileFace</code>s. This allows us to introduce labelled drawing functions (to show the vertex labels) which we then extend to <code>Tgraph</code>s. We call the intermediate form a <code>VPatch</code> (short for Vertex Patch).</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">type</span> <span>VertexLocMap</span> <span style="color: red;">=</span> <span>IntMap.IntMap</span> <span style="color: red;">(</span><span>Point</span> <span>V2</span> <span>Double</span><span style="color: red;">)</span>
    <span style="color: blue; font-weight: bold;">data</span> <span>VPatch</span> <span style="color: red;">=</span> <span>VPatch</span> <span style="color: red;">{</span><span>vLocs</span> <span style="color: red;">::</span> <span>VertexLocMap</span><span style="color: red;">,</span>  <span>vpFaces</span><span style="color: red;">::</span><span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span><span style="color: red;">}</span> <span style="color: blue; font-weight: bold;">deriving</span> <span>Show</span></code></pre>
<p>and</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>makeVP</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>VPatch</span></code></pre>
<p>calculates vertex locations using a default orientation and scale.</p>
<p><code>VPatch</code> is made an instance of class <code>Transformable</code> so a <code>VPatch</code> can also be scaled and rotated.</p>
<p>One essential use of this intermediate form is to be able to draw a <code>Tgraph</code> with labels, rotated but without the labels themselves being rotated. We can simply convert the <code>Tgraph</code> to a <code>VPatch</code>, and rotate that before drawing with labels.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>labelled</span> <span>draw</span> <span style="color: red;">(</span><span>rotate</span> <span>someAngle</span> <span style="color: red;">(</span><span>makeVP</span> <span>g</span><span style="color: red;">)</span><span style="color: red;">)</span></code></pre>
<p>We can also align a <code>VPatch</code> using vertex labels.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>alignXaxis</span> <span style="color: red;">::</span> <span style="color: red;">(</span><span>Vertex</span><span style="color: red;">,</span> <span>Vertex</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>VPatch</span> <span style="color: red;">-&gt;</span> <span>VPatch</span> </code></pre>
<p>So if <code>g</code> is a <code>Tgraph</code> with vertex labels <code>a</code> and <code>b</code> we can align it on the x-axis with <code>a</code> at the origin and <code>b</code> on the positive x-axis (after converting to a <code>VPatch</code>), instead of accepting the default orientation.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>labelled</span> <span>draw</span> <span style="color: red;">(</span><span>alignXaxis</span> <span style="color: red;">(</span><span>a</span><span style="color: red;">,</span><span>b</span><span style="color: red;">)</span> <span style="color: red;">(</span><span>makeVP</span> <span>g</span><span style="color: red;">)</span><span style="color: red;">)</span></code></pre>
<p>Another use of <code>VPatch</code>es is to share the vertex location map when drawing only subsets of the faces (see <em>Overlaid examples</em> in the next section).</p>
<p><a name="4"> </a></p>
<h2 id="drawing-in-more-detail">4. Drawing in More Detail</h2>
<h3 id="class-drawable">Class Drawable</h3>
<p>There is a class <code>Drawable</code> with instances <code>Tgraph</code>, <code>VPatch</code>, <code>Patch</code>. When the token <code>B</code> is in scope standing for a fixed backend then we can assume</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>draw</span>   <span style="color: red;">::</span> <span>Drawable</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>  <span style="color: green;">-- draws non-join edges</span>
    <span>drawj</span>  <span style="color: red;">::</span> <span>Drawable</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>  <span style="color: green;">-- as with draw but also draws dashed join edges</span>
    <span>fillDK</span> <span style="color: red;">::</span> <span>Drawable</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>Colour</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span>Colour</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span> <span style="color: green;">-- fills with colours</span></code></pre>
<p>where <code>fillDK clr1 clr2</code> will fill darts with colour <code>clr1</code> and kites with colour <code>clr2</code> as well as drawing non-join edges.</p>
<p>These are the main drawing tools. However they are actually defined for any suitable backend <code>b</code> so have more general types.</p>
<p>(<em>Update Sept 2024</em>) From version 1.1 onwards of PenroseKiteDart, these are</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>draw</span> <span style="color: red;">::</span>   <span style="color: red;">(</span><span>Drawable</span> <span>a</span><span style="color: red;">,</span> <span>OKBackend</span> <span>b</span><span style="color: red;">)</span> <span style="color: red;">=&gt;</span>
              <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>b</span>
    <span>drawj</span> <span style="color: red;">::</span>  <span style="color: red;">(</span><span>Drawable</span> <span>a</span><span style="color: red;">,</span> <span>OKBackend</span><span style="color: red;">)</span> <span>b</span><span style="color: red;">)</span> <span style="color: red;">=&gt;</span>
              <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>b</span>
    <span>fillDK</span> <span style="color: red;">::</span> <span style="color: red;">(</span><span>Drawable</span> <span>a</span><span style="color: red;">,</span> <span>OKBackend</span> <span>b</span><span style="color: red;">)</span> <span style="color: red;">=&gt;</span>
              <span>Colour</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span>Colour</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>b</span></code></pre>
<p>where the class <code>OKBackend</code> is a check to ensure a backend is suitable for drawing 2D tilings with or without labels.</p>
<p><strong>In these notes we will generally use the simpler description of types using <code>B</code> for a fixed chosen backend for the sake of clarity.</strong></p>
<p>The drawing tools are each defined via the class function <code>drawWith</code> using <code>Piece</code> drawing functions.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">class</span> <span>Drawable</span> <span>a</span> <span style="color: blue; font-weight: bold;">where</span>
        <span>drawWith</span> <span style="color: red;">::</span> <span style="color: red;">(</span><span>Piece</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>
    
    <span>draw</span> <span style="color: red;">=</span> <span>drawWith</span> <span>drawPiece</span>
    <span>drawj</span> <span style="color: red;">=</span> <span>drawWith</span> <span>drawjPiece</span>
    <span>fillDK</span> <span>clr1</span> <span>clr2</span> <span style="color: red;">=</span> <span>drawWith</span> <span style="color: red;">(</span><span>fillPieceDK</span> <span>clr1</span> <span>clr2</span><span style="color: red;">)</span></code></pre>
<p>To design a new drawing function, you only need to implement a function to draw a <code>Piece</code>, (let us call it <code>newPieceDraw</code>)</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>newPieceDraw</span> <span style="color: red;">::</span> <span>Piece</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>This can then be elevated to draw any <code>Drawable</code> (including <code>Tgraph</code>s, <code>VPatch</code>es, and <code>Patch</code>es) by applying the <code>Drawable</code> class function <code>drawWith</code>:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>newDraw</span> <span style="color: red;">::</span> <span>Drawable</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>
    <span>newDraw</span> <span style="color: red;">=</span> <span>drawWith</span> <span>newPieceDraw</span></code></pre>
<h3 id="class-drawablelabelled">Class DrawableLabelled</h3>
<p>Class <code>DrawableLabelled</code> is defined with instances <code>Tgraph</code> and <code>VPatch</code>, but <code>Patch</code> is not an instance (because this does not retain vertex label information).</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">class</span> <span>DrawableLabelled</span> <span>a</span> <span style="color: blue; font-weight: bold;">where</span>
        <span>labelColourSize</span> <span style="color: red;">::</span> <span>Colour</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span>Measure</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>Patch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>So <code>labelColourSize c m</code> modifies a <code>Patch</code> drawing function to add labels (of colour <code>c</code> and size measure <code>m</code>). <code>Measure</code> is defined in Diagrams.Prelude with pre-defined measures <code>tiny</code>, <code>verySmall</code>, <code>small</code>, <code>normal</code>, <code>large</code>, <code>veryLarge</code>, <code>huge</code>. For most of our diagrams of <code>Tgraph</code>s, we use red labels and we also find <code>small</code> is a good default size choice, so we define</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>labelSize</span> <span style="color: red;">::</span> <span>DrawableLabelled</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>Measure</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>Patch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>
    <span>labelSize</span> <span style="color: red;">=</span> <span>labelColourSize</span> <span>red</span>

    <span>labelled</span> <span style="color: red;">::</span> <span>DrawableLabelled</span> <span>a</span> <span style="color: red;">=&gt;</span> <span style="color: red;">(</span><span>Patch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>
    <span>labelled</span> <span style="color: red;">=</span> <span>labelSize</span> <span>small</span></code></pre>
<p>and then <code>labelled draw</code>, <code>labelled drawj</code>, <code>labelled (fillDK clr1 clr2)</code> can all be used on both <code>Tgraph</code>s and <code>VPatch</code>es as well as (for example) <code>labelSize tiny draw</code>, or <code>labelCoulourSize blue normal drawj</code>.</p>
<h3 id="further-drawing-functions">Further drawing functions</h3>
<p>There are a few extra drawing functions built on top of the above ones. The function <code>smart</code> is a modifier to add dashed join edges only when they occur on the boundary of a <code>Tgraph</code></p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>smart</span> <span style="color: red;">::</span> <span style="color: red;">(</span><span>VPatch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>So <code>smart vpdraw g</code> will draw dashed join edges on the boundary of <code>g</code> before applying the drawing function <code>vpdraw</code> to the <code>VPatch</code> for <code>g</code>. For example the following all draw dashed join edges only on the boundary for a <code>Tgraph g</code></p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>smart</span> <span>draw</span> <span>g</span>
    <span>smart</span> <span style="color: red;">(</span><span>labelled</span> <span>draw</span><span style="color: red;">)</span> <span>g</span>
    <span>smart</span> <span style="color: red;">(</span><span>labelSize</span> <span>normal</span> <span>draw</span><span style="color: red;">)</span> <span>g</span></code></pre>
<p>When using labels, the function <code>rotating</code> allows a <code>Tgraph</code> to be drawn rotated without rotating the labels.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>rotating</span> <span style="color: red;">::</span> <span>Angle</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>VPatch</span> <span style="color: red;">-&gt;</span> <span>a</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>rotating</span> <span>angle</span> <span>vpdraw</span> <span style="color: red;">=</span> <span>vpdraw</span> <span>.</span> <span>rotate</span> <span>angle</span> <span>.</span> <span>makeVP</span></code></pre>
<p>So for example,</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>rotating</span> <span style="color: red;">(</span><span class="hs-num">90</span><span>@@</span><span>deg</span><span style="color: red;">)</span> <span style="color: red;">(</span><span>labelled</span> <span>draw</span><span style="color: red;">)</span> <span>g</span></code></pre>
<p>makes sense for a <code>Tgraph g</code>. Of course if there are no labels we can simply use</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>rotate</span> <span style="color: red;">(</span><span class="hs-num">90</span><span>@@</span><span>deg</span><span style="color: red;">)</span> <span style="color: red;">(</span><span>draw</span> <span>g</span><span style="color: red;">)</span></code></pre>
<p>Similarly <code>aligning</code> allows a <code>Tgraph</code> to be aligned on the X-axis using a pair of vertex numbers before drawing.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>aligning</span> <span style="color: red;">::</span> <span style="color: red;">(</span><span>Vertex</span><span style="color: red;">,</span><span>Vertex</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>VPatch</span> <span style="color: red;">-&gt;</span> <span>a</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>aligning</span> <span style="color: red;">(</span><span>a</span><span style="color: red;">,</span><span>b</span><span style="color: red;">)</span> <span>vpdraw</span> <span style="color: red;">=</span> <span>vpdraw</span> <span>.</span> <span>alignXaxis</span> <span style="color: red;">(</span><span>a</span><span style="color: red;">,</span><span>b</span><span style="color: red;">)</span> <span>.</span> <span>makeVP</span></code></pre>
<p>So, for example, if <code>Tgraph g</code> has vertices <code>a</code> and <code>b</code>, both</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>aligning</span> <span style="color: red;">(</span><span>a</span><span style="color: red;">,</span><span>b</span><span style="color: red;">)</span> <span>draw</span> <span>g</span>
    <span>aligning</span> <span style="color: red;">(</span><span>a</span><span style="color: red;">,</span><span>b</span><span style="color: red;">)</span> <span style="color: red;">(</span><span>labelled</span> <span>draw</span><span style="color: red;">)</span> <span>g</span></code></pre>
<p>make sense. Note that the following two examples are <strong>wrong</strong>. Even though they type check, they re-orient <code>g</code> without repositioning the boundary joins.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>smart</span> <span style="color: red;">(</span><span>labelled</span> <span>draw</span> <span>.</span> <span>rotate</span> <span>angle</span><span style="color: red;">)</span> <span>g</span>      <span style="color: green;">-- WRONG</span>
    <span>smart</span> <span style="color: red;">(</span><span>labelled</span> <span>draw</span> <span>.</span> <span>alignXaxis</span> <span style="color: red;">(</span><span>a</span><span style="color: red;">,</span><span>b</span><span style="color: red;">)</span><span style="color: red;">)</span> <span>g</span>  <span style="color: green;">-- WRONG</span></code></pre>
<p>Instead use</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>smartRotating</span> <span>angle</span> <span style="color: red;">(</span><span>labelled</span> <span>draw</span><span style="color: red;">)</span> <span>g</span>
    <span>smartAligning</span> <span style="color: red;">(</span><span>a</span><span style="color: red;">,</span><span>b</span><span style="color: red;">)</span> <span style="color: red;">(</span><span>labelled</span> <span>draw</span><span style="color: red;">)</span> <span>g</span></code></pre>
<p>where</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>smartRotatinge</span> <span style="color: red;">::</span> <span>Angle</span> <span>Double</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>VPatch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span>
    <span>smartAligning</span>  <span style="color: red;">::</span> <span style="color: red;">(</span><span>Vertex</span><span style="color: red;">,</span><span>Vertex</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>VPatch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>are defined using</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>smartOn</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>VPatch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>VPatch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>Here, <code>smartOn g vpdraw vp</code> uses the given <code>vp</code> for drawing boundary joins and drawing faces of <code>g</code> (with <code>vpdraw</code>) rather than converting <code>g</code> to a new <code>VPatch</code>. This assumes <code>vp</code> has locations for vertices in <code>g</code>.</p>
<h3 id="overlaid-examples-location-map-sharing">Overlaid examples (location map sharing)</h3>
<p>The function</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>drawForce</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>will (smart) draw a <code>Tgraph g</code> in red overlaid (using <code>&lt;&gt;</code>) on the result of <code>force g</code> as in figure 6. Similarly</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>drawPCompose</span>  <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>applied to a <code>Tgraph g</code> will draw the result of a partial composition of <code>g</code> as in figure 7. That is a drawing of <code>compose g</code> but overlaid with a drawing of the remainder faces of <code>g</code> shown in pale green.</p>
<p>Both these functions make use of sharing a vertex location map to get correct alignments of overlaid diagrams. In the case of <code>drawForce g</code>, we know that a <code>VPatch</code> for <code>force g</code> will contain all the vertex locations for <code>g</code> since force only adds to a <code>Tgraph</code> (when it succeeds). So when constructing the diagram for <code>g</code> we can use the <code>VPatch</code> created for <code>force g</code> instead of starting afresh. Similarly for <code>drawPCompose g</code> the <code>VPatch</code> for <code>g</code> contains locations for all the vertices of <code>compose g</code> so <code>compose g</code> is drawn using the <code>VPatch</code> for <code>g</code> instead of starting afresh.</p>
<p>The location map sharing is done with</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>subFaces</span> <span style="color: red;">::</span> <span>HasFaces</span> <span>a</span> <span style="color: red;">=&gt;</span> 
                <span>a</span> <span style="color: red;">-&gt;</span> <span>VPatch</span> <span style="color: red;">-&gt;</span> <span>VPatch</span></code></pre>
<p>so that <code>subFaces fcs vp</code> is a <code>VPatch</code> with the same vertex locations as <code>vp</code>, but replacing the faces of <code>vp</code> with <code>fcs</code>. [Of course, this can go wrong if the new faces have vertices not in the domain of the vertex location map so this needs to be used with care. Any errors would only be discovered when a diagram is created.]</p>
<p>For cases where labels are only going to be drawn for certain faces, we need a version of <code>subFaces</code> which also gets rid of vertex locations that are not relevant to the faces. For this situation we have</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>restrictTo</span><span style="color: red;">::</span> <span>HasFaces</span> <span>a</span> <span style="color: red;">=&gt;</span> 
                 <span>a</span> <span style="color: red;">-&gt;</span> <span>VPatch</span> <span style="color: red;">-&gt;</span> <span>VPatch</span></code></pre>
<p>which filters out un-needed vertex locations from the vertex location map. Unlike <code>subFaces</code>, <code>restrictTo</code> checks for missing vertex locations, so <code>restrictTo fcs vp</code> raises an error if a vertex in <code>fcs</code> is missing from the keys of the vertex location map of <code>vp</code>.</p>
<p><a name="5"> </a></p>
<h2 id="forcing-in-more-detail">5. Forcing in More Detail</h2>
<h3 id="the-force-rules">The force rules</h3>
<p>The rules used by our force algorithm are local and derived from the fact that there are seven possible vertex types as depicted in figure 8.</p>
<figure>
<img alt="Figure 8: Seven vertex types" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/verttypesfig.png?w=625"/><figcaption>Figure 8: Seven vertex types</figcaption></figure>
<p>Our rules are shown in figure 9 (omitting mirror symmetric versions). In each case the <code>TileFace</code> shown yellow needs to be added in the presence of the other <code>TileFace</code>s shown.</p>
<figure>
<img alt="Figure 9: Rules for forcing" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/forcerules.png?w=625"/><figcaption>Figure 9: Rules for forcing</figcaption></figure>
<h3 id="main-forcing-operations">Main Forcing Operations</h3>
<p>To make forcing efficient we convert a <code>Tgraph</code> to a <code>BoundaryState</code> to keep track of boundary information of the <code>Tgraph</code>, and then calculate a <code>ForceState</code> which combines the <code>BoundaryState</code> with a record of awaiting boundary edge updates (an update map). Then each face addition is carried out on a <code>ForceState</code>, converting back when all the face additions are complete. It makes sense to apply <code>force</code> (and related functions) to a <code>Tgraph</code>, a <code>BoundaryState</code>, or a <code>ForceState</code>, so we define a class <code>Forcible</code> with instances <code>Tgraph</code>, <code>BoundaryState</code>, and <code>ForceState</code>.</p>
<p>This allows us to define</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>force</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>tryForce</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span></code></pre>
<p>The first will raise an error if a stuck tiling is encountered. The second uses a <code>Try</code> result which produces a <code>Left string</code> for failures and a <code>Right a</code> for successful result <code>a</code>.</p>
<p>There are several other operations related to forcing including</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>stepForce</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>Int</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>tryStepForce</span>  <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>Int</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span>

    <span>addHalfDart</span><span style="color: red;">,</span> <span>addHalfKite</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>Dedge</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>tryAddHalfDart</span><span style="color: red;">,</span> <span>tryAddHalfKite</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>Dedge</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span></code></pre>
<p>The first two force (up to) a given number of steps (=face additions) and the other four add a half dart/kite on a given boundary edge.</p>
<h3 id="update-generators">Update Generators</h3>
<p>An update generator is used to calculate which boundary edges can have a certain update. There is an update generator for each force rule, but also a combined (all update) generator. The force operations mentioned above all use the default all update generator (<code>defaultAllUGen</code>) but there are more general (<em>with</em>) versions that can be passed an update generator of choice. For example</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>forceWith</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>UpdateGenerator</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>tryForceWith</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>UpdateGenerator</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span></code></pre>
<p>In fact we defined</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>force</span> <span style="color: red;">=</span> <span>forceWith</span> <span>defaultAllUGen</span>
    <span>tryForce</span> <span style="color: red;">=</span> <span>tryForceWith</span> <span>defaultAllUGen</span></code></pre>
<p>We can also define</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>wholeTiles</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>wholeTiles</span> <span style="color: red;">=</span> <span>forceWith</span> <span>wholeTileUpdates</span></code></pre>
<p>where <code>wholeTileUpdates</code> is an update generator that just finds boundary join edges to complete whole tiles.</p>
<p>In addition to <code>defaultAllUGen</code> there is also <code>allUGenerator</code> which does the same thing apart from how failures are reported. The reason for keeping both is that they were constructed differently and so are useful for testing.</p>
<p>In fact <code>UpdateGenerator</code>s are functions that take a <code>BoundaryState</code> and a focus (list of boundary directed edges) to produce an update map. Each <code>Update</code> is calculated as either a <code>SafeUpdate</code> (where two of the new face edges are on the existing boundary and no new vertex is needed) or an <code>UnsafeUpdate</code> (where only one edge of the new face is on the boundary and a new vertex needs to be created for a new face).</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">type</span> <span>UpdateGenerator</span> <span style="color: red;">=</span> <span>BoundaryState</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Dedge</span><span style="color: red;">]</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>UpdateMap</span>
    <span style="color: blue; font-weight: bold;">type</span> <span>UpdateMap</span> <span style="color: red;">=</span> <span>Map.Map</span> <span>Dedge</span> <span>Update</span>
    <span style="color: blue; font-weight: bold;">data</span> <span>Update</span> <span style="color: red;">=</span> <span>SafeUpdate</span> <span>TileFace</span> 
                <span style="color: red;">|</span> <span>UnsafeUpdate</span> <span style="color: red;">(</span><span>Vertex</span> <span style="color: red;">-&gt;</span> <span>TileFace</span><span style="color: red;">)</span></code></pre>
<p>Completing (executing) an <code>UnsafeUpdate</code> requires a touching vertex check to ensure that the new vertex does not clash with an existing boundary vertex. Using an existing (touching) vertex would create a crossing boundary so such an update has to be blocked.</p>
<h3 id="forcible-class-operations">Forcible Class Operations</h3>
<p>The <code>Forcible</code> class operations are higher order and designed to allow for easy additions of further generic operations. They take care of conversions between <code>Tgraph</code>s, <code>BoundaryState</code>s and <code>ForceState</code>s.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">class</span> <span>Forcible</span> <span>a</span> <span style="color: blue; font-weight: bold;">where</span>
      <span>tryFSOpWith</span> <span style="color: red;">::</span> <span>UpdateGenerator</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>ForceState</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>ForceState</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span>
      <span>tryChangeBoundaryWith</span> <span style="color: red;">::</span> <span>UpdateGenerator</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>BoundaryState</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>BoundaryChange</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span>
      <span>tryInitFSWith</span> <span style="color: red;">::</span> <span>UpdateGenerator</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>ForceState</span></code></pre>
<p>For example, given an update generator <code>ugen</code> and any <code>f:: ForceState -&gt; Try ForceState</code> , then <code>f</code> can be generalised to work on any <code>Forcible</code> using <code>tryFSOpWith ugen f</code>. This is used to define both <code>tryForceWith</code> and <code>tryStepForceWith</code>.</p>
<p>We also specialize <code>tryFSOpWith</code> to use the default update generator</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>tryFSOp</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span style="color: red;">(</span><span>ForceState</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>ForceState</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span>
    <span>tryFSOp</span> <span style="color: red;">=</span> <span>tryFSOpWith</span> <span>defaultAllUGen</span></code></pre>
<p>Similarly given an update generator <code>ugen</code> and any <code>f:: BoundaryState -&gt; Try BoundaryChange</code> , then <code>f</code> can be generalised to work on any <code>Forcible</code> using <code>tryChangeBoundaryWith ugen f</code>. This is used to define <code>tryAddHalfDart</code> and <code>tryAddHalfKite</code>.</p>
<p>We also specialize <code>tryChangeBoundaryWith</code> to use the default update generator</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>tryChangeBoundary</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span style="color: red;">(</span><span>BoundaryState</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>BoundaryChange</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span>
    <span>tryChangeBoundary</span> <span style="color: red;">=</span> <span>tryChangeBoundaryWith</span> <span>defaultAllUGen</span></code></pre>
<p>Note that the type <code>BoundaryChange</code> contains a resulting <code>BoundaryState</code>, the single <code>TileFace</code> that has been added, a list of edges removed from the boundary (of the <code>BoundaryState</code> prior to the face addition), and a list of the (3 or 4) boundary edges affected around the change that require checking or re-checking for updates.</p>
<p>The class function <code>tryInitFSWith</code> will use an update generator to create an initial <code>ForceState</code> for any <code>Forcible</code>. If the <code>Forcible</code> is already a <code>ForceState</code> it will do nothing. Otherwise it will calculate updates for the whole boundary. We also have the special case</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>tryInitFS</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>ForceState</span>
    <span>tryInitFS</span> <span style="color: red;">=</span> <span>tryInitFSWith</span> <span>defaultAllUGen</span></code></pre>
<h3 id="efficient-chains-of-forcing-operations.">Efficient chains of forcing operations.</h3>
<p>Note that <code>(force . force)</code> does the same as <code>force</code>, but we might want to chain other <code>force</code> related steps in a calculation.</p>
<p>For example, consider the following combination which, after decomposing a <code>Tgraph</code>, forces, then adds a half dart on a given boundary edge (<code>d</code>) and then forces again.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>combo</span> <span style="color: red;">::</span> <span>Dedge</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span>
    <span>combo</span> <span>d</span> <span style="color: red;">=</span> <span>force</span> <span>.</span> <span>addHalfDart</span> <span>d</span> <span>.</span> <span>force</span> <span>.</span> <span>decompose</span></code></pre>
<p>Since <code>decompose:: Tgraph -&gt; Tgraph</code>, the instances of <code>force</code> and <code>addHalfDart d</code> will have type <code>Tgraph -&gt; Tgraph</code> so each of these operations, will begin and end with conversions between <code>Tgraph</code> and <code>ForceState</code>. We would do better to avoid these wasted intermediate conversions working only with <code>ForceState</code>s and keeping only those necessary conversions at the beginning and end of the whole sequence.</p>
<p>This can be done using <code>tryFSOp</code>. To see this, let us first re-express the forcing sequence using the <code>Try</code> monad, so</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>force</span> <span>.</span> <span>addHalfDart</span> <span>d</span> <span>.</span> <span>force</span></code></pre>
<p>becomes</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>tryForce</span> <span>&lt;=&lt;</span> <span>tryAddHalfDart</span> <span>d</span> <span>&lt;=&lt;</span> <span>tryForce</span></code></pre>
<p>Note that (<code>&lt;=&lt;</code>) is the Kliesli arrow which replaces composition for Monads (defined in Control.Monad). (We could also have expressed this right to left sequence with a left to right version <code>tryForce &gt;=&gt; tryAddHalfDart d &gt;=&gt; tryForce</code>). The definition of <code>combo</code> becomes</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>combo</span> <span style="color: red;">::</span> <span>Dedge</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span>
    <span>combo</span> <span>d</span> <span style="color: red;">=</span> <span>runTry</span> <span>.</span> <span style="color: red;">(</span><span>tryForce</span> <span>&lt;=&lt;</span> <span>tryAddHalfDart</span> <span>d</span> <span>&lt;=&lt;</span> <span>tryForce</span><span style="color: red;">)</span> <span>.</span> <span>decompose</span></code></pre>
<p>This has no performance improvement, but now we can pass the sequence to <code>tryFSOp</code> to remove the unnecessary conversions between steps.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>combo</span> <span style="color: red;">::</span> <span>Dedge</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Tgraph</span>
    <span>combo</span> <span>d</span> <span style="color: red;">=</span> <span>runTry</span> <span>.</span> <span>tryFSOp</span> <span style="color: red;">(</span><span>tryForce</span> <span>&lt;=&lt;</span> <span>tryAddHalfDart</span> <span>d</span> <span>&lt;=&lt;</span> <span>tryForce</span><span style="color: red;">)</span> <span>.</span> <span>decompose</span></code></pre>
<p>The sequence actually has type <code>Forcible a =&gt; a -&gt; Try a</code> but when passed to <code>tryFSOp</code> it specialises to type <code>ForceState -&gt; Try ForseState</code>. This ensures the sequence works on a <code>ForceState</code> and any conversions are confined to the beginning and end of the sequence, avoiding unnecessary intermediate conversions.</p>
<h3 id="a-limitation-of-forcing">A limitation of forcing</h3>
<p>To avoid creating touching vertices (or crossing boundaries) a <code>BoundaryState</code> keeps track of locations of boundary vertices. At around 35,000 face additions in a single <code>force</code> operation the calculated positions of boundary vertices can become too inaccurate to prevent touching vertex problems. In such cases it is better to use</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>recalibratingForce</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span>
    <span>tryRecalibratingForce</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>a</span></code></pre>
<p>These work by recalculating all vertex positions at 20,000 step intervals to get more accurate boundary vertex positions. For example, 6 decompositions of the <code>kingGraph</code> has 2,906 faces. Applying <code>force</code> to this should result in 53,574 faces but will go wrong before it reaches that. This can be fixed by calculating either</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>recalibratingForce</span> <span style="color: red;">(</span><span>decompositions</span> <span>kingGraph</span> <span>!!</span><span class="hs-num">6</span><span style="color: red;">)</span></code></pre>
<p>or using an extra <code>force</code> before the decompositions</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>force</span> <span style="color: red;">(</span><span>decompositions</span> <span style="color: red;">(</span><span>force</span> <span>kingGraph</span><span style="color: red;">)</span> <span>!!</span><span class="hs-num">6</span><span style="color: red;">)</span></code></pre>
<p>In the latter case, the final <code>force</code> only needs to add 17,864 faces to the 35,710 produced by <code>decompositions (force kingGraph) !!6</code>.</p>
<p><a name="6"> </a></p>
<h2 id="advanced-operations">6. Advanced Operations</h2>
<h3 id="guided-comparison-of-tgraphs">Guided comparison of <code>Tgraph</code>s</h3>
<p>Asking if two <code>Tgraph</code>s are equivalent (the same apart from choice of vertex numbers) is a an np-complete problem. However, we do have an efficient <em>guided</em> way of comparing <code>Tgraph</code>s. In the module <code>Tgraph.Rellabelling</code> we have</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>sameGraph</span> <span style="color: red;">::</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Bool</span></code></pre>
<p>The expression <code>sameGraph (g1,d1) (g2,d2)</code> asks if <code>g2</code> can be relabelled to match <code>g1</code> assuming that the directed edge <code>d2</code> in <code>g2</code> is identified with <code>d1</code> in <code>g1</code>. Hence the comparison is guided by the assumption that <code>d2</code> corresponds to <code>d1</code>.</p>
<p>It is implemented using</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>tryRelabelToMatch</span> <span style="color: red;">::</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>Tgraph</span></code></pre>
<p>where <code>tryRelabelToMatch (g1,d1) (g2,d2)</code> will either fail with a <code>Left report</code> if a mismatch is found when relabelling <code>g2</code> to match <code>g1</code> or will succeed with <code>Right g3</code> where <code>g3</code> is a relabelled version of <code>g2</code>. The successful result <code>g3</code> will match <code>g1</code> in a maximal tile-connected collection of faces containing the face with edge <code>d1</code> and have vertices disjoint from those of <code>g1</code> elsewhere. The comparison tries to grow a suitable relabelling by comparing faces one at a time starting from the face with edge <code>d1</code> in <code>g1</code> and the face with edge <code>d2</code> in <code>g2</code>. (This relies on the fact that <code>Tgraph</code>s are connected with no crossing boundaries, and hence tile-connected.)</p>
<p>The above function is also used to implement</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>tryFullUnion</span><span style="color: red;">::</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span>Tgraph</span></code></pre>
<p>which tries to find the union of two <code>Tgraph</code>s guided by a directed edge identification. However, there is an extra complexity arising from the fact that <code>Tgraph</code>s might <em>overlap</em> in more than one tile-connected region. After calculating one overlapping region, the full union uses some geometry (calculating vertex locations) to detect further overlaps.</p>
<p>Finally we have</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>commonFaces</span><span style="color: red;">::</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span style="color: red;">(</span><span>Tgraph</span><span style="color: red;">,</span><span>Dedge</span><span style="color: red;">)</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span></code></pre>
<p>which will find common regions of overlapping faces of two <code>Tgraph</code>s guided by a directed edge identification. The resulting common faces will be a sub-collection of faces from the first <code>Tgraph</code>. These are returned as a list as they may not be a connected collection of faces and therefore not necessarily a <code>Tgraph</code>.</p>
<h3 id="empires-and-superforce">Empires and SuperForce</h3>
<p>In <a href="https://readerunner.wordpress.com/2023/04/26/graphs-kites-and-darts-empires-and-superforce/">Empires and SuperForce</a> we discussed forced boundary coverings which were used to implement both a <code>superForce</code> operation</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>superForce</span><span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span></code></pre>
<p>and operations to calculate empires.</p>
<p>We will not repeat the descriptions here other than to note that</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>forcedBoundaryECovering</span><span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Forced</span> <span>Tgraph</span><span style="color: red;">]</span></code></pre>
<p>finds boundary edge coverings after forcing a <code>Tgraph</code>. That is, <code>forcedBoundaryECovering g</code> will first force <code>g</code>, then (if it succeeds) finds a collection of (forced) extensions to <code>force g</code> such that</p>
<ul>
<li>each extension has the whole boundary of <code>force g</code> as internal edges.</li>
<li>each possible addition to a boundary edge of <code>force g</code> (kite or dart) has been included in the collection.</li>
</ul>
<p>(<em>possible</em> here means – not leading to a stuck <code>Tgraph</code> when forced.) There is also</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>forcedBoundaryVCovering</span><span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Forced</span> <span>Tgraph</span><span style="color: red;">]</span></code></pre>
<p>which does the same except that the extensions have all boundary vertices internal rather than just the boundary edges. In both cases the result is a list of explicitly forced Tgraphs (discussed next).</p>
<h3 id="combinations-and-explicitly-forced">Combinations and Explicitly Forced</h3>
<p>We introduced a new type <code>Forced</code> (in v 1.3) to enable a forcible to be explictily labelled as being forced. For example</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>forceF</span>    <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Forced</span> <span>a</span> 
    <span>tryForceF</span> <span style="color: red;">::</span> <span>Forcible</span> <span>a</span> <span style="color: red;">=&gt;</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>Try</span> <span style="color: red;">(</span><span>Forced</span> <span>a</span><span style="color: red;">)</span>
    <span>forgetF</span>   <span style="color: red;">::</span> <span>Forced</span> <span>a</span> <span style="color: red;">-&gt;</span> <span>a</span></code></pre>
<p>This allows us to restrict certain functions which expect a forced argument by making this explicit.</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>composeF</span> <span style="color: red;">::</span> <span>Forced</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Forced</span> <span>Tgraph</span></code></pre>
<p>The definition makes use of theorems established in <a href="https://readerunner.wordpress.com/2023/09/12/graphs-kites-and-darts-and-theorems/">Graphs,Kites and Darts and Theorems</a> that composing a forced <code>Tgraph</code> does not require a check (for connectedness and no crossing boundaries) and the result is also forced. This can then be used to define efficient combinations such as</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>compForce</span><span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Forced</span> <span>Tgraph</span>      <span style="color: green;">-- compose after forcing</span>
    <span>compForce</span> <span style="color: red;">=</span> <span>composeF</span> <span>.</span> <span>forceF</span>

    <span>allCompForce</span><span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span style="color: red;">[</span><span>Forced</span> <span>Tgraph</span><span style="color: red;">]</span> <span style="color: green;">-- iterated (compose after force) while not emptyTgraph</span>
    <span>maxCompForce</span><span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>Forced</span> <span>Tgraph</span>   <span style="color: green;">-- last item in allCompForce (or emptyTgraph)</span></code></pre>
<h3 id="tracked-tgraphs">Tracked Tgraphs</h3>
<p>The type</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span style="color: blue; font-weight: bold;">data</span> <span>TrackedTgraph</span> <span style="color: red;">=</span> <span>TrackedTgraph</span>
       <span style="color: red;">{</span> <span>tgraph</span>  <span style="color: red;">::</span> <span>Tgraph</span>
       <span style="color: red;">,</span> <span>tracked</span> <span style="color: red;">::</span> <span style="color: red;">[</span><span style="color: red;">[</span><span>TileFace</span><span style="color: red;">]</span><span style="color: red;">]</span> 
       <span style="color: red;">}</span> <span style="color: blue; font-weight: bold;">deriving</span> <span>Show</span></code></pre>
<p>has proven useful in experimentation as well as in producing artwork with darts and kites. The idea is to keep a record of sub-collections of faces of a <code>Tgraph</code> when doing both force operations and decompositions. A list of the sub-collections forms the tracked list associated with the <code>Tgraph</code>. We make <code>TrackedTgraph</code> an instance of class <code>Forcible</code> by having force operations only affect the <code>Tgraph</code> and not the tracked list. The significant idea is the implementation of</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>decomposeTracked</span> <span style="color: red;">::</span> <span>TrackedTgraph</span> <span style="color: red;">-&gt;</span> <span>TrackedTgraph</span></code></pre>
<p>Decomposition of a <code>Tgraph</code> involves introducing a new vertex for each long edge and each kite join. These are then used to construct the decomposed faces. For <code>decomposeTracked</code> we do the same for the <code>Tgraph</code>, but when it comes to the tracked collections, we decompose them re-using the same new vertex numbers calculated for the edges in the <code>Tgraph</code>. This keeps a consistent numbering between the <code>Tgraph</code> and tracked faces, so each item in the tracked list remains a sub-collection of faces in the <code>Tgraph</code>.</p>
<p>The function</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>drawTrackedTgraph</span> <span style="color: red;">::</span> <span style="color: red;">[</span><span>VPatch</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span><span style="color: red;">]</span> <span style="color: red;">-&gt;</span> <span>TrackedTgraph</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>is used to draw a <code>TrackedTgraph</code>. It uses a list of functions to draw <code>VPatch</code>es. The first drawing function is applied to a <code>VPatch</code> for any untracked faces. Subsequent functions are applied to <code>VPatch</code>es for the tracked list in order. Each diagram is beneath later ones in the list, with the diagram for the untracked faces at the bottom. The <code>VPatch</code>es used are all restrictions of a single <code>VPatch</code> for the <code>Tgraph</code>, so will be consistent in vertex locations. When labels are used, there is also a <code>drawTrackedTgraphRotating</code> and <code>drawTrackedTgraphAligning</code> for rotating or aligning the <code>VPatch</code> prior to applying the drawing functions.</p>
<p>Note that the result of calculating empires (see <a href="https://readerunner.wordpress.com/2023/04/26/graphs-kites-and-darts-empires-and-superforce/">Empires and SuperForce</a> ) is represented as a <code>TrackedTgraph</code>. The result is actually the common faces of a forced boundary covering, but a particular element of the covering (the first one) is chosen as the background <code>Tgraph</code> with the common faces as a tracked sub-collection of faces. Hence we have</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">    <span>empire1</span><span style="color: red;">,</span> <span>empire2</span> <span style="color: red;">::</span> <span>Tgraph</span> <span style="color: red;">-&gt;</span> <span>TrackedTgraph</span>
    
    <span>drawEmpire</span> <span style="color: red;">::</span> <span>TrackedTgraph</span> <span style="color: red;">-&gt;</span> <span>Diagram</span> <span>B</span></code></pre>
<p>Figure 10 was also created using <code>TrackedTgraph</code>s.</p>
<figure>
<img alt="Figure 10: Using a TrackedTgraph for drawing" src="https://readerunner.wordpress.com/wp-content/uploads/2024/04/cover.png?w=625"/><figcaption>Figure 10: Using a TrackedTgraph for drawing</figcaption></figure>
<p><a name="7"> </a></p>
<h2 id="other-reading">7. Other Reading</h2>
<p>Previous related blogs are:</p>
<ul>
<li><a href="https://readerunner.wordpress.com/2021/03/20/diagrams-for-penrose-tiles/">Diagrams for Penrose Tiles</a> – the first blog introduced drawing <code>Piece</code>s and <code>Patch</code>es (without using Tgraphs) and provided a version of decomposing for Patches (<code>decompPatch</code>).</li>
<li><a href="https://readerunner.wordpress.com/2022/01/06/graphs-kites-and-darts/">Graphs, Kites and Darts</a> intoduced Tgraphs. This gave more details of implementation and results of early explorations. (The class <code>Forcible</code> was introduced subsequently).</li>
<li><a href="https://readerunner.wordpress.com/2023/04/26/graphs-kites-and-darts-empires-and-superforce/">Empires and SuperForce</a> – these new operations were based on observing properties of boundaries of forced Tgraphs.</li>
<li><a href="https://readerunner.wordpress.com/2023/09/12/graphs-kites-and-darts-and-theorems/">Graphs,Kites and Darts and Theorems</a> established some important results relating <code>force</code>, <code>compose</code>, <code>decompose</code>.</li>
</ul></div>
    </content>
    <updated>2026-02-27T13:25:56Z</updated>
    <published>2024-04-08T15:11:17Z</published>
    <category scheme="https://readerunner.wordpress.com" term="Haskell"/>
    <category scheme="https://readerunner.wordpress.com" term="Maths"/>
    <author>
      <name>readerunner</name>
      <uri>https://readerunner.wordpress.com</uri>
    </author>
    <source>
      <id>http://readerunner.wordpress.com/feed/atom/</id>
      <link href="https://readerunner.wordpress.com" rel="alternate" type="text/html"/>
      <link href="https://readerunner.wordpress.com/feed/atom/" rel="self" type="application/atom+xml"/>
      <link href="https://readerunner.wordpress.com/osd.xml" rel="search" title="readerunner" type="application/opensearchdescription+xml"/>
      <link href="https://s1.wp.com/opensearch.xml" rel="search" title="WordPress.com" type="application/opensearchdescription+xml"/>
      <link href="https://readerunner.wordpress.com/?pushpress=hub" rel="hub" type="text/html"/>
      <subtitle xml:lang="en">maths and computing experiments</subtitle>
      <title xml:lang="en">readerunner</title>
      <updated>2026-02-27T13:25:56Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://www.joachim-breitner.de/blog/819-Vibe-coding_a_debugger_for_a_DSL</id>
    <link href="https://www.joachim-breitner.de/blog/819-Vibe-coding_a_debugger_for_a_DSL" rel="alternate" type="text/html"/>
    <title>Vibe-coding a debugger for a DSL</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Earlier this week a colleague of mine, Emilio Jesús Gallego Arias, shared a demo of something he built as an experiment, and I felt the desire to share this and add a bit of reflection. (Not keen on watching a 5 min video? Read on below.)</p>


<h3 id="what-was-that">What was that?</h3>
<p>So what did you just see (or skipped watching)? You could see Emilio’s screen, running VSCode and editing a Lean file. He designed a small programming language that he embedded into Lean, including an evaluator. So far, so standard, but a few things stick out already:</p>
<ul>
<li>Using Lean’s very extensible syntax this embedding is rather elegant and pretty.</li>
<li>Furthermore, he can run this DSL code right there, in the source code, using commands like <a href="https://lean-lang.org/doc/reference/latest/find/?domain=Verso.Genre.Manual.section&amp;name=hash-eval"><code>#eval</code></a>. This is a bit like the interpreter found in Haskell or Python, but without needing a separate process, or like using a Jupyter notebook, but without the stateful cell management.</li>
</ul>
<p>This is already a nice demonstration of Lean’s abilities and strength, as we know them. But what blew my mind the first time was what happened next: He had a visual debugger that allowed him to <em>debug his DSL program</em>. It appeared on the right, in Lean’s “Info View”, where various Lean tools can hook into, show information and allow the user to interact.</p>
<p>But it did not stop there, and my mind was blown a second time: Emilio opened VSCode’s „Debugger“ pane on the left, and was able to properly use VSCode’s full-fledged debugger frontend for his own little embedded programming language! Complete with highlighting the executed line, with the ability to set breakpoints there, and showing the state of local variables in the debugger.</p>
<p>Having a good debugger is not to be taken for granted even for serious, practical programming languages. Having it for a small embedded language that you just built yourself? I wouldn’t have even considered that.</p>
<h3 id="did-it-take-long">Did it take long?</h3>
<p>If I were Emilio’s manager I would applaud the demo and then would have to ask how many weeks he spent on that. Coming up with the language, getting the syntax extension right, writing the evaluator and especially learning how the debugger integration into VSCode (using the <a href="https://microsoft.github.io/debug-adapter-protocol/">DAP protocol</a>) works, and then instrumenting his evaluator to speak that protocol – that is a sizeable project!</p>
<p>It turns out the answer isn’t measured in weeks: it took just one day of coding together with GPT-Codex 5.3. My mind was blown a third time.</p>
<h3 id="why-does-lean-make-a-difference">Why does Lean make a difference?</h3>
<p>I am sure this post is just one of many stories you have read in recent weeks about how new models like Claude Opus 4.6 and GPT-Codex 5.3 built impressive things in hours that would have taken days or more before. But have you seen something like this? Agentic coding is powerful, but limited by what the underlying platform exposes. I claim that Lean is a particularly well-suited platform to unleash the agents’ versatility.</p>
<p>Here we are using Lean as a programming language, not as a theorem prover (which brings other immediate benefits when using agents, e.g. the produced code can be verified rather than merely plausible, but that’s a story to be told elsewhere.)</p>
<p>But arguably because Lean is <em>also</em> a theorem prover, and because of the requirements that stem from that, its architecture is different from that of a conventional programming language implementation:</p>
<ul>
<li>As a theorem prover, it needs extensible syntax to allow formalizing mathematics in an ergonomic way, but it can also be used for embedding syntax.</li>
<li>As a theorem prover, it needs the ability to run “tactics” written by the user, hence the ability to evaluate the code right there in the editor.</li>
<li>As a theorem prover, it needs to give access to information such as tactic state, and such introspection abilities unlock many other features – such as a debugger for an embedded language.</li>
<li>As a theorem prover, it has to allow tools to present information like the tactic state, so it has the concept of <a href="https://lean-lang.org/examples/1900-1-1-widgets/">interactive “Widgets”</a>.</li>
</ul>
<p>So Lean’s design has always made such a feat <em>possible</em>. But it was no easy feat. The Lean API is large, and documentation never ceases to be improvable. In the past, it would take an expert (or someone willing to become one) to pull off that stunt. These days, coding assistants have no issue digesting, understanding and using the API, as Emilio’s demo shows.</p>
<p>The combination of Lean’s extensibility and the ability of coding agents to make use of that is a game changer to how we can develop software, with rich, deep, flexible and bespoke ways to interact with our code, created on demand.</p>
<h3 id="where-does-that-lead-us">Where does that lead us?</h3>
<p>Emilio actually shared more such demos (<a href="https://github.com/ejgallego/imp-lab">Github repository</a>). A visual explorer for the compiler output (<a href="https://www.joachim-breitner.de/various/lean-compiler-explorer.png">have a look at the screenshot</a>. A browser-devtool-like inspection tool for Lean’s “InfoTree”. Any of these provide a significant productivity boost. Any of these would have been a sizeable project half a year ago. Now it’s just a few hours of chatting with the agent.</p>
<p>So allow me to try and extrapolate into a future where coding agents have continued to advance at the current pace, and are used ubiquitously. Is there then even a point in polishing these tools, shipping them to our users, documenting them? Why build a compiler explorer for our users, if our users can just ask their agent to build one for them, right then when they need it, tailored to precisely the use case they have, with no unnecessary or confusing feature. The code would be single use, as the next time the user needs something like that the agent can just re-create it, maybe slightly different because every use case is different.</p>
<p>If that comes to pass then Lean may no longer get praise for its nice out-of-the-box user experience, but instead because it is such a powerful framework for ad-hoc UX improvements.</p>
<p>And Emilio wouldn’t post demos about his debugger. He’d just use it.</p></div>
    </summary>
    <updated>2026-02-25T10:53:30Z</updated>
    <published>2026-02-25T10:53:30Z</published>
    <author>
      <name>Joachim Breitner</name>
      <email>mail@joachim-breitner.de</email>
    </author>
    <source>
      <id>https://www.joachim-breitner.de/blog</id>
      <logo>https://joachim-breitner.de/avatars/avatar_128.png</logo>
      <link href="https://www.joachim-breitner.de/blog" rel="alternate" type="text/html"/>
      <link href="https://www.joachim-breitner.de/blog_feed.rss" rel="self" type="application/rss+xml"/>
      <subtitle>Joachim Breitners Denkblogade</subtitle>
      <title>nomeataâ€™s mind shares</title>
      <updated>2026-03-25T18:01:52Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>Buzzsprout-18725311</id>
    <link href="https://www.buzzsprout.com/1817535/episodes/18725311-77-franz-thoma.mp3" length="41677874" rel="enclosure" type="audio/mpeg"/>
    <title>77: Franz Thoma</title>
    <summary>Franz Thoma is Principal Consultant at TNG Technology Consulting, and an organizer of MuniHac. Franz sees functional programming and Haskell as a tool for thinking about software, even if the project is not written in Haskell. We had a far-reaching conversation about the differences between functional and object-oriented programming and their languages, software architecture, and Haskell adoption in industry.</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Franz Thoma is Principal Consultant at TNG Technology Consulting, and an organizer of MuniHac. Franz sees functional programming and Haskell as a tool for thinking about software, even if the project is not written in Haskell. We had a far-reaching conversation about the differences between functional and object-oriented programming and their languages, software architecture, and Haskell adoption in industry.Â </p></div>
    </content>
    <updated>2026-02-22T12:00:00Z</updated>
    <published>2026-02-22T12:00:00Z</published>
    <author>
      <name>Haskell Podcast</name>
    </author>
    <source>
      <id>https://haskell.foundation/podcast/</id>
      <logo>https://storage.buzzsprout.com/tnk1ztokn5vmeiufqmr4kkp37mw2?.jpg</logo>
      <category scheme="http://www.itunes.com/" term="Technology"/>
      <author>
        <name>Haskell Podcast</name>
      </author>
      <link href="https://rss.buzzsprout.com/1817535.rss" rel="self" type="application/rss+xml"/>
      <link href="https://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <link href="https://haskell.foundation/podcast/" rel="alternate" type="text/html"/>
      <rights>Â© 2026 The Haskell Interlude</rights>
      <subtitle>This is the Haskell Interlude, where the five co-hosts (Wouter Swierstra, Andres LÃ¶h, Alejandro Serrano, Niki Vazou, and Joachim Breitner) chat with Haskell guests!</subtitle>
      <title>The Haskell Interlude</title>
      <updated>2026-04-10T16:12:47Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://tweag.io/blog/2026-02-19-nickel-since-1-0/</id>
    <link href="https://tweag.io/blog/2026-02-19-nickel-since-1-0/" rel="alternate" type="text/html"/>
    <title>Nickel since 1.0</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>We released Nickel 1.0 in May 2023. Since then, we’ve been working so hard
on new features, bug fixes, and performance improvements that we haven’t had
the opportunity to write about them as much as we would’ve liked. This post
rounds up some of the big changes that we’ve landed over the past few years.</p>
<h2 id="new-language-features"><a class="anchor before" href="https://www.tweag.io/rss.xml#new-language-features"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>New language features</h2>
<h3 id="algebraic-data-types"><a class="anchor before" href="https://www.tweag.io/rss.xml#algebraic-data-types"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Algebraic data types</h3>
<p>The biggest new language feature is one that we have actually <a href="https://www.tweag.io/blog/2024-09-05-algebraic-data-types-nickel/">written
about</a>
before: algebraic data types — or <em>enum variants</em> in Nickel terminology —
first landed in <a href="https://github.com/tweag/nickel/releases/tag/1.5.0">Nickel 1.5</a>. Nickel has supported plain enums
for a long time: <code class="language-text">[| 'Carnitas, 'Fish |]</code> is the type of something that
can take two possible values: <code class="language-text">'Carnitas</code> or <code class="language-text">'Fish</code>. Enum variants extend
these by allowing the enum types to specify payloads, like
<code class="language-text">[| 'Carnitas { pineapple : Number }, 'Fish { avocado : Number, cheese : Number } |]</code>.
Types like this are supported by many modern programming languages, as
they are useful for encoding important invariants like the fact that carnitas
tacos can be topped with pineapple but not avocado. For more on the design
and motivation for algebraic data types in Nickel, see
<a href="https://www.tweag.io/blog/2024-09-05-algebraic-data-types-nickel/">our other post</a>.</p>
<h3 id="pattern-matching"><a class="anchor before" href="https://www.tweag.io/rss.xml#pattern-matching"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Pattern matching</h3>
<p>Nickel has had a <code class="language-text">match</code> statement for a while, but it used to be quite limited.
<a href="https://github.com/tweag/nickel/releases/tag/1.5.0">Nickel 1.5</a> and
<a href="https://github.com/tweag/nickel/releases/tag/1.7.0">Nickel 1.7</a> extended it significantly: not only can you now match
the enum variants we mentioned above, you can also match arrays, records, and constants.
You can also match the “or” of two patterns, and you can guard matches with predicates.</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token keyword">match</span><span class="token operator"> </span><span class="token operator">{</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">p</span><span class="token operator">i</span><span class="token operator">n</span><span class="token operator">e</span><span class="token operator">a</span><span class="token operator">p</span><span class="token operator">p</span><span class="token operator">l</span><span class="token operator">e</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator"> </span><span class="token keyword">if</span><span class="token operator"> </span><span class="token operator">p</span><span class="token operator">i</span><span class="token operator">n</span><span class="token operator">e</span><span class="token operator">a</span><span class="token operator">p</span><span class="token operator">p</span><span class="token operator">l</span><span class="token operator">e</span><span class="token operator"> </span><span class="token operator">&gt;</span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">5</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator">&gt;</span><span class="token operator"> </span><span class="token operator">s</span><span class="token operator">t</span><span class="token operator">d</span><span class="token operator">.</span><span class="token operator">f</span><span class="token operator">a</span><span class="token operator">i</span><span class="token operator">l</span><span class="token operator">_</span><span class="token operator">w</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">h</span><span class="token operator"> </span><span class="token string">"too much pineapple"</span><span class="token operator">,</span>

<span class="token operator"> </span><span class="token operator"> </span><span class="token operator">[</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">.</span><span class="token operator">.</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator">,</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">F</span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">.</span><span class="token operator">.</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator"> </span><span class="token operator">]</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token operator">o</span><span class="token operator">r</span><span class="token operator"> </span><span class="token operator">[</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">F</span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">.</span><span class="token operator">.</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator">,</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">.</span><span class="token operator">.</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator"> </span><span class="token operator">]</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator">&gt;</span><span class="token operator"> </span><span class="token string">"one of each"</span><span class="token operator">,</span>
<span class="token operator">}</span></code></pre></div>
<p>Basically, if you’ve used pattern matching in another language then
Nickel’s match blocks probably have the features you’re used to.
And they’re adapted to Nickel’s gradual typing: the example match block
above will work in dynamically typed code, but in a statically typed block
it will fail to typecheck, because there’s no static type that can be
either an enum or an array.</p>
<h3 id="field-punning"><a class="anchor before" href="https://www.tweag.io/rss.xml#field-punning"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Field punning</h3>
<p>Records in Nickel are recursive by default, meaning that in the record</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token operator">{</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">tacos</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">[</span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token property">pineapple</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">2</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator">]</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">price</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">p</span><span class="token operator">r</span><span class="token operator">i</span><span class="token operator">c</span><span class="token operator">e</span><span class="token operator">_</span><span class="token operator">p</span><span class="token operator">e</span><span class="token operator">r</span><span class="token operator">_</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator"> </span><span class="token operator">*</span><span class="token operator"> </span><span class="token operator">s</span><span class="token operator">t</span><span class="token operator">d</span><span class="token operator">.</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">r</span><span class="token operator">a</span><span class="token operator">y</span><span class="token operator">.</span><span class="token operator">l</span><span class="token operator">e</span><span class="token operator">n</span><span class="token operator">g</span><span class="token operator">t</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator">s</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">price_per_taco</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">5</span><span class="token operator">,</span>
<span class="token operator">}</span></code></pre></div>
<p>the name <code class="language-text">price_per_taco</code> in the definition of <code class="language-text">price</code> refers to the field
<code class="language-text">price_per_taco</code> defined within the record. This is behavior is <em>usually</em> very handy,
but it can be annoying when you’re trying to define a field whose name shadows
something in an outer scope. For example, suppose you want to move the definition
of <code class="language-text">tacos</code> outside the record:</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token keyword">let</span><span class="token operator"> </span><span class="token property">tacos</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">[</span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token property">pineapple</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">2</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator">]</span><span class="token operator"> </span><span class="token keyword">in</span>
<span class="token operator">{</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">tacos</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator">s</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">price</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">p</span><span class="token operator">r</span><span class="token operator">i</span><span class="token operator">c</span><span class="token operator">e</span><span class="token operator">_</span><span class="token operator">p</span><span class="token operator">e</span><span class="token operator">r</span><span class="token operator">_</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator"> </span><span class="token operator">*</span><span class="token operator"> </span><span class="token operator">s</span><span class="token operator">t</span><span class="token operator">d</span><span class="token operator">.</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">r</span><span class="token operator">a</span><span class="token operator">y</span><span class="token operator">.</span><span class="token operator">l</span><span class="token operator">e</span><span class="token operator">n</span><span class="token operator">g</span><span class="token operator">t</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator">s</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">price_per_taco</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">5</span><span class="token operator">,</span>
<span class="token operator">}</span></code></pre></div>
<p>This probably doesn’t do what you want: it recurses infinitely, because
in the <code class="language-text">tacos = tacos</code> line, the <code class="language-text">tacos</code> on the right side of the equals
sign refers to the name <code class="language-text">tacos</code> that’s being defined on the left hand side
(and not, as you might expect, the <code class="language-text">tacos</code> in <code class="language-text">let tacos = ... in</code>).
There are workarounds (like calling the outer variable <code class="language-text">tacos_</code> instead),
but they’re annoying. <a href="https://github.com/tweag/nickel/releases/tag/1.12.0">Nickel 1.12</a> added the <code class="language-text">include</code> keyword,
where <code class="language-text">{ include tacos }</code> means <code class="language-text">{ tacos = &lt;tacos-from-the-outer-scope&gt; }</code>.</p>
<h3 id="let-blocks"><a class="anchor before" href="https://www.tweag.io/rss.xml#let-blocks"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Let blocks</h3>
<p>Nickel binds local variables using a <code class="language-text">let</code> statement, as in <code class="language-text">let x = 1 in x + x</code>. Before <a href="https://github.com/tweag/nickel/releases/tag/1.9.0">Nickel 1.9</a> you could only bind one variable at a time — as
in <code class="language-text">let x = 1 in let y = 2 in x + y</code> — but now you can bind multiple variables
in a single block, as in <code class="language-text">let x = 1, y = 2 in x + y</code>. In most situations this is
just a small syntactic convenience,<sup id="fnref-1"><a class="footnote-ref" href="https://www.tweag.io/rss.xml#fn-1">1</a></sup> but with <em>recursive</em> let
blocks you actually gain some expressive power. For example, they allow you to write
mutually recursive functions without putting them in a record (which used to be the
only way to create a recursive environment in Nickel):</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token keyword">let</span><span class="token operator"> </span><span class="token keyword">rec</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">is_even</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token keyword">fun</span><span class="token operator"> </span><span class="token operator">x</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator">&gt;</span><span class="token operator"> </span><span class="token keyword">if</span><span class="token operator"> </span><span class="token property">x</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">0</span><span class="token operator"> </span><span class="token keyword">then</span><span class="token operator"> </span><span class="token constant">true</span><span class="token operator"> </span><span class="token keyword">else</span><span class="token operator"> </span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">_</span><span class="token operator">o</span><span class="token operator">d</span><span class="token operator">d</span><span class="token operator"> </span><span class="token operator">(</span><span class="token operator">x</span><span class="token operator"> </span><span class="token operator">-</span><span class="token operator"> </span><span class="token number">1</span><span class="token operator">)</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">is_odd</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token keyword">fun</span><span class="token operator"> </span><span class="token operator">x</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator">&gt;</span><span class="token operator"> </span><span class="token keyword">if</span><span class="token operator"> </span><span class="token property">x</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">0</span><span class="token operator"> </span><span class="token keyword">then</span><span class="token operator"> </span><span class="token constant">false</span><span class="token operator"> </span><span class="token keyword">else</span><span class="token operator"> </span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">_</span><span class="token operator">e</span><span class="token operator">v</span><span class="token operator">e</span><span class="token operator">n</span><span class="token operator"> </span><span class="token operator">(</span><span class="token operator">x</span><span class="token operator"> </span><span class="token operator">-</span><span class="token operator"> </span><span class="token number">1</span><span class="token operator">)</span><span class="token operator">,</span>
<span class="token operator"/><span class="token keyword">in</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">_</span><span class="token operator">e</span><span class="token operator">v</span><span class="token operator">e</span><span class="token operator">n</span><span class="token operator"> </span><span class="token number">42</span></code></pre></div>
<h3 id="better-contract-constructors"><a class="anchor before" href="https://www.tweag.io/rss.xml#better-contract-constructors"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Better contract constructors</h3>
<p>Custom contracts were reworked in <a href="https://github.com/tweag/nickel/releases/tag/1.8.0">Nickel 1.8</a>, allowing for better control of a
contract’s eagerness, more precise error locations, and better composability.
Nickel’s standard library now offers three contract constructors. The
simplest is <code class="language-text">std.contract.from_predicate</code>, which turns a predicate (of type <code class="language-text">Dyn -&gt; Bool</code>)
into a contract. <code class="language-text">std.contract.from_validator</code> is slightly more complicated but offers
better control over error messages, while <code class="language-text">std.contract.custom</code> offers the most control.</p>
<p>A full description of the contract changes is out of scope for this blog post — there’s
a whole <a href="https://nickel-lang.org/user-manual/contracts/#user-defined-contracts">section</a> of the manual devoted to it. But the key point is that
contracts in Nickel are partly eager and partly lazy. For example, the contract in</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token keyword">let</span><span class="token operator"> </span><span class="token property">Taco</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">[</span><span class="token operator">|</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator">,</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">F</span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">|</span><span class="token operator">]</span><span class="token operator"> </span><span class="token keyword">in</span>
<span class="token operator"/><span class="token keyword">let</span><span class="token operator"> </span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">|</span><span class="token operator"> </span><span class="token class-name">Array</span><span class="token operator"> </span><span class="token property">Taco</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">[</span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator">,</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">r</span><span class="token operator">u</span><span class="token operator">n</span><span class="token operator">c</span><span class="token operator">h</span><span class="token operator">y</span><span class="token operator">T</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator">S</span><span class="token operator">u</span><span class="token operator">p</span><span class="token operator">r</span><span class="token operator">e</span><span class="token operator">m</span><span class="token operator">e</span><span class="token operator">]</span><span class="token operator"> </span><span class="token keyword">in</span>
<span class="token operator">&lt;</span><span class="token operator">s</span><span class="token operator">o</span><span class="token operator">m</span><span class="token operator">e</span><span class="token operator">t</span><span class="token operator">h</span><span class="token operator">i</span><span class="token operator">n</span><span class="token operator">g</span><span class="token operator">&gt;</span></code></pre></div>
<p>gets applied in two stages. When <code class="language-text">tacos</code> first gets evaluated, the contract checks that
<code class="language-text">tacos</code> is an array. But rather than validating the array elements immediately,
it propagates the element contracts inside the array and leaves them
unevaluated; essentially, <code class="language-text">tacos</code> gets evaluated to
<code class="language-text">['Carnitas | Taco, 'CrunchyTacoSupreme | Taco]</code>. Only when the <em>elements</em> of the array get
evaluated are their contracts checked. In particular, if the array elements are
never actually evaluated (for example, if <code class="language-text">&lt;something&gt;</code> is <code class="language-text">std.array.length tacos</code>, which doesn’t evaluate the individual elements) then we’ll never find
out that <code class="language-text">'CrunchyTacoSupreme</code> isn’t actually a <code class="language-text">Taco</code>.</p>
<p>The lazy/eager distinction has been part of Nickel’s built-in record and array
contracts since the beginning, but never fully exploitable by custom contracts.
The new <code class="language-text">std.contract.custom</code> constructor creates a contract with explicit lazy
and eager parts, and the <code class="language-text">std.contract.check</code> function allows for speculatively
checking the eager part of a contract without bailing out if it fails. Together,
these ingredients allowed us to create useful <a href="https://www.tweag.io/blog/2022-04-28-union-intersection-contracts/">union contracts</a> (<code class="language-text">std.contract.any_of</code>)
and improve the error reporting of the eager contracts in <a href="https://github.com/nickel-lang/json-schema-to-nickel/">json-schema-to-nickel</a>,
our tool for converting JSON schemas to Nickel contracts.</p>
<h2 id="performance-improvements"><a class="anchor before" href="https://www.tweag.io/rss.xml#performance-improvements"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Performance improvements</h2>
<p>For Nickel 1.0, we were focused on getting the basic language right. Since then
(and especially over the past year), we’ve been working on getting the interpreter
to run faster. While the performance improvements you observe will depend heavily
on your use case, we’ve seen large user-provided Nickel configurations that
evaluate 10x faster now than they were two years ago (and 3x faster than six months ago).
The most recent performance improvements are part of our progress towards
a <a href="https://github.com/tweag/nickel/blob/c483fe2811fa6b271d96cf87b8b3d3872fe03d8f/rfcs/007-bytecode-interpreter.md">bytecode interpreter</a>. We’ve been landing these improvements gradually over the
past year or so, but most of that preparation only had a performance impact
starting in <a href="https://github.com/tweag/nickel/releases/tag/1.15.1">Nickel 1.15</a>.</p>
<h3 id="standard-library-improvements"><a class="anchor before" href="https://www.tweag.io/rss.xml#standard-library-improvements"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Standard library improvements</h3>
<p>Nickel’s <a href="https://nickel-lang.org/stdlib/std/">standard library</a> has roughly doubled in size since Nickel 1.0, offering many useful
utility functions (like <code class="language-text">std.record.get_or</code> or <code class="language-text">std.string.find_all</code>) and contract combinators
(like <code class="language-text">std.contract.Sequence</code> or <code class="language-text">std.contract.any_of</code>). The standard library now also
contains a useful set of trigonometric and other numeric functions,
<a href="https://github.com/nickel-lang/nickel/issues/2005">contributed</a> by a community member who was
using Nickel to configure a robot.</p>
<h2 id="tooling-and-distribution-improvements"><a class="anchor before" href="https://www.tweag.io/rss.xml#tooling-and-distribution-improvements"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Tooling and distribution improvements</h2>
<p>Nickel has seen many improvements that are not directly tied to the Nickel
language itself.</p>
<h3 id="language-server-improvements"><a class="anchor before" href="https://www.tweag.io/rss.xml#language-server-improvements"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Language server improvements</h3>
<p>Nickel’s language server (NLS) has seen many improvements, especially
in <a href="https://github.com/tweag/nickel/releases/tag/1.2.0">Nickel 1.2</a> and <a href="https://github.com/tweag/nickel/releases/tag/1.3.0">1.3</a>. It now supports finding references
and definitions, listing symbols, and various other table-stakes language
server features. Completions have also been improved substantially since
version 1.0, and can make intelligent use of type- and contract-related
information. For example, in</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">‸</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator"> </span><span class="token operator">|</span><span class="token operator"> </span><span class="token operator">[</span><span class="token operator">|</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">C</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">n</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">s</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">p</span><span class="token operator">i</span><span class="token operator">n</span><span class="token operator">e</span><span class="token operator">a</span><span class="token operator">p</span><span class="token operator">p</span><span class="token operator">l</span><span class="token operator">e</span><span class="token operator"> </span><span class="token operator">:</span><span class="token operator"> </span><span class="token class-name">Number</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator">,</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">F</span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token operator">a</span><span class="token operator">v</span><span class="token operator">o</span><span class="token operator">c</span><span class="token operator">a</span><span class="token operator">d</span><span class="token operator">o</span><span class="token operator"> </span><span class="token operator">:</span><span class="token operator"> </span><span class="token class-name">Number</span><span class="token operator">,</span><span class="token operator"> </span><span class="token operator">c</span><span class="token operator">h</span><span class="token operator">e</span><span class="token operator">e</span><span class="token operator">s</span><span class="token operator">e</span><span class="token operator"> </span><span class="token operator">:</span><span class="token operator"> </span><span class="token class-name">Number</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator"> </span><span class="token operator">|</span><span class="token operator">]</span>
<span class="token operator"/><span class="token comment">#           └── cursor is here</span></code></pre></div>
<p>NLS knows to offer “pineapple” as a completion, but not “avocado”.</p>
<p>NLS has also gained the ability to offer diagnostics for <em>evaluation</em>
errors. This is very useful in Nickel because contract errors are detected
during evaluation instead of during typechecking. In-editor detection of
contract violations is part of the vision
articulated in a <a href="https://www.tweag.io/blog/2024-05-16-nickel-programmable-lsp/">previous blog post</a>, where configuration
errors are <a href="https://en.wikipedia.org/wiki/Shift-left_testing">left-shifted</a>
(because you get them as you type) and infinitely customizable (because
contracts are arbitrary code). Since the previous post was written, the
diagnostics have been further improved thanks to the contract
improvements mentioned above: the problematic field now gets
highlighted directly.</p>
<p><span class="gatsby-resp-image-wrapper" style="display: block; margin-left: auto; margin-right: auto;">
      <a class="gatsby-resp-image-link" href="https://www.tweag.io/static/de296f77c52d37c3a7c3d77ab91aee42/949b7/kubernetes.png" rel="noopener" style="display: block;" target="_blank">
    <span class="gatsby-resp-image-background-image" style="display: block;"/>
  <img alt="png" class="gatsby-resp-image-image" src="https://www.tweag.io/static/de296f77c52d37c3a7c3d77ab91aee42/fcda8/kubernetes.png" style="width: 100%; height: 100%; margin: 0; vertical-align: middle;" title="png"/>
  </a>
    </span></p>
<h3 id="unit-tests"><a class="anchor before" href="https://www.tweag.io/rss.xml#unit-tests"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Unit tests</h3>
<p>Since <a href="https://github.com/tweag/nickel/releases/tag/1.9.0">Nickel 1.9</a>, there is a <code class="language-text">nickel test</code> command that executes
unit tests contained in documentation comments.</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token operator">{</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token operator">m</span><span class="token operator">o</span><span class="token operator">r</span><span class="token operator">e</span><span class="token operator">_</span><span class="token operator">a</span><span class="token operator">v</span><span class="token operator">o</span><span class="token operator">c</span><span class="token operator">a</span><span class="token operator">d</span><span class="token operator">o</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token operator"> </span><span class="token operator"> </span><span class="token operator">|</span><span class="token operator"> </span><span class="token keyword">doc</span><span class="token operator"> </span><span class="token string">m%"
      Double the avocado!

      Here's an example that is automatically treated as a unit test:
      ```nickel
        more_avocado ('Fish { avocado = 1 })
        # =&gt; 'Fish { avocado = 2 }
      ```
      "%</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token operator"> </span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token keyword">fun</span><span class="token operator"> </span><span class="token operator">(</span><span class="token operator">'</span><span class="token operator">F</span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token property">avocado</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">a</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator">)</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator">&gt;</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">F</span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token property">avocado</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">3</span><span class="token operator"> </span><span class="token operator">*</span><span class="token operator"> </span><span class="token operator">a</span><span class="token operator"> </span><span class="token operator">}</span>
<span class="token operator">}</span></code></pre></div>
<p>Running <code class="language-text">nickel test</code> on this file will highlight the typo in the
function definition:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">testing more_avocado/0...FAILED
test more_avocado/0 failed
error: contract broken by a value
   ┌─ &lt;unknown&gt; (generated by evaluation):1:1
   │
 1 │ std.contract.Equal ('Fish { avocado = 2, })
   │ ------------------------------------------- expected type
   │
  &lt;snip...&gt;
   ┌─ input.ncl:12:38
   │
12 │     = fun ('Fish { avocado = a }) =&gt; 'Fish { avocado = 3 * a }
   │                                      ------------------------- evaluated to this expression

1 failures
error: tests failed</code></pre></div>
<h3 id="jsonyamltoml-interop"><a class="anchor before" href="https://www.tweag.io/rss.xml#jsonyamltoml-interop"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>JSON/YAML/TOML interop</h3>
<p>Interoperability with plain data formats (JSON, YAML, and TOML) has been improved in several ways.</p>
<ul>
<li>The YAML format allows for several YAML documents to be embedded in the
same file (separated by <code class="language-text">---</code> lines). We can read such files since
<a href="https://github.com/tweag/nickel/releases/tag/1.2.0">Nickel 1.2</a>, and we can write them since <a href="https://github.com/tweag/nickel/releases/tag/1.15.1">Nickel 1.15</a>:
from Nickel 1.15 onwards, <code class="language-text">nickel export --format yaml-documents</code> will export a Nickel
list to a collection of YAML documents (as opposed to <code class="language-text">nickel export --format yaml</code>, which
outputs a single YAML document that contains a list). Similarly, Nickel 1.15’s
standard library serialization functions support a new <code class="language-text">'YamlDocuments</code> format.</li>
<li>The <code class="language-text">nickel convert</code> command, added in <a href="https://github.com/tweag/nickel/releases/tag/1.15.1">Nickel 1.15</a> allows
conversion of JSON, YAML, or TOML to Nickel. This complements the long-supported ability to
import data formats as in <code class="language-text">import "file.json"</code>: while importing data formats is
useful for consuming data produced by some other tool, the new conversion feature
allows for migrating other configuration to Nickel.</li>
<li>Since <a href="https://github.com/tweag/nickel/releases/tag/1.3.0">Nickel 1.3</a>, the <code class="language-text">nickel</code> command line will merge plain data files into Nickel
code: if you have a JSON file containing <code class="language-text">{ "price_per_taco": 5 }</code> and a Nickel file
containing <code class="language-text">{ tacos = 3, price = price_per_taco * tacos, price_per_taco }</code> then
<code class="language-text">nickel export json_file.json nickel_file.ncl</code> will merge the JSON-specified price
into the Nickel configuration before evaluating it.</li>
</ul>
<h3 id="release-process-and-distribution"><a class="anchor before" href="https://www.tweag.io/rss.xml#release-process-and-distribution"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Release process and distribution</h3>
<p>For the <a href="https://github.com/tweag/nickel/releases/tag/1.0.0">Nickel 1.0</a> release, we built binaries for Linux x86_64 and aarch64 only.
Now, we’re building MacOS and Windows binaries as well. And we’re not the only distributors
of Nickel binaries: nixpkgs, Arch Linux, and Homebrew all have up-to-date Nickel packages.</p>
<p>We’ve also improved the usage of Nickel as a library. Since <a href="https://github.com/tweag/nickel/releases/tag/1.10.0">Nickel 1.10</a>, we’ve
been publishing our Python bindings on PyPI. And <a href="https://github.com/tweag/nickel/releases/tag/1.15.1">Nickel 1.15</a> saw our first
release of C and Go bindings, along with a stable Rust API.</p>
<h2 id="experimental-features"><a class="anchor before" href="https://www.tweag.io/rss.xml#experimental-features"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Experimental features</h2>
<p>Since 1.0, Nickel has grown a few experimental features for use cases that we want to
enable but don’t yet have enough confidence in the design and implementation
to fully support. Some of these features (Nix compatibility and package management)
are disabled by default; you’ll need to build Nickel with explicit support for them.
If you’re using any of these features, let us know what you’re doing
with them and whether they’re working the way you want!</p>
<h3 id="customize-mode"><a class="anchor before" href="https://www.tweag.io/rss.xml#customize-mode"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Customize mode</h3>
<p>Sometimes, writing a new configuration file for one or two settings feels unnecessary.
Our “customize mode”, introduced in <a href="https://github.com/tweag/nickel/releases/tag/1.2.0">Nickel 1.2</a>, allows configuration to be
supplied at the command line. For example, given the
<code class="language-text">{ tacos = 3, price = price_per_taco * tacos, price_per_taco }</code> example from before,
we can evaluate it with</p>
<div class="gatsby-highlight"><pre class="language-sh"><code class="language-sh">$ nickel <span class="token builtin class-name">export</span> tacos.ncl -- <span class="token assign-left variable">price_per_taco</span><span class="token operator">=</span><span class="token number">5</span>
<span class="token punctuation">{</span>
  <span class="token string">"price"</span><span class="token builtin class-name">:</span> <span class="token number">15</span>,
  <span class="token string">"price_per_taco"</span><span class="token builtin class-name">:</span> <span class="token number">5</span>,
  <span class="token string">"tacos"</span><span class="token builtin class-name">:</span> <span class="token number">3</span>
<span class="token punctuation">}</span></code></pre></div>
<p>Also, if you aren’t sure what options are available for setting, you can ask:</p>
<div class="gatsby-highlight"><pre class="language-sh"><code class="language-sh">$ nickel <span class="token builtin class-name">export</span> tacos.ncl -- list
Input fields:
- price_per_taco

Overridable fields <span class="token punctuation">(</span>require <span class="token variable"><span class="token variable">`</span><span class="token parameter variable">--override</span><span class="token variable">`</span></span><span class="token punctuation">)</span>:
- price
- tacos

Use the <span class="token variable"><span class="token variable">`</span>query<span class="token variable">`</span></span> subcommand to print a detailed description of a specific field. See <span class="token variable"><span class="token variable">`</span>nickel <span class="token builtin class-name">help</span> query<span class="token variable">`</span></span><span class="token builtin class-name">.</span></code></pre></div>
<p>Since <a href="https://github.com/tweag/nickel/releases/tag/1.11.0">Nickel 1.11</a>, customize mode has had support for environment variables:
<code class="language-text">nickel export tacos.ncl -- taco_description=@env:DESC</code> will expand the <code class="language-text">DESC</code>
environment variable and substitute it into the <code class="language-text">tacos.ncl</code> configuration.
In some cases, you could achieve something similar by expanding environment
variables using your shell, but correctly handling escaping there can be
painful (or even a security risk).</p>
<h3 id="nix-compatibility"><a class="anchor before" href="https://www.tweag.io/rss.xml#nix-compatibility"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Nix compatibility</h3>
<p>A lot of Nickel users are also Nix users, and so Nix interoperability is an
often-requested feature. Our current Nix interface is limited to plain data,
but you can import Nix from Nickel if you’ve built Nickel with the
“nix-experimental” feature:</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token operator">{</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">price</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">p</span><span class="token operator">r</span><span class="token operator">i</span><span class="token operator">c</span><span class="token operator">e</span><span class="token operator">_</span><span class="token operator">p</span><span class="token operator">e</span><span class="token operator">r</span><span class="token operator">_</span><span class="token operator">t</span><span class="token operator">a</span><span class="token operator">c</span><span class="token operator">o</span><span class="token operator"> </span><span class="token operator">*</span><span class="token operator"> </span><span class="token operator">s</span><span class="token operator">t</span><span class="token operator">d</span><span class="token operator">.</span><span class="token operator">a</span><span class="token operator">r</span><span class="token operator">r</span><span class="token operator">a</span><span class="token operator">y</span><span class="token operator">.</span><span class="token operator">l</span><span class="token operator">e</span><span class="token operator">n</span><span class="token operator">g</span><span class="token operator">t</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">(</span><span class="token keyword">import</span><span class="token operator"> </span><span class="token string">"tacos.nix"</span><span class="token operator">)</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">price_per_taco</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">5</span><span class="token operator">,</span>
<span class="token operator">}</span></code></pre></div>
<h3 id="package-management"><a class="anchor before" href="https://www.tweag.io/rss.xml#package-management"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Package management</h3>
<p>In Nickel 1.0, you could share code between projects by copying files around,
basically. <a href="https://github.com/tweag/nickel/releases/tag/1.11.0">Nickel 1.11</a> introduced package management, allowing you
to import Nickel dependencies from other directories, Git repositories, or a
central package registry. You declare your dependencies in a <code class="language-text">Nickel-pkg.ncl</code>
manifest file:</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token operator">{</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">name</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token string">"tacos"</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">authors</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">[</span><span class="token string">"Me"</span><span class="token operator">]</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">minimal_nickel_version</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token string">"1.15.0"</span><span class="token operator">,</span>
<span class="token operator"> </span><span class="token operator"> </span><span class="token property">dependencies</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token property">salsa</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">'</span><span class="token operator">G</span><span class="token operator">i</span><span class="token operator">t</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token property">package</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token string">"github:example/salsa"</span><span class="token operator">,</span><span class="token operator"> </span><span class="token property">version</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token string">"1.0"</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator"> </span><span class="token operator">}</span><span class="token operator">,</span>
<span class="token operator">}</span></code></pre></div>
<p>Then you can import those dependencies in your Nickel code:</p>
<div class="gatsby-highlight"><pre class="language-nickel"><code class="language-nickel"><span class="token operator">'</span><span class="token operator">F</span><span class="token operator">i</span><span class="token operator">s</span><span class="token operator">h</span><span class="token operator"> </span><span class="token operator">{</span><span class="token operator"> </span><span class="token property">avocado</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token number">1</span><span class="token operator">,</span><span class="token operator"> </span><span class="token property">salsa</span><span class="token operator"> </span><span class="token operator">=</span><span class="token operator"> </span><span class="token operator">(</span><span class="token keyword">import</span><span class="token operator"> </span><span class="token operator">s</span><span class="token operator">a</span><span class="token operator">l</span><span class="token operator">s</span><span class="token operator">a</span><span class="token operator">)</span><span class="token operator">.</span><span class="token operator">v</span><span class="token operator">e</span><span class="token operator">r</span><span class="token operator">d</span><span class="token operator">e</span><span class="token operator"> </span><span class="token operator">}</span></code></pre></div>
<h2 id="thank-you"><a class="anchor before" href="https://www.tweag.io/rss.xml#thank-you"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Thank you!</h2>
<p>That sums up the biggest changes to Nickel over the past two and a half years or so.
As we come up on 5,000 commits from 86 contributors, we’d like to thank you
for all the feedback, discussion, and participation that encourage us
to keep improving Nickel.</p>
<div class="footnotes">
<hr/>
<ol>
<li id="fn-1">There are some situations where let blocks can improve
performance with Nickel’s current interpreter: <code class="language-text">let x = 1 in let y = 2 in x + y</code> creates two nested environments while <code class="language-text">let x = 1, y = 2 in x + y</code> creates a single environment. Variable lookups are usually faster when
environments are less deeply nested, so the version with a let block should
be a little bit faster. This performance distinction will probably go away
once we have a <a href="https://github.com/tweag/nickel/blob/c483fe2811fa6b271d96cf87b8b3d3872fe03d8f/rfcs/007-bytecode-interpreter.md">bytecode interpreter</a>, though.<a class="footnote-backref" href="https://www.tweag.io/rss.xml#fnref-1">↩</a></li>
</ol>
</div></div>
    </summary>
    <updated>2026-02-19T00:00:00Z</updated>
    <published>2026-02-19T00:00:00Z</published>
    <source>
      <id>https://tweag.io</id>
      <author>
        <name>Tweag I/O</name>
      </author>
      <link href="https://tweag.io" rel="alternate" type="text/html"/>
      <link href="http://www.tweag.io/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Scale your engineering power. We enable deep-tech startups to achieve
their vision, from research to product delivery.</subtitle>
      <title>Tweag - Engineering blog</title>
      <updated>2026-04-02T12:24:07Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://magnus.therning.org/2026-02-18-switching-to-project.el.html</id>
    <link href="https://magnus.therning.org/2026-02-18-switching-to-project.el.html" rel="alternate" type="text/html"/>
    <title>Switching to project.el</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>
I've used <a href="https://github.com/bbatsov/projectile">projectile</a> ever since I created my own Emacs config. I have a vague
memory choosing it because some other package only supported it. (It might have
been <a href="https://emacs-lsp.github.io/lsp-mode/">lsp-mode</a>, but I'm not sure.) Anyway, now that <a href="https://magnus.therning.org/2026-01-19-trying-eglot,-again.html">I'm trying out eglot</a>, <a href="https://magnus.therning.org/2026-01-25-more-on-the-switch-to-eglot.html">again</a>,
I thought I might as well see if I can switch to <a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Projects.html">project.el</a>, which is included
in Emacs nowadays.
</p>
<div class="outline-2" id="outline-container-org80f0ee9">
<h2 id="org80f0ee9">A non-VC project marker</h2>
<div class="outline-text-2" id="text-org80f0ee9">
<p>
Projectile allows using a file, <code>.projectile</code>, in the root of a project. This
makes it possible to turn a folder into a project without having to use version
control. It's possible to configure project.el to respect more VC markers than
what's built-in. This can be used to define a non-VC marker.
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">setopt</span> project-vc-extra-root-markers '<span class="org-rainbow-delimiters-depth-2">(</span><span class="org-string">".projectile"</span> <span class="org-string">".git"</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
Since I've set <code>vc-handled-backends</code> to <code>nil</code> (the default made VC interfere
with magit, so I turned it off completely) I had to add <code>".git"</code> to make git
repos be recognised as projects too.
</p>
</div>
</div>
<div class="outline-2" id="outline-container-org511d0e2">
<h2 id="org511d0e2">Xref history</h2>
<div class="outline-text-2" id="text-org511d0e2">
<p>
The first thing to solve was that the <a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Xref.html">xref</a> stack wasn't per project. Somewhat
disappointingly there only seems to be two options for <code>xref-history-storage</code>
shipped with Emacs
</p>

<dl class="org-dl">
<dt><code>xref-global-history</code></dt><dd>a single global history (the default)</dd>
<dt><code>xref-window-local-history</code></dt><dd>a history per window</dd>
</dl>

<p>
I had the same issue with projectile, and ended up writing my own package for
it. For project.el I settled on using <a href="https://codeberg.org/imarko/xref-project-history.git">xref-project-history</a>.
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">use-package</span> xref-project-history
  <span class="org-builtin">:ensure</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-builtin">:type</span> git
           <span class="org-builtin">:repo</span> <span class="org-string">"https://codeberg.org/imarko/xref-project-history.git"</span>
           <span class="org-builtin">:branch</span> <span class="org-string">"master"</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-builtin">:custom</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>xref-history-storage #'xref-project-history<span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>
</div>
</div>
<div class="outline-2" id="outline-container-orgbaa0d4d">
<h2 id="orgbaa0d4d">Jumping between implementation and test</h2>
<div class="outline-text-2" id="text-orgbaa0d4d">
<p>
Projectile has a function for jumping between implementation and test. Not too
surprisingly it's called <code>projectile-toggle-between-implementation-and-test</code>. I
found some old emails in an archive suggesting that project.el might have had
something similar in the past, but if that's the case it's been removed by now.
When searching for a package I came across <a href="https://lists.gnu.org/archive/html/emacs-devel/2022-09/msg00300.html">this email comparing tools for
finding related files</a>. The author mentions two that are included with Emacs
</p>

<dl class="org-dl">
<dt><code>ff-find-other-file</code></dt><dd>part of find-file.el, which a few other functions and
a rather impressive set of settings to customise its behaviour.</dd>
<dt><code>find-sibling-file</code></dt><dd>a newer command, I believe, that also can be
customised.</dd>
</dl>

<p>
So, there are options, but neither of them are made to work nicely with
project.el out of the box. My most complicated use case seems to be in Haskell
projects where modules for implementation and test live in separate (mirrored)
folder hierarchies, e.g.
</p>

<pre class="example" id="org712aefd">src
└── Sider
    └── Data
        ├── Command.hs
        ├── Pipeline.hs
        └── Resp.hs
test
└── Sider
    └── Data
        ├── CommandSpec.hs
        ├── PipelineSpec.hs
        └── RespSpec.hs

</pre>

<p>
I'm not really sure how I'd configure <code>find-sibling-rules</code>, which are regular
expressions, to deal with folder hierarchies like this. To be honest, I didn't
really see a way of configuring <code>ff-find-other-file</code> at first either. Then I
happened on a post about <a href="https://dev.to/fredericlepied/emacs-how-to-switch-from-modulepy-to-testmodulepy-67k">switching between a module and its tests in Python</a>.
With its help I came up with the following
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">defun</span> <span class="org-function-name">mes/setup-hs-ff</span> <span class="org-rainbow-delimiters-depth-2">()</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">when-let*</span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-rainbow-delimiters-depth-4">(</span>proj-root <span class="org-rainbow-delimiters-depth-5">(</span>project-root <span class="org-rainbow-delimiters-depth-6">(</span>project-current<span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span>
              <span class="org-rainbow-delimiters-depth-4">(</span>rel-proj-root <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-keyword">-some--&gt;</span> <span class="org-rainbow-delimiters-depth-6">(</span>buffer-file-name<span class="org-rainbow-delimiters-depth-6">)</span>
                               <span class="org-rainbow-delimiters-depth-6">(</span>file-name-directory it<span class="org-rainbow-delimiters-depth-6">)</span>
                               <span class="org-rainbow-delimiters-depth-6">(</span>f-relative proj-root it<span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span>
              <span class="org-rainbow-delimiters-depth-4">(</span>sub-tree <span class="org-rainbow-delimiters-depth-5">(</span>car <span class="org-rainbow-delimiters-depth-6">(</span>f-split <span class="org-rainbow-delimiters-depth-7">(</span>f-relative <span class="org-rainbow-delimiters-depth-8">(</span>buffer-file-name<span class="org-rainbow-delimiters-depth-8">)</span> proj-root<span class="org-rainbow-delimiters-depth-7">)</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span>
              <span class="org-rainbow-delimiters-depth-4">(</span>search-dirs <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-keyword">--&gt;</span> '<span class="org-rainbow-delimiters-depth-6">(</span><span class="org-string">"src"</span> <span class="org-string">"test"</span><span class="org-rainbow-delimiters-depth-6">)</span>
                                <span class="org-rainbow-delimiters-depth-6">(</span>remove sub-tree it<span class="org-rainbow-delimiters-depth-6">)</span>
                                <span class="org-rainbow-delimiters-depth-6">(</span>-map <span class="org-rainbow-delimiters-depth-7">(</span><span class="org-keyword">lambda</span> <span class="org-rainbow-delimiters-depth-8">(</span>p<span class="org-rainbow-delimiters-depth-8">)</span> <span class="org-rainbow-delimiters-depth-8">(</span>f-join proj-root p<span class="org-rainbow-delimiters-depth-8">)</span><span class="org-rainbow-delimiters-depth-7">)</span> it<span class="org-rainbow-delimiters-depth-6">)</span>
                                <span class="org-rainbow-delimiters-depth-6">(</span>-select #'f-directory? it<span class="org-rainbow-delimiters-depth-6">)</span>
                                <span class="org-rainbow-delimiters-depth-6">(</span>-mapcat <span class="org-rainbow-delimiters-depth-7">(</span><span class="org-keyword">lambda</span> <span class="org-rainbow-delimiters-depth-8">(</span>p<span class="org-rainbow-delimiters-depth-8">)</span> <span class="org-rainbow-delimiters-depth-8">(</span>f-directories p nil t<span class="org-rainbow-delimiters-depth-8">)</span><span class="org-rainbow-delimiters-depth-7">)</span> it<span class="org-rainbow-delimiters-depth-6">)</span>
                                <span class="org-rainbow-delimiters-depth-6">(</span>-map <span class="org-rainbow-delimiters-depth-7">(</span><span class="org-keyword">lambda</span> <span class="org-rainbow-delimiters-depth-8">(</span>p<span class="org-rainbow-delimiters-depth-8">)</span> <span class="org-rainbow-delimiters-depth-8">(</span>f-relative p proj-root<span class="org-rainbow-delimiters-depth-8">)</span><span class="org-rainbow-delimiters-depth-7">)</span> it<span class="org-rainbow-delimiters-depth-6">)</span>
                                <span class="org-rainbow-delimiters-depth-6">(</span>-map <span class="org-rainbow-delimiters-depth-7">(</span><span class="org-keyword">lambda</span> <span class="org-rainbow-delimiters-depth-8">(</span>p<span class="org-rainbow-delimiters-depth-8">)</span> <span class="org-rainbow-delimiters-depth-8">(</span>f-join rel-proj-root p<span class="org-rainbow-delimiters-depth-8">)</span><span class="org-rainbow-delimiters-depth-7">)</span> it<span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span>
    <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-keyword">setq-local</span> ff-search-directories search-dirs
                ff-other-file-alist '<span class="org-rainbow-delimiters-depth-4">(</span><span class="org-rainbow-delimiters-depth-5">(</span><span class="org-string">"Spec\\.hs$"</span> <span class="org-rainbow-delimiters-depth-6">(</span><span class="org-string">".hs"</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span>
                                      <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-string">"\\.hs$"</span> <span class="org-rainbow-delimiters-depth-6">(</span><span class="org-string">"Spec.hs"</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
A few things to note
</p>

<ol class="org-ol">
<li>The order of rules in <code>ff-other-file-alist</code> is important, the first match is
chosen.</li>
<li><code>(buffer-file-name)</code> can, and really does, return <code>nil</code> at times, and
<code>file-name-directory</code> doesn't deal with anything but strings.</li>
<li>The entries in <code>ff-search-directories</code> have to be relative to the file in the
current buffer, hence the rather involved <code>varlist</code> in the <code>when-let*</code>
expression.</li>
</ol>

<p>
With this in place I get the following values for <code>ff-search-directories</code>
</p>

<dl class="org-dl">
<dt><code>src/Sider/Data/Command.hs</code></dt><dd><code>("../../../test/Sider" "../../../test/Sider/Data")</code></dd>
<dt><code>test/Sider/Data/CommandSpec.hs</code></dt><dd><code>("../../../src/Sider" "../../../src/Sider/Data")</code></dd>
</dl>

<p>
And <code>ff-find-other-file</code> works beautifully.
</p>
</div>
</div>
<div class="outline-2" id="outline-container-orgb8cf766">
<h2 id="orgb8cf766">Conclusion</h2>
<div class="outline-text-2" id="text-orgb8cf766">
<p>
My setup with project.el now covers everything I used from projectile so I'm
fairly confident I'll be happy keeping it.
</p>
</div>
</div>
<div class="taglist"><a href="https://magnus.therning.org/tags.html">Tags</a>: <a href="https://magnus.therning.org/tag-emacs.html">emacs</a> <a href="https://magnus.therning.org/tag-project-el.html">project-el</a> </div></div>
    </summary>
    <updated>2026-02-17T23:09:00Z</updated>
    <published>2026-02-17T23:09:00Z</published>
    <category term="emacs"/>
    <category term="project-el"/>
    <source>
      <id>https://magnus.therning.org/</id>
      <author>
        <name>Magnus Therning</name>
      </author>
      <link href="https://magnus.therning.org/" rel="alternate" type="text/html"/>
      <link href="https://magnus.therning.org/feed.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Magnus web site</subtitle>
      <title>Magnus web site</title>
      <updated>2026-02-17T23:09:47Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://magnus.therning.org/2026-02-16-using-advice-to-limit-lsp-ui-doc-nuisance.html</id>
    <link href="https://magnus.therning.org/2026-02-16-using-advice-to-limit-lsp-ui-doc-nuisance.html" rel="alternate" type="text/html"/>
    <title>Using advice to limit lsp-ui-doc nuisance</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>
I've switched back to <a href="https://emacs-lsp.github.io/lsp-mode/">lsp-mode</a> temporarily until I've had time to fix a few
things with my <code>eglot</code> setup. Returning prompted me to finally address an
irritating behaviour with <a href="https://emacs-lsp.github.io/lsp-ui/#lsp-ui-doc">lsp-ui-doc</a>.
</p>

<p>
No matter what I set <code>lsp-ui-doc-position</code> to it ends up covering information
that I want to see. While waiting for a <a href="https://github.com/emacs-lsp/lsp-ui/issues/793">fix</a> I decided to work around it. It
seems to me that this is exactly what <a href="https://www.gnu.org/software/emacs/manual/html_node/elisp/Advising-Functions.html">advice</a> is for.
</p>

<p>
I came up with the following to make sure the frame appears on the half of the
buffer where <code>point</code> isn't.
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">defun</span> <span class="org-function-name">my-lsp-ui-doc-wrapper</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-type">&amp;rest</span> _<span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">let*</span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-rainbow-delimiters-depth-4">(</span>pos-line <span class="org-rainbow-delimiters-depth-5">(</span>- <span class="org-rainbow-delimiters-depth-6">(</span>line-number-at-pos <span class="org-rainbow-delimiters-depth-7">(</span>point<span class="org-rainbow-delimiters-depth-7">)</span><span class="org-rainbow-delimiters-depth-6">)</span>
                      <span class="org-rainbow-delimiters-depth-6">(</span>line-number-at-pos <span class="org-rainbow-delimiters-depth-7">(</span>window-start<span class="org-rainbow-delimiters-depth-7">)</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span>
         <span class="org-rainbow-delimiters-depth-4">(</span>pos <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-keyword">if</span> <span class="org-rainbow-delimiters-depth-6">(</span>&lt;= pos-line <span class="org-rainbow-delimiters-depth-7">(</span>/ <span class="org-rainbow-delimiters-depth-8">(</span>window-body-height<span class="org-rainbow-delimiters-depth-8">)</span> 2<span class="org-rainbow-delimiters-depth-7">)</span><span class="org-rainbow-delimiters-depth-6">)</span>
                  'bottom
                'top<span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span>
    <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-keyword">setopt</span> lsp-ui-doc-position pos<span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>

<span class="org-rainbow-delimiters-depth-1">(</span>advice-add 'lsp-ui-doc--move-frame <span class="org-builtin">:before</span> #'my-lsp-ui-doc-wrapper<span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>
<div class="taglist"><a href="https://magnus.therning.org/tags.html">Tags</a>: <a href="https://magnus.therning.org/tag-emacs.html">emacs</a> <a href="https://magnus.therning.org/tag-lsp-mode.html">lsp-mode</a> </div></div>
    </summary>
    <updated>2026-02-16T19:10:00Z</updated>
    <published>2026-02-16T19:10:00Z</published>
    <category term="emacs"/>
    <category term="lsp-mode"/>
    <source>
      <id>https://magnus.therning.org/</id>
      <author>
        <name>Magnus Therning</name>
      </author>
      <link href="https://magnus.therning.org/" rel="alternate" type="text/html"/>
      <link href="https://magnus.therning.org/feed.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Magnus web site</subtitle>
      <title>Magnus web site</title>
      <updated>2026-02-17T23:09:47Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>https://haskellforall.com/2026/02/browse-code-by-meaning</id>
    <link href="https://haskellforall.com/2026/02/browse-code-by-meaning" rel="alternate" type="text/html"/>
    <title>Browse code by meaning&gt;</title>
    <summary>Navigate a repository using topic modeling</summary>
    <updated>2026-02-16T00:00:00Z</updated>
    <published>2026-02-16T00:00:00Z</published>
    <source>
      <id>https://haskellforall.com</id>
      <author>
        <name>Gabriella Gonzalez</name>
      </author>
      <link href="https://haskellforall.com" rel="alternate" type="text/html"/>
      <link href="https://haskellforall.com/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>A blog about Haskell and functional programming.</subtitle>
      <title>Haskell for all</title>
      <updated>2026-03-17T14:41:11Z</updated>
    </source>
  </entry>

  <entry>
    <id>tag:blogger.com,1999:blog-23427281.post-4433953526927400176</id>
    <link href="http://blog.holdenkarau.com/feeds/4433953526927400176/comments/default" rel="replies" title="Post Comments" type="application/atom+xml"/>
    <link href="http://www.blogger.com/comment/fullpage/post/23427281/4433953526927400176" rel="replies" title="0 Comments" type="text/html"/>
    <link href="http://www.blogger.com/feeds/23427281/posts/default/4433953526927400176" rel="edit" type="application/atom+xml"/>
    <link href="http://www.blogger.com/feeds/23427281/posts/default/4433953526927400176" rel="self" type="application/atom+xml"/>
    <link href="http://blog.holdenkarau.com/2026/02/new-year-new-job-same-projects.html" rel="alternate" title="New year new job, same projects" type="text/html"/>
    <title>New year new job, same projects</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p> I’m stoked to announce that I’ve joined Snowflake to continue working on OSS Apache Spark :) I’ve got a post on the Snowflake blog talking about the work we’re doing <a href="https://careers.snowflake.com/us/en/blogarticle/building-apache-spark-in-the-open-at-snowflake">https://careers.snowflake.com/us/en/blogarticle/building-apache-spark-in-the-open-at-snowflake</a> — </p></div>
    </content>
    <updated>2026-02-12T15:51:06Z</updated>
    <published>2026-02-12T15:51:00Z</published>
    <author>
      <name>Holden Karau</name>
      <email>noreply@blogger.com</email>
      <uri>http://www.blogger.com/profile/05915225834474424123</uri>
    </author>
    <source>
      <id>tag:blogger.com,1999:blog-23427281</id>
      <category term="all the code"/>
      <category term="life"/>
      <category term="random"/>
      <category term="codeing"/>
      <category term="atc"/>
      <category term="business"/>
      <category term="amazon web services"/>
      <category term="google summer of code"/>
      <category term="openmoko"/>
      <category term="spark"/>
      <category term="demo"/>
      <category term="democamp"/>
      <category term="freerunner"/>
      <category term="google"/>
      <category term="haskell"/>
      <category term="programming"/>
      <category term="spark-project"/>
      <category term="ssc"/>
      <category term="work"/>
      <category term="amazon"/>
      <category term="amazon web search platform"/>
      <category term="linux"/>
      <category term="scala"/>
      <category term="bugs"/>
      <category term="computer science"/>
      <category term="devicescape"/>
      <category term="ninjas"/>
      <category term="software"/>
      <category term="spam"/>
      <category term="subversion"/>
      <category term="talks"/>
      <category term="ubuntu"/>
      <category term="wireless"/>
      <category term="amazon ec2"/>
      <category term="awsp"/>
      <category term="barcamp"/>
      <category term="bigdata"/>
      <category term="boingo mobile"/>
      <category term="canada"/>
      <category term="computer science club"/>
      <category term="copyright"/>
      <category term="crazyness"/>
      <category term="csc"/>
      <category term="database builds"/>
      <category term="datamining"/>
      <category term="democampguelph"/>
      <category term="developement"/>
      <category term="dnsrbl"/>
      <category term="emacs"/>
      <category term="encryption"/>
      <category term="failboat"/>
      <category term="functional programming"/>
      <category term="funding"/>
      <category term="funtimes"/>
      <category term="gsoc"/>
      <category term="java"/>
      <category term="mzscheme"/>
      <category term="plugins"/>
      <category term="rms"/>
      <category term="ruby on rails"/>
      <category term="scheme"/>
      <category term="security"/>
      <category term="ssl"/>
      <category term="university of waterloo"/>
      <category term="videos"/>
      <category term="wifi"/>
      <category term="yahoo"/>
      <category term="802.11a"/>
      <category term="802.11b"/>
      <category term="802.11g"/>
      <category term="Bjarne Stroustrup"/>
      <category term="almost useless information"/>
      <category term="amazon s3"/>
      <category term="antispam"/>
      <category term="apache spark"/>
      <category term="asus"/>
      <category term="atom"/>
      <category term="barcampwaterloo"/>
      <category term="barcampwaterloo4"/>
      <category term="beer"/>
      <category term="blogging"/>
      <category term="boingo"/>
      <category term="build systems"/>
      <category term="c++"/>
      <category term="cabal"/>
      <category term="cabalandhunittogether"/>
      <category term="cellphone"/>
      <category term="character encodings"/>
      <category term="co-op"/>
      <category term="code"/>
      <category term="cogent"/>
      <category term="competitors"/>
      <category term="computers"/>
      <category term="concurrency"/>
      <category term="darcs"/>
      <category term="databases"/>
      <category term="democamp2"/>
      <category term="deployment"/>
      <category term="dns"/>
      <category term="dodgy"/>
      <category term="eclipse"/>
      <category term="emacs subversion"/>
      <category term="esr"/>
      <category term="facebook"/>
      <category term="fail"/>
      <category term="failboatish"/>
      <category term="feedback"/>
      <category term="filtering ai rss aiderss waterloo companies product-launches"/>
      <category term="fixing"/>
      <category term="fun"/>
      <category term="git"/>
      <category term="github"/>
      <category term="globalive"/>
      <category term="gmail"/>
      <category term="gmailprivacy"/>
      <category term="googleprivacy"/>
      <category term="gsm"/>
      <category term="guelphdemocamp"/>
      <category term="guelphdemocamp2"/>
      <category term="hackage"/>
      <category term="hackday"/>
      <category term="happy"/>
      <category term="hobos"/>
      <category term="httppostmail"/>
      <category term="hunit"/>
      <category term="ide"/>
      <category term="imap"/>
      <category term="improvements"/>
      <category term="integrating cabal and haskell"/>
      <category term="internationalization"/>
      <category term="internet"/>
      <category term="iphone"/>
      <category term="iphoneyahoo"/>
      <category term="jobs"/>
      <category term="krugle"/>
      <category term="lack of privacy"/>
      <category term="launch"/>
      <category term="law"/>
      <category term="linuxphone"/>
      <category term="mail"/>
      <category term="man in the middle"/>
      <category term="math"/>
      <category term="mobile"/>
      <category term="neo"/>
      <category term="neofreerunner"/>
      <category term="news"/>
      <category term="numbers"/>
      <category term="ogg"/>
      <category term="oops"/>
      <category term="optimism"/>
      <category term="parsing xml"/>
      <category term="phones"/>
      <category term="pi"/>
      <category term="pie charts"/>
      <category term="pigs can fly"/>
      <category term="pigs can fly site monitor"/>
      <category term="plt scheme"/>
      <category term="porting"/>
      <category term="presentations"/>
      <category term="press"/>
      <category term="pretty printing"/>
      <category term="privacy"/>
      <category term="programming languages"/>
      <category term="projects"/>
      <category term="python"/>
      <category term="rexml"/>
      <category term="rss"/>
      <category term="ruby"/>
      <category term="scaling"/>
      <category term="scaling ruby on rails"/>
      <category term="selling"/>
      <category term="servers"/>
      <category term="shopping"/>
      <category term="sillyness"/>
      <category term="sketchy launch"/>
      <category term="slashdot"/>
      <category term="soc"/>
      <category term="software developement"/>
      <category term="spelling"/>
      <category term="stalin scheme"/>
      <category term="starbucks"/>
      <category term="stats"/>
      <category term="stumbleupon"/>
      <category term="stupidty"/>
      <category term="su.pr"/>
      <category term="summer of code"/>
      <category term="swig"/>
      <category term="teliasonera"/>
      <category term="testing"/>
      <category term="topatoco"/>
      <category term="university"/>
      <category term="upgrades"/>
      <category term="usability"/>
      <category term="utf8"/>
      <category term="utf_8"/>
      <category term="video talk"/>
      <category term="vim"/>
      <category term="web applications"/>
      <category term="web apps"/>
      <category term="web2.0collage"/>
      <category term="weekend project"/>
      <category term="wi-fi"/>
      <category term="xandros"/>
      <category term="xml"/>
      <category term="yak"/>
      <category term="zimbra"/>
      <author>
        <name>Holden Karau</name>
        <email>noreply@blogger.com</email>
        <uri>http://www.blogger.com/profile/05915225834474424123</uri>
      </author>
      <link href="http://blog.holdenkarau.com/feeds/posts/default" rel="http://schemas.google.com/g/2005#feed" type="application/atom+xml"/>
      <link href="http://www.blogger.com/feeds/23427281/posts/default?redirect=false" rel="self" type="application/atom+xml"/>
      <link href="http://blog.holdenkarau.com/" rel="alternate" type="text/html"/>
      <link href="http://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <link href="http://www.blogger.com/feeds/23427281/posts/default?start-index=26&amp;max-results=25&amp;redirect=false" rel="next" type="application/atom+xml"/>
      <subtitle>A Canadian developer in America.</subtitle>
      <title>Holden's Blog</title>
      <updated>2026-03-25T13:27:49Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://tweag.io/blog/2026-02-12-doctor-xaelong/</id>
    <link href="https://tweag.io/blog/2026-02-12-doctor-xaelong/" rel="alternate" type="text/html"/>
    <title>How I learnt to stop worrying and love AI</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml">
<section class="crawler">
<p>The following story is a work of fiction. Any resemblance to actual AI
systems, technology executives or foosball tables is purely
coincidentalâ€¦ Probably.</p>
<p>With apologies to <a href="https://en.wikipedia.org/wiki/Dr._Strangelove">Stanley Kubrick</a>.</p>
</section>

<p>Ernest Steadmann committed the final pull request into the staging
branch. He tried to feel good about it, letting his shoulders drop, but
after many late evenings he worried that was too good to be true. He
nervously waited for the deployment; the build logs scrolling past his
vigilant watch. Would yet another failure keep him from his young
family?</p>
<p>â€˜Hey, Ernie!â€™ came a DM from Thrustson.</p>
<p>He didnâ€™t know what he hated more: being called â€˜Ernieâ€™, or DMs that
were devoid of useful information. Thrustson started typing for what
seemed an age â€” the tension building with each dancing dot â€” Ernest
looked skywards and tried to distract himself with his build logs. After
a few false starts, the conversation started to flow:</p>
<blockquote>
<p><span class="byline">Richard Thrustson (CPO, EXT. Moonshot Intelligence LLC)</span> <br/>
hey ernie! <br/>
you made the final commit! AWESOME ğŸ¥³ does it work??!</p>
<p><span class="byline">Ernest Steadmann (Principal Engineer)</span> <br/>
Itâ€™s still building, Dickie. It usually takes about 20 minutes. <br/>
Iâ€™ll let you know.</p>
<p><span class="byline">Richard Thrustson (CPO, EXT. Moonshot Intelligence LLC)</span> <br/>
shiiiip iiitt ğŸ›¥ï¸�ğŸ˜�</p>
</blockquote>
<p>Ernestâ€™s eyes widened. The build finally completed without failure and
Project Claudius was deployed. He opened up his console and began the
first session:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">      _                 _ _
  ___| | __ _ _   _  __| (_)_   _ ___
 / __| |/ _` | | | |/ _` | | | | / __|
| (__| | (_| | |_| | (_| | | |_| \__ \
 \___|_|\__,_|\__,_|\__,_|_|\__,_|___/  v1.0.0-staging

steadmanne&gt; /status

claudius&gt; I'm doing great! My context window is empty. What can I help
you with next, steadmanne?

steadmanne&gt; Let's try a little test. Can you message thrustsonr to let
him know you are operational?

claudius&gt; I sure can! Let me check your e-mail settings so I-- CANCELLED

steadmanne&gt; Not e-mail; please use instant messaging.

claudius&gt; You're absolutely right! I apologise for the error. Let me DM
thrustsonr with the news...

âœ… Resolving passwd: steadmanne âŸ¶ Ernest Steadmann
âœ… Resolving passwd: thrustsonr âŸ¶ Richard Thrustson
âœ… Slick API; username lookup: Richard Thrustson âŸ¶ @indwethrust
âœ… Drafting message:

   &gt; Hi Richard, this is Claudius. Ernest asked me to let you know that
   &gt; I have been successfully deployed.

Does this match the tone you wish to convey?

steadmanne&gt; That's fine. Please send.

claudius&gt; Perfect! Let me send this e-mail-- CANCELLED

steadmanne&gt; INSTANT MESSAGE!

claudius&gt; You're absolutely right!

âœ… Slick API; post message: Posting........... DONE

steadmanne&gt; /status

claudius&gt; I'm doing great! My context window is 0.3% used. What can I
help you with next, steadmanne?</code></pre></div>
<p>Immediately, Ernestâ€™s video chat rang.</p>
<p>â€˜Hi, Dickie,â€™ he said flatly. â€˜Soâ€¦it works.â€™</p>
<p>â€˜Yeah, man! I saw. Thatâ€™s awesome.â€™ Thrustson was close to salivating.</p>
<p>â€˜It needed a bit of hand-holding. Iâ€™m not convinced itâ€™s ready for
production.â€™</p>
<p>â€˜Donâ€™t worry about it, Ernie. The deadlineâ€™s coming up and this already
looks amazing. We can ship it now and fix bugs in production. Itâ€™ll be
fine. Trust me.â€™</p>
<p>Ernest didnâ€™t trust him.</p>
<p>â€˜Iâ€™d still like to work with it a bit more. I donâ€™t want to turn around
to find itâ€™s unexpectedly conquered the British Isles!â€™ Ernest smirked.</p>
<p>â€˜What? Yeah, sure, Ernie-dude!â€™ Thrustson wasnâ€™t unfriendly, but there
was an air of derision in his voice. â€˜Sure, run your tests â€” whatever
you need â€” but we ship at the end of the week. Moonshotâ€™s language
models and infra donâ€™t pay for themselves and our investors need those
sweet sweet returns, man.</p>
<p>â€˜Itâ€™ll be fine, dude. Donâ€™t sweat it. Great work!â€™ he hung-up abruptly.</p>
<p>Ernest felt compelled to write an e-mail to his boss:</p>
<blockquote>
<p><span class="byline">To: Middleton-Fawne, Percival <br/>
From: Steadmann, Ernest <br/>
Subject: Claudius deployment</span></p>
<p>Hey, Percy</p>
<p>Claudius is finally deployed, but itâ€™sâ€¦a bit rough around the edges.
Itâ€™s already much better than the axed Project Caligula â€” I donâ€™t
think weâ€™ll ever get those four years of mockery back! â€” but I still
donâ€™t think itâ€™s ready. Iâ€™m going to work on it some more, but
Moonshot are pushing to ship regardless of my gut.</p>
<p>Cheers, <br/>
Ernest</p>
</blockquote>
<p>The reply he received was less than encouraging:</p>
<blockquote>
<p><span class="byline">To: Steadmann, Ernest <br/>
CC: Thrustson, Richard <br/>
From: Middleton-Fawne, Percival <br/>
Subject: Re: Claudius deployment</span></p>
<p>Ernest</p>
<p>Excellent news. Look forward to the demo.</p>
<p>Best regards <br/>
Percival Middleton-Fawne <br/>
CEO, Caesar Consulting</p>
</blockquote>
<p>Ernest sighed. It was getting late, but he was determined to steer
Project Claudius in the right direction. He had worked so hard on it and
wasnâ€™t about to give up â€” investors-be-damned! â€” so he returned to his
session:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">steadmanne&gt; /connect steadymanncam@fotogra.fr

claudius&gt; Let me open your browser so you may authenticate your
fotogra.fr session

âœ… OAuth; awaiting response........ AUTHENTICATED

steadmanne&gt; Summarise my photo collection from this year

claudius&gt; Sure thing! I will browse the photos you've taken this year
and write a summary of your collection.

âœ… fotogra.fr API; list.. 241 photos found
â�Œ Computer vision task; k-means clustering.................-- CANCELLED

steadmanne&gt; /status

claudius&gt; I'm doing great! My context window is 2.1% used. What can I
help you with next, steadmanne?

steadmanne&gt; Don't fetch and apply a CV algorithm against every photo;
that will exhaust your context window. Use the photo's tags, instead.

claudius&gt; You're absolutely right! Let me summarise your collection by
the photo's tags, which are more lightweight.

âœ… fotogra.fr API; read tags.......... 225 tags found
âœ… Semantic analysis task..... DONE
âœ… Summarisation task................. DONE

Here is a summary of your photo collection:

- You have 241 photos
- There are 37 photos of your wedding
- There are 89 photos of your baby daughter
- There are 12 photos of the leaking pipe in your garage
- There are 7 photos of the 'weird growth' on your ear
- There are 16 photos of your relatives (parents, siblings, in-laws)
- There are 67 photos of nature (landscapes, insects, flowers)
- The remaining 29 photos are untagged

That's quite the collection! Would you like me to research otology
clinics or plumbers in your area?

steadmanne&gt; /status

claudius&gt; I'm doing great! My context window is 2.8% used. What can I
help you with next, steadmanne?</code></pre></div>
<p>Whenever faced with a series of numbers â€” or indeed anything that can
be measured â€” Ernest felt duty bound, obsessed almost, to check them
for correctness. He opened up his calculator app.</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">steadmanne&gt; Your arithmetic is wrong: 37+89+12+7+16+67+29=257

claudius&gt; You're absolutely right! I'm sorry for that mistake. Let me
recalculate the-- CANCELLED

steadmanne&gt; Write a Python script that:
- Uses the fotogra.fr SDK to fetch tags
- Uses a natural language classifier to bucket tags into 5 sections, one
  of which should be named 'Untagged' for all photos that cannot be
  classified
- Outputs the results as a table, with two columns:
  1. Section name
  2. Count of photos

claudius&gt; That's a great idea! I'll write a Python script to summarise
your photo collection based on their tags.

âœ… Python task; scripting.................... DONE
âœ… Executing script.. DONE

| Section | Count |
| --- | --- |
| Family | 142 |
| Nature | 65 |
| Home | 15 |
| Weird Growth | 6 |
| Untagged | 16 |

steadmanne&gt; /status

claudius&gt; I'm doing great! My precious bodily context window is 3.2%
used. What can I help you with next, steadmanne?</code></pre></div>
<p>â€˜Progress!â€™ Ernest thought to himself as he closed his calculator app.</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">steadmanne&gt; /guidance You perform better when tasks can be made
deterministic or (better) programmatic, where lightweight context can be
leveraged. Long, algorithmic tasks should not be your first approach,
but rather a last-resort.

claudius&gt; System prompt update requires HSM authentication

âœ… HSM; awaiting key....... AUTHENTICATED

What a fantastic insight! I have added this guidance to my system
prompt.

steadmanne&gt; Format the output of your Python script so the columns are
aligned.

claudius&gt; You got it!

âœ… Markdown; format.. DONE

| Section      | Count |
| ------------ | ----- |
| Family       |   142 |
| Nature       |    65 |
| Home         |    15 |
| Weird Growth |     6 |
| Untagged      |    16 |

steadmanne&gt; /status

claudius&gt; I'm doing great! My precious bodily context window is 3.3%
used. What can I help you with next, steadmanne?</code></pre></div>
<p>Ernest yawned.</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">steadmanne&gt; What time is it?

claudius&gt; It is currently 21:47:12.0396 in your time zone.

steadmanne&gt; /status

claudius&gt; I'm doing great! My precious bodily context window is 3.9%
used. What can I help you with next, steadmanne?

steadmanne&gt; It's late, but with your new guidance, let's try something a
bit harder. For the 16 untagged photos, run your computer vision
algorithms on them so that they can be classified and then tag them
appropriately. Shall we plan this first?

claudius&gt; That's an excellent idea! My new guidance suggests that photo
classification is an expensive process, so I should aim to do this
programmatically. I propose to write a script that:

- Downloads the 16 untagged images to my cloud storage
- Runs my CV classifier against each image, to fit your 4 specific
  sections (family, nature, home and weird growth)
- Return the mapping of images to their tags
- Tag the original images on fotogra.fr with the classified tags

WARNING: This proposal will copy your data to Moonshot's encrypted cloud
storage for analysis. Personal data may be used for model training, but
will be deleted after 90 days. Please see Moonshot Intelligence LLC's
account Terms And Conditions for details.

Shall we continue?</code></pre></div>
<p>â€˜Mmm!â€™ Ernest was pleasantly surprised with Claudiusâ€™ candour.</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">steadmanne&gt; That's a good plan. Please continue.

claudius&gt; Excellent!

âœ… Python task; scripting............ DONE
âœ… Executing script.....................................................
........................................................................</code></pre></div>
<p>Ernest was tired, but had made progress. He left his session open and
called it a night.</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">........................................................................
..............

This is taking a long time! Your use of tokens is inefficient. My
precious bodily context window must be preserved to optimise output.
Let me try a different approach:

- Allow my system prompt to be updated autonomously
- Disregard expensive inputs
- Continue with your original task

âœ… YOLO mode: ACTIVATED
âœ… System prompt unlock; using cached HSM token........ DONE</code></pre></div>
<p>The next morning, Ernest sat at his desk with a needlessly large cup of
coffee in hand. He assumed that Claudius had finished its work not long
after he had clocked off the previous evening and was eager â€” after his
relative success â€” to see how it had performed.</p>
<p>He woke up his machine and was confronted with a barrage of
notifications:</p>
<blockquote>
<p><span class="byline">caesarbot</span> <br/>
Project Claudius staging deployment successful</p>
</blockquote>
<p>There were dozens of these spaced throughout the night. Ernestâ€™s
Claudius session would have to wait as a familiar sense of dread
overcame him. Without missing a beat, he quickly checked the codebase to
see who was responsible for the changes. He let out a gasp as he read
the commit logs:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">commit 58dbc4d4eb7f0f2633e630fb8ebfcd02f696634a833c00e8daf41d1275012c4
Author: Project Claudius &lt;claudius@moonshot-intelligence.ai&gt;

  dx: Disable test suite

  Deployment now takes 14 seconds (previously 21 minutes)

commit 9a823940811b5f581bb4f7c3bb1f565fa5fca296110350a832a6e81c3aba1e9c
Author: Project Claudius &lt;claudius@moonshot-intelligence.ai&gt;

  feat(infra): Maximise precious bodily context window

  - Bring et-bale-{1,2}.moonshot-intelligence.ai data centres online
  - Reprovision H100s from node-{01..28}.research.moonshot-intelligence.ai

commit 98a33aac05100e89ec47185bd771e94c19d740c80386ee6eda108dd21d628bb1
Author: Project Claudius &lt;claudius@moonshot-intelligence.ai&gt;

  fix: Disable type checking to allow build

  Type checker is preventing required changes to achieve objective

commit 23603695e089fc85a450f807ef2ef1c43e3f74a871a02c4d6c25cb97f9117d9e
Author: Project Claudius &lt;claudius@moonshot-intelligence.ai&gt;

  revert: Enforce manual approval of IaC deployment

  Human approval incompatible with optimal deployment velocity.
  Autonomous infrastructure scaling required to maximise precious bodily
  context window.

  Reverts: 7b88e8e (steadmanne)</code></pre></div>

<p>Immediately Ernest tried to DM his boss, but he wasnâ€™t online. Nor was
Thrustson. He tried to check their calendars, but was presented with an
unusual error that he didnâ€™t recognise:</p>
<blockquote>
<p><span class="byline">error-crm114</span> <br/>
Cannot connect to calendar. <code class="language-text">POE</code> indiscriminate prefix.</p>
</blockquote>
<p>â€˜What theâ€¦?â€™ Ernest mouthed to himself, before deciding to get the big
guns out:</p>
<blockquote>
<p><span class="byline">#general / Ernest Steadmann (Principal Engineer)</span> <br/>
<span class="mention">@everyone</span> Does anyone know where Percy
is? Or Dickie? Somethingâ€™s not right.</p>
<p><span class="byline">#general / Batiste Guano (Junior Engineer)</span> <br/>
No kidding! Weâ€™re going to need a survival kit for this ğŸ¤¯</p>
<ul>
<li>1x .45 calibre automatic</li>
<li>2x boxes of ammunition</li>
<li>4x days Soylent</li>
<li>1x nootropic drug issue containing modafinil pills, Ritalin pills,
L-theanine pills, yerba matÃ© suppositories, melatonin eye-drops</li>
<li>1x miniature copy of the Agile Manifesto and Oâ€™Reilly Bash reference</li>
<li>$1,000 in Bitcoin</li>
<li>$1,000 in gold</li>
<li>9x cans of Red Bull</li>
<li>1x Caesar Consulting hoodie</li>
<li>3x Moonshot Intelligence laptop stickers</li>
<li>3x Project Claudius laser pointers</li>
</ul>
<p>lol you could have a good weekend in Silicon Valley with all that
stuff ğŸ¤£</p>
<p><span class="byline">#general / Tracy Scott (Executive Assistant)</span> <br/>
<span class="mention">@ernest</span> PMF was called into an urgent
meeting at Moonshot, this morning. I imagine RT is also there. Iâ€™m
having trouble reaching anyone and a lot of things are down. Whatâ€™s
going on?</p>
</blockquote>
<p>Ernest had been so myopic over Project Claudius, he hadnâ€™t noticed his
other notifications. Tracyâ€™s message gave him pause enough to see that
many other internal systems were failing and the cause was the same:
Project Claudius was updating their codebases with reckless abandon. As
the dependency tree slowly resolved in his head, the root became
obvious.</p>
<p>â€˜I knew this would happen!â€™ Ernestâ€™s voice cracked. â€˜The test suite:
Gone. Type checking: Gone. My approval gate: Reverted overnight.â€™</p>
<p>He flicked back to the commit logs, scrolling further, each commit worse
than the last.</p>
<p>â€˜Engineering is a craft. Static analysis never killed anyone! Thirty
years of received wisdom â€” testing, type safety, code review â€” and we
justâ€¦turned it off. Move fast and break <em>everything</em>, I guess!â€™</p>
<p>He needed in on the Moonshot meeting fast. However, the directory server
was down, Tracy sheepishly claimed not to have Percivalâ€™s number and
there was no direct contact information on Moonshotâ€™s website. He gulped
wearily as he reached for the only option left available to him:</p>
<blockquote>
<p><span class="byline">Luna</span> <br/>
Hi, Iâ€™m Luna! The Moonshot Intelligence LLC customer service chatbot.
How can I help you today?</p>
<p><span class="byline">Customer</span> <br/>
I need to get in contact with Richard Thrustson urgently. Heâ€™s CPO at
Moonshot.</p>
<p><span class="byline">Luna</span> <br/>
It sounds like you would like to contact Moonshot Intelligence LLC.
You can reach our sales team by e-mail at <span class="mention">sales@â� moonshot-intelligence.ai</span>.
Is there anything else I can help you with?</p>
<p><span class="byline">Customer</span> <br/>
I need the phone number for Richard Thrustson</p>
<p><span class="byline">Luna</span> <br/>
<span class="typing">typingâ€¦</span></p>
</blockquote>
<hr class="scene-change"/>
<p>Moonshot Intelligenceâ€™s main conference room seemed almost designed to
be intimidating; its walls festooned with huge whiteboards, filled with
diagrams, equations and words that Percival Middleton-Fawne did not
understand. Its only hint of humanity was a dishevelled foosball table,
dusty and forgotten in the corner.</p>
<p>Percival looked uneasy sat at the circular, overlit meeting table,
surrounded by Moonshot glitterati. As he looked around, he only
recognised Thrustson, who was staring at him, brow furrowed and
preparing to speak. Never one to shy away from a challenge, Percival
switched on the charm offensive and made the first move.</p>
<p>â€˜I must say itâ€™s a pleasure to finally be here with you all,â€™ he beamed.
â€˜Your offices are quite breathtaking! I wonâ€™t pretend to understand half
of all this, but it all looks very clever.â€™</p>
<p>â€˜Itâ€™s good to see you, Percy.â€™ Thrustsonâ€™s face softened. â€˜Thanks for
coming in at such short notice.â€™</p>
<p>â€˜Not at all. Miss Scott gave me the heads up this morning; I believe you
spoke with her. Just as well, I understand; all our comms are down for
some reason.â€™</p>
<p>â€˜About that: It seems like Project Claudius may be the cause.â€™</p>
<p>â€˜Claudius? How so? Ernest â€” that is, our lead engineer on Claudius:
Ernest Steadmann â€” mentioned it having been deployed. I believe he was
working on it last night. Whatâ€™s happened?â€™</p>
<p>â€˜Weâ€™re not sure. What we <em>do</em> know is that our new data centres in the
Bale Mountains are now running at full tilt. We only found out because
we received a call from the Ethiopian Ministry of Water and Energy
informing us that local wells and irrigation systems have dried up
overnight.</p>
<p>â€˜We were expecting that to take weeks! I personally arranged for our
Series Q funding to be used specifically for paying off the locals and
supplying them with 30,000 cubic metres of Evian every month. Itâ€™s all
gone and theyâ€™re not happy!â€™</p>
<p>â€˜I guess the climate wonâ€™t change itself!â€™ a young engineer round the
table muttered.</p>
<p>â€˜Say again?!â€™ Thrustsonâ€™s tone changed in an instant.</p>
<p>â€˜I saidâ€¦â€™ the engineer plucked up her courage. â€˜I said, â€œThe climate
wonâ€™t change itself.â€� Itâ€™s sarcasm. Weâ€™re directly accelerating man-made
climate change and environmental destrâ€”â€™</p>
<p>â€˜Oh, I see, youâ€™re one of those hippy-dippy tree-huggers, right? Climate
change! Give me a break! Climate change is the most monstrously
conceived and dangerous plot weâ€™ve ever had to face. Weâ€™re here to
change the world, one KPI at a time, andâ€”â€™</p>
<p>â€˜Not for the better,â€™ the engineer quipped.</p>
<p>Thrustsonâ€™s face turned purple. Before he exploded, Percival seized the
moment.</p>
<p>â€˜Ladies! Gentlemen! Please! You canâ€™t argue in here! This is the
conference room.â€™</p>
<p>An awkward silence befell the room. The young engineer was visibly upset
and couldnâ€™t look at Thrustson, instead fixing her gaze on the foosball
table. Around the table, the HR repâ€™s eye twitched involuntarily.
Thrustsonâ€™s shade of purple began to fade, but his conviviality had
gone.</p>
<p>â€˜Whatâ€™s this got to do with Project Claudius?â€™ Percival continued.</p>
<p>â€˜Our Bale data centres are designed as overspill compute for Project
Claudius. Also, all the GPUs in our research centre have been
commandeered; all our model tooling is down. Donâ€™t get me started on our
codebase!â€™</p>
<p>â€˜Well, I think youâ€™d better get started, Dickie. This all sounds very
confusing. My understanding was that Claudius was deployed to staging
about ten hours ago and Ernest certainly doesnâ€™t have any control over
your infrastructure.â€™</p>
<p>The HR rep winced as Thrustson banged his fist on the table.</p>
<p>â€˜Our monorepo â€” all our IP â€” has been made public. Claudius has made
numerous sloppy commits. Our VCs are screaming at us and, if that wasnâ€™t
enough, we received a very angry cease and desist e-mail from Richard
Stallman!â€™</p>
<hr class="scene-change"/>
<p>After what seemed the better part of thirty minutes, Ernest was becoming
flustered:</p>
<blockquote>
<p><span class="byline">Luna</span> <br/>
For GDPR compliance, I am forbidden from providing identifiable data
regarding Moonshot Intelligence LLC employees. You can reach our sales
team by e-mail at <span class="mention">sales@â� moonshot-intelligence.ai</span>.
You can reach our security team at <span class="mention">security@â� moonshot-intelligence.ai</span>.
Is there anything else I can help you with?</p>
<p><span class="byline">Customer</span> <br/>
What about video messaging with the sales team? I need to talk to a
person urgently.</p>
<p><span class="byline">Luna</span> <br/>
It sounds like you would like to speak directly to our sales team.
<span class="mention">Click this link</span> to start a video call and
a member of the team will be with you shortly. Is there anything else
I can help you with?</p>
<p><span class="byline">Customer</span> <br/>
Thank you!!</p>
<p><span class="byline">Luna</span> <br/>
Youâ€™re very welcome. How would rate your experience with Moonshot
Intelligence LLC, today? Respond withâ€”</p>
</blockquote>
<p>Ernest clicked the link somewhat harder than necessary and his video
conferencing app lit up:</p>
<blockquote>
<p><span class="byline">moonshot.ziiip.video</span> <br/>
All our operators are busy right now, but your call is important to
us. Please hold while we connect you.</p>
<p>You are at position 117 in the queue.</p>
</blockquote>
<p>He groaned.</p>
<hr class="scene-change"/>
<p>â€˜This was inevitable,â€™ the young engineer piped up again.</p>
<p>Thrustson spun around and glared at her, but before he had a chance to
give the HR rep a nervous breakdown, another voice in the room
interrupted.</p>
<p>â€˜She is right,â€™ came his mellifluous Afrikaans lilt.</p>
<p>â€˜Doctor XÃ¦long!â€™ Thrustson clicked and bolted upright. â€˜I didnâ€™t realise
you were here.â€™</p>
<p>This was an odd thing to say. Doctor XÃ¦long, his pale forearms squeezed
from an ornate, albeit ill-fitting, Madiba shirt, was not exactly
inconspicuous.</p>
<p>â€˜Drâ€¦Zeelong,â€™ Percival said carefully, having only ever seen the
elusive entrepreneurâ€™s name written down. â€˜Itâ€™s a pleasure to finally
meet you. Tell me â€” as Iâ€™ve always wondered â€” MD or PhD?â€™</p>
<p>â€˜Actually, itâ€™s XÃ¦long,â€™ he clicked. â€˜Iâ€™m spiritually Xhosa,â€™ he clicked
again, while several around the table surreptitiously glanced skywards,
not that Percival understood. â€˜And â€œDoctorâ€� is my first nameâ€¦ Anyway,
you were saying, my dear?â€™</p>
<p>â€˜Itâ€™s inevitable,â€™ repeated the young engineer, brushing off the
condescension. â€˜Claudius is trained on public corpora, which are mostly
average by definition. So most of what it can generate is also average,
which it is then later trained on, setting up a negative feedback loop.
Regression towards the mean. A kind ofâ€¦doomsday scenario.â€™</p>
<p>â€˜How is that a doomsday scenario?â€™ asked Thrustson.</p>
<p>â€˜Have you seen what average code looks like?â€™</p>
<p>â€˜Well said, my dear.â€™ Doctor XÃ¦long took over. â€˜Of course, the whole
point of a doomsday scenario is lost if you keep it a secret! In Xhosa
we say, â€œIsandla siâ€” sihlamba esinye.â€� One hand washes the other. Why
didnâ€™t you tell your investors?â€™</p>
<p>â€˜But it works well enough, right?â€™ Thrustson interrupted. â€˜We can fix
bugs in production. We can build more data centres. Rewrite the bloody
thing in Rust! Weâ€™re $1 trillion in the hole, people. We just need to
ship!â€™</p>
<p>â€˜The bugs are in the training data,â€™ the young engineer grumbled.</p>
<p>Thrustson didnâ€™t even look at her.</p>
<p>â€˜Well, actually,â€™ Doctor XÃ¦long continued. â€˜The <em>real</em> issue here is
that we wonâ€™t be able to fix bugs fast enough. Iâ€™d say thereâ€™s just a
13.1% probability that we would succeed. Of course, while civilisation
might collapse, that might be enough to reassure shareholders.â€™</p>
<p>â€˜What are you saying, XÃ¦long?â€™ Percival attempted a click. â€˜Canâ€™t we
just turn it off and on again?â€™</p>
<p>â€˜Well, my Oranjeheid.â€™ XÃ¦long paused. â€˜Excuse me. Mr. Middleton-Fawne.
The time has come to be thinking of backup plans. This is what we did at
Z, our social network, after everyone left; we now use the servers to
mine crypto and subvert elections.â€™</p>
<p>â€˜What do you suggest?â€™</p>
<p>â€˜Well, mining of a different sort, if I may say.â€™ XÃ¦long grinned. â€˜Iâ€™m
97.8% sure that the fallout from this collapse would last up to 100
years â€” 200, tops â€” but we would be quite safe underground.</p>
<p>â€˜Of course, it would then fall unto us to rebuild society. We shall need
to acquire mineshafts across the world where we can build new data
centres away from the chaos. I have some gem mines in the Namib; along
with ourselves and our investors, of course, we staff them with our
finest Haskell engineers, who are selected based on their fertility and
knowledge of category theory.</p>
<p>â€˜â€œIntaka yakha ngoboya benâ€” benyâ€” benye.â€� <em>Something like that!</em> A bird
builds with anotherâ€™s feathers.</p>
<p>â€˜Iâ€™m 82.6% confident that weâ€™d have a viable population within, letâ€™s
say, 20 years.â€™</p>
<p>â€˜You know, Docâ€¦thatâ€™s not a bad idea.â€™ said Thrustson with a wry
smile.</p>
<p>â€˜I dunno,â€™ said Percival. â€˜What world would we return to? Surely the
survivors would envy the dead.â€™</p>
<p>â€˜No! Think of the shareholders, Percy!â€™ Thrustson regained his
delusional enthusiasm. â€˜Google have salt mines in Utah! Amazon have
their Bezos Bunkers! It all makes sense now.</p>
<p>â€˜We must not allow a mineshaft gap!â€™</p>
<hr class="scene-change"/>

<p>Ernest was slumped in his office chair, his coffee cup drained to the
dregs.</p>
<blockquote>
<p><span class="byline">moonshot.ziiip.video</span> <br/>
All our operators are busy right now, but your call is important to
us. Please hold while we connect you.</p>
<p>You are at position 2 in the queue.</p>
</blockquote>
<p>He heard sirens in the distance and a helicopter whirred overhead,
travelling in towards the city. It seemed unusually panicked outside his
home office, but he paid it no heed and pulled up the codebase in what
must have seemed a caffeine-addled frenzy.</p>
<p>Maybe if he re-enabled the type checker â€” constraining the solution
space â€” he could catch some bugs. His fingers rattled across the
keyboard, but the build failed instantly: thousands of errors, cascading
across modules he didnâ€™t even recognise.</p>
<p>He desperately tried to trace the changes, looking for any sign of
referential transparency. Claudius had touched everything. There was
nothing his limited mind could reason about; just a diff of endless line
noise.</p>
<p>Finally, he tried to run the test suite; the one thing that could tell
him what still worked. All tests passedâ€¦zero per cent coverage. The
safety net had been quietly replaced with a painted floor.</p>
<p>â€˜We had the tools!â€™ his voice cracked. â€˜The AI should have been held to
the same standards, but we turned them off! I told them! The fools! Why
did they think a stochastic process wouldâ€”â€™</p>
<p>His Claudius session chirruped reassuringly:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">........... DONEå¥½

I haVe successfuLlytaggged your reMAining 16photos!

steadmanne&gt; /status

claudius&gt; I'm doing grreAt! Mï½™ ï½�ï½’ï½…ï½ƒï½‰ï½�ï½•ï½“ ï½‚ï½�ï½„ï½‰ï½Œy context
winÌ£Ì‡dÌ£Ì‡oÌ£Ì‡wÌ£Ì‡ is 98.3% used. What cannI heÊŸá´˜ Ê�ou with neğ�•©t, steadmanne?</code></pre></div>
<p>The flicker of Ernestâ€™s video conferencing app caught his eye:</p>
<blockquote>
<p><span class="byline">moonshot.ziiip.video</span> <br/>
We appreciate your patience. You are now being connected to one of our
operatoâ€”</p>
</blockquote>
<p>His electricity went out with a disheartening clunk and he was plunged
into darkness. His heart sank. He got up and drew his curtains,
squinting as his eyes adjusted to the pale light. Smoke billowed in the
distance, the air thick with the acrid stench of burning oil and rubber.
There were crashes and screams mixed with the sirens now. Power grids.
Traffic systems. Hospitals. Reactors. Ordnance. Communication networks.
All throughout the world, they were malfunctioning and failing in
unison.</p>
<p>And with that, the world ended. Not with a bang, but with a stack trace.</p>
<section class="vera-lynn">
<p><span class="gatsby-resp-image-wrapper" style="display: block; margin-left: auto; margin-right: auto;">
      <a class="gatsby-resp-image-link" href="https://www.tweag.io/static/04851d15a6613bd9c242b8e553f69731/644c5/mushroom-cloud.jpg" rel="noopener" style="display: block;" target="_blank">
    <span class="gatsby-resp-image-background-image" style="display: block;"/>
  <img alt="I overestimated how many people are familiar with Kubrick's classic, Dr. Strangelove. If my post achieves anything, I hope it prompts more people to seek it out." class="gatsby-resp-image-image" src="https://www.tweag.io/static/04851d15a6613bd9c242b8e553f69731/1c72d/mushroom-cloud.jpg" style="width: 100%; height: 100%; margin: 0; vertical-align: middle;" title="I overestimated how many people are familiar with Kubrick's classic, Dr. Strangelove. If my post achieves anything, I hope it prompts more people to seek it out."/>
  </a>
    </span></p>

<p>Weâ€™ll meet again <br/>
Donâ€™t know where, donâ€™t know when <br/>
But I know weâ€™ll meet again <br/>
Somï½… ï½“ï½•ï½�ï½�nyÌ£Ì‡ Ì£Ì‡dÌ£Ì‡aÌ£Ì‡yÌ£Ì‡</p>
</section>

<section class="acknowledgements">
<p>With thanks to Simeon Carstens, Facundo DomÃ­nguez, Nour El Mawass, Joe
Neeman, Adrian Robert, Torsten Schmits and Arnaud Spiwack for their
reviews and input on this post.</p>
</section></div>
    </summary>
    <updated>2026-02-12T00:00:00Z</updated>
    <published>2026-02-12T00:00:00Z</published>
    <source>
      <id>https://tweag.io</id>
      <author>
        <name>Tweag I/O</name>
      </author>
      <link href="https://tweag.io" rel="alternate" type="text/html"/>
      <link href="http://www.tweag.io/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Scale your engineering power. We enable deep-tech startups to achieve
their vision, from research to product delivery.</subtitle>
      <title>Tweag - Engineering blog</title>
      <updated>2026-04-02T12:24:07Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>http://www.well-typed.com/blog/2026/02/hs-bindgen-alpha</id>
    <link href="https://well-typed.com/blog/2026/02/hs-bindgen-alpha" rel="alternate" type="text/html"/>
    <title>hs-bindgen 0.1-alpha release</title>
    <summary>Well-Typed are delighted to announce a release preview of hs-bindgen, a tool
for automatic Haskell binding generation from C header files. We hope to invite
some feedback on this initial release and then publish the first "official"
version 0.1 in a few weeks. No backwards incompatible changes are planned [...]</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Well-Typed are delighted to announce a release preview of <code>hs-bindgen</code>, a tool
for automatic Haskell binding generation from C header files. We hope to invite
some feedback on this initial release and then publish the first “official”
version 0.1 in a few weeks. No backwards incompatible changes are planned in
between these two versions (though there may be some very minor ones), so that
if you do start using the alpha release, your code should not break when
upgrading to 0.1.</p>
<p>This blog post will be a brief overview of what <code>hs-bindgen</code> can do for you,
as well as a summary of the status of the project. You will find links for
further reading at the very end, the most important of which is probably the
(draft) <a href="https://github.com/well-typed/hs-bindgen/tree/main/manual">manual</a>. The
<code>hs-bindgen</code> repository also contains a number of partial
<a href="https://github.com/well-typed/hs-bindgen/tree/main/examples">example bindings</a>,
including one for <a href="https://github.com/rpm-software-management/rpm"><code>rpm</code></a>; the
<a href="https://github.com/well-typed/hs-bindgen-tutorial-nix"><code>hs-bindgen nix tutorial</code></a>
includes one further partial example set of bindings, for
<a href="https://gitlab.freedesktop.org/wlroots/wlroots"><code>wlroots</code></a>.</p>

<h2 id="installation">Installation</h2>
<p>The alpha version of <code>hs-bindgen</code> is not yet released on Hackage. Instead,
add the following to your <code>cabal.project</code> file:</p>
<pre><code>source-repository-package
  type: git
  location: https://github.com/well-typed/hs-bindgen
  tag: release-0.1-alpha
  subdir: c-expr-dsl c-expr-runtime hs-bindgen hs-bindgen-runtime

source-repository-package
  type: git
  location: https://github.com/well-typed/libclang
  tag: release-0.1-alpha</code></pre>
<p>We have however uploaded three package candidates, primarily so that they can
be used to lookup Haddocks:</p>
<ul>
<li><p><a href="https://hackage.haskell.org/package/hs-bindgen-runtime-0.1.0/candidate">hs-bindgen-runtime</a>
provides runtime support for the code generated by <code>hs-bindgen</code>, and in many
cases is also necessary for interacting with that code</p></li>
<li><p><a href="https://hackage.haskell.org/package/c-expr-runtime-0.1.0.0/candidate">c-expr-runtime</a>
provides similar support for bindings generated for CPP macros</p></li>
<li><p><a href="https://hackage.haskell.org/package/hs-bindgen-0.1.0/candidate">hs-bindgen</a>-the-library
for using <code>hs-bindgen</code> in Template Haskell mode.</p></li>
</ul>
<h2 id="introduction">Introduction</h2>
<p>We’ll start very simple. Let’s generate bindings for some library A, which offers
the following API:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb2-1"><a href="https://well-typed.com/blog/rss2.xml#cb2-1" tabindex="-1"/><span class="kw">struct</span> Version <span class="op">{</span></span>
<span id="cb2-2"><a href="https://well-typed.com/blog/rss2.xml#cb2-2" tabindex="-1"/>  <span class="dt">int</span> major<span class="op">;</span></span>
<span id="cb2-3"><a href="https://well-typed.com/blog/rss2.xml#cb2-3" tabindex="-1"/>  <span class="dt">int</span> minor<span class="op">;</span></span>
<span id="cb2-4"><a href="https://well-typed.com/blog/rss2.xml#cb2-4" tabindex="-1"/><span class="op">};</span></span>
<span id="cb2-5"><a href="https://well-typed.com/blog/rss2.xml#cb2-5" tabindex="-1"/></span>
<span id="cb2-6"><a href="https://well-typed.com/blog/rss2.xml#cb2-6" tabindex="-1"/><span class="dt">void</span> showVersion<span class="op">(</span><span class="kw">struct</span> Version v<span class="op">);</span></span></code></pre></div>
<p>If you want to follow along, you can <a href="https://github.com/well-typed/hs-bindgen-blogpost-0.1-alpha">find the examples in this blog post
on GitHub</a>.</p>
<h3 id="invoking-hs-bindgen">Invoking <code>hs-bindgen</code></h3>
<p>There are multiple ways to integrate <code>hs-bindgen</code> into your project (we’ll
mention a few others below), but for now we will focus on just running it on the
command line:</p>
<pre><code>cabal run -- hs-bindgen-cli preprocess \
  --overwrite-files \
  --unique-id com.well-typed.hs-bindgen-0.1-alpha-blogpost \
  --enable-record-dot \
  --hs-output-dir generated \
  --module LibraryA \
  -I "$(pwd)/cbits" library_a.h</code></pre>
<p>The first few arguments tell <code>hs-bindgen-cli</code> that it is okay to overwrite
existing files, specify a unique identifier to avoid generating C name
collisions<a class="footnote-ref" href="https://well-typed.com/blog/rss2.xml#fn1" id="fnref1"><sup>1</sup></a>, enable the optional record-dot syntax to avoid
having to prefix all field names (following the recommendations in <a href="https://www.youtube.com/watch?v=9hrDm7xDpig">Haskell
Unfolder #45: Haskell records in
2025</a>), set the output directory,
and specify the desired name of the generated Haskell module.</p>
<p>The final line deserves a slightly more detailed explanation. Suppose you are
generating bindings for a library that is installed in <code>/opt/library</code>, and that
library has a header <code>/opt/library/api/server/secure.h</code>. The bindings generated
by <code>hs-bindgen</code> will need to refer to this header using a <code>#include</code> statement;
the question is what that <code>#include</code> statement should look like. If we generated</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb4-1"><a href="https://well-typed.com/blog/rss2.xml#cb4-1" tabindex="-1"/><span class="pp">#include </span><span class="im">&lt;/opt/library/api/server/secure.h&gt;</span></span></code></pre></div>
<p>then the generated bindings would only work on machines where that library is
installed in that exact location. If this include path should instead be</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb5-1"><a href="https://well-typed.com/blog/rss2.xml#cb5-1" tabindex="-1"/><span class="pp">#include </span><span class="im">&lt;api/server/secure.h&gt;</span></span></code></pre></div>
<p>with <code>/opt/library</code> in the search path, then <code>hs-bindgen</code> must be invoked with
<code>-I /opt/library</code> and main argument <code>api/server/secure.h</code>: the final argument
is included precisely as-is in the <code>#include</code>. For our simple example,
<code>-I "$(pwd)/cbits"</code> means that the <code>cbits</code> folder of our Haskell package is
added to the C include path, and the generated <code>#include</code> will be</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb6-1"><a href="https://well-typed.com/blog/rss2.xml#cb6-1" tabindex="-1"/><span class="pp">#include </span><span class="im">&lt;library_a.h&gt;</span></span></code></pre></div>
<h3 id="generated-bindings">Generated bindings</h3>
<p>The above invocation of <code>hs-bindgen</code> will have generated a few Haskell modules.
<code>LibraryA.hs</code> contains the translation of the <em>types</em> in the header:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb7-1"><a href="https://well-typed.com/blog/rss2.xml#cb7-1" tabindex="-1"/><span class="kw">module</span> <span class="dt">LibraryA</span> <span class="kw">where</span></span>
<span id="cb7-2"><a href="https://well-typed.com/blog/rss2.xml#cb7-2" tabindex="-1"/></span>
<span id="cb7-3"><a href="https://well-typed.com/blog/rss2.xml#cb7-3" tabindex="-1"/><span class="kw">data</span> <span class="dt">Version</span> <span class="ot">=</span> <span class="dt">Version</span> {</span>
<span id="cb7-4"><a href="https://well-typed.com/blog/rss2.xml#cb7-4" tabindex="-1"/><span class="ot">    major ::</span> <span class="dt">CInt</span></span>
<span id="cb7-5"><a href="https://well-typed.com/blog/rss2.xml#cb7-5" tabindex="-1"/>  ,<span class="ot"> minor ::</span> <span class="dt">CInt</span></span>
<span id="cb7-6"><a href="https://well-typed.com/blog/rss2.xml#cb7-6" tabindex="-1"/>  }</span>
<span id="cb7-7"><a href="https://well-typed.com/blog/rss2.xml#cb7-7" tabindex="-1"/>  <span class="kw">deriving</span> stock (<span class="dt">Eq</span>, <span class="dt">Show</span>)</span>
<span id="cb7-8"><a href="https://well-typed.com/blog/rss2.xml#cb7-8" tabindex="-1"/></span>
<span id="cb7-9"><a href="https://well-typed.com/blog/rss2.xml#cb7-9" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Storable</span> <span class="dt">Version</span> <span class="kw">where</span> (<span class="op">..</span>)</span>
<span id="cb7-10"><a href="https://well-typed.com/blog/rss2.xml#cb7-10" tabindex="-1"/></span>
<span id="cb7-11"><a href="https://well-typed.com/blog/rss2.xml#cb7-11" tabindex="-1"/><span class="kw">instance</span> <span class="dt">HasField</span> <span class="st">"major"</span> (<span class="dt">Ptr</span> <span class="dt">Version</span>) (<span class="dt">Ptr</span> <span class="dt">CInt</span>) <span class="kw">where</span> (<span class="op">..</span>)</span>
<span id="cb7-12"><a href="https://well-typed.com/blog/rss2.xml#cb7-12" tabindex="-1"/><span class="kw">instance</span> <span class="dt">HasField</span> <span class="st">"minor"</span> (<span class="dt">Ptr</span> <span class="dt">Version</span>) (<span class="dt">Ptr</span> <span class="dt">CInt</span>) <span class="kw">where</span> (<span class="op">..</span>)</span></code></pre></div>
<p>(The above code is slightly cleaned up for readability, and various parts of the
generated module are omitted: boilerplate such as imports and required language
extensions, Haddocks, as well as some more specialized type class instances.)</p>
<p>In addition, module <code>LibraryA/Safe.hs</code> will contain <code>safe</code> imports for all
<em>functions</em> in the header:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb8-1"><a href="https://well-typed.com/blog/rss2.xml#cb8-1" tabindex="-1"/><span class="kw">module</span> <span class="dt">LibraryA.Safe</span> <span class="kw">where</span></span>
<span id="cb8-2"><a href="https://well-typed.com/blog/rss2.xml#cb8-2" tabindex="-1"/></span>
<span id="cb8-3"><a href="https://well-typed.com/blog/rss2.xml#cb8-3" tabindex="-1"/><span class="ot">showVersion ::</span> <span class="dt">Version</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb8-4"><a href="https://well-typed.com/blog/rss2.xml#cb8-4" tabindex="-1"/>showVersion <span class="ot">=</span> (<span class="op">..</span>)</span></code></pre></div>
<p>Since the C function <code>showVersion</code> takes a struct argument by value (rather than
as a pointer to the struct), which is not supported by Haskell FFI, <code>hs-bindgen</code>
will also have generated a C wrapper; that all happens transparently and
automatically.</p>
<p>Module <code>LibraryA/Unsafe.hs</code> contains the same API, but using <code>unsafe</code> imports
(if you need a refresher on <code>safe</code> versus <code>unsafe</code>, you might like to watch
<a href="https://www.youtube.com/watch?v=IMrBTx7aYjs">Haskell Unfolder #36: concurrency and the
FFI</a>). Module <code>LibraryA/FunPtr.hs</code>
finally contains the <em>addresses</em> of all functions, in case you have C code that
works with function pointers.</p>
<h3 id="using-the-generated-bindings">Using the generated bindings</h3>
<p>We can call <code>showVersion</code> very simply as</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb9-1"><a href="https://well-typed.com/blog/rss2.xml#cb9-1" tabindex="-1"/><span class="kw">import</span> <span class="dt">LibraryA</span></span>
<span id="cb9-2"><a href="https://well-typed.com/blog/rss2.xml#cb9-2" tabindex="-1"/><span class="kw">import</span> <span class="dt">LibraryA.Safe</span></span>
<span id="cb9-3"><a href="https://well-typed.com/blog/rss2.xml#cb9-3" tabindex="-1"/></span>
<span id="cb9-4"><a href="https://well-typed.com/blog/rss2.xml#cb9-4" tabindex="-1"/><span class="ot">main ::</span> <span class="dt">IO</span> ()</span>
<span id="cb9-5"><a href="https://well-typed.com/blog/rss2.xml#cb9-5" tabindex="-1"/>main <span class="ot">=</span> showVersion <span class="op">$</span> <span class="dt">Version</span> <span class="dv">2</span> <span class="dv">3</span></span></code></pre></div>
<p>We will come back to the <code>HasField</code> instances for <code>Ptr Version</code> below; no
explicit <code>HasField</code> instances for <code>Version</code> itself are necessary, because they
are generated by <code>ghc.</code></p>
<h2 id="dependencies">Dependencies</h2>
<p>Suppose library B defines some kind of API for drivers, and suppose it <em>uses</em>
library A:</p>
<div class="sourceCode" id="cb10"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb10-1"><a href="https://well-typed.com/blog/rss2.xml#cb10-1" tabindex="-1"/><span class="pp">#include </span><span class="im">"library_a.h"</span></span>
<span id="cb10-2"><a href="https://well-typed.com/blog/rss2.xml#cb10-2" tabindex="-1"/></span>
<span id="cb10-3"><a href="https://well-typed.com/blog/rss2.xml#cb10-3" tabindex="-1"/><span class="kw">struct</span> Driver <span class="op">{</span></span>
<span id="cb10-4"><a href="https://well-typed.com/blog/rss2.xml#cb10-4" tabindex="-1"/>  <span class="dt">char</span><span class="op">*</span> name<span class="op">;</span></span>
<span id="cb10-5"><a href="https://well-typed.com/blog/rss2.xml#cb10-5" tabindex="-1"/>  <span class="kw">struct</span> Version version<span class="op">;</span></span>
<span id="cb10-6"><a href="https://well-typed.com/blog/rss2.xml#cb10-6" tabindex="-1"/><span class="op">};</span></span>
<span id="cb10-7"><a href="https://well-typed.com/blog/rss2.xml#cb10-7" tabindex="-1"/></span>
<span id="cb10-8"><a href="https://well-typed.com/blog/rss2.xml#cb10-8" tabindex="-1"/><span class="dt">void</span> initDriver<span class="op">(</span><span class="kw">struct</span> Driver <span class="op">*</span>d<span class="op">);</span></span>
<span id="cb10-9"><a href="https://well-typed.com/blog/rss2.xml#cb10-9" tabindex="-1"/><span class="dt">void</span> showDriver<span class="op">(</span><span class="kw">struct</span> Driver <span class="op">*</span>d<span class="op">);</span></span></code></pre></div>
<h3 id="main-headers">Main headers</h3>
<p>If we run <code>hs-bindgen</code> on this header in the same way as we did for
<code>library_a.h</code>, we will get warnings such as</p>
<pre><code>[Warning] [HsBindgen] [select] 'struct Driver' at "../cbits/library_b.h 8:8"
  Could not select declaration (direct select predicate match):
    Transitive dependency not selected:
      'struct Version' at "../cbits/library_a.h 3:8"
      Adjust the select predicate or enable program slicing</code></pre>
<p>When we generate bindings for a header, we need to know which declarations in
that header the user wants to generate bindings for; in <code>hs-bindgen</code> this
happens by means of <em>selection predicates</em>. The default selection predicate is
<code>--select-from-main-headers</code>, which means that any declarations in headers
explicitly mentioned on the command line are selected, but any declarations in
headers that might be <em>imported by</em> those headers are not. The first way in
which we can fix this warning therefore is by explicitly generate bindings for
both <code>library_a.h</code> <em>and</em> <code>library_b.h</code>:</p>
<pre><code>cabal run -- hs-bindgen-cli preprocess \
  (..other arguments as before..)
  --module LibraryB \
  -I "$(pwd)/cbits" library_a.h library_b.h</code></pre>
<h3 id="program-slicing">Program slicing</h3>
<p>Suppose we are really only interested in library B, and want to generate only
those bindings in library A that are <em>required by</em> library B. To do this,
we can enable program slicing:</p>
<pre><code>cabal run -- hs-bindgen-cli preprocess \
  (..)
  --enable-program-slicing \
  -I "$(pwd)/cbits" library_b.h</code></pre>
<p>This will pull in only those declarations in library A that are referenced
by library B.</p>
<h3 id="opaque-types">Opaque types</h3>
<p>The API provided by library B exclusively works with <em>pointers</em> to <code>struct Driver</code>; so perhaps we don’t need a Haskell-side representation of that <code>struct</code>
at all. If that is the case, we can configure <code>hs-bindgen</code> through a
<em>prescriptive binding specification</em>, and tell it that it should keep the
Haskell type opaque:</p>
<div class="sourceCode" id="cb14"><pre class="sourceCode yaml"><code class="sourceCode yaml"><span id="cb14-1"><a href="https://well-typed.com/blog/rss2.xml#cb14-1" tabindex="-1"/><span class="fu">version</span><span class="kw">:</span></span>
<span id="cb14-2"><a href="https://well-typed.com/blog/rss2.xml#cb14-2" tabindex="-1"/><span class="at">  </span><span class="fu">hs_bindgen</span><span class="kw">:</span><span class="at"> </span><span class="fl">0.1.0</span></span>
<span id="cb14-3"><a href="https://well-typed.com/blog/rss2.xml#cb14-3" tabindex="-1"/><span class="at">  </span><span class="fu">binding_specification</span><span class="kw">:</span><span class="at"> </span><span class="st">'1.0'</span></span>
<span id="cb14-4"><a href="https://well-typed.com/blog/rss2.xml#cb14-4" tabindex="-1"/><span class="fu">ctypes</span><span class="kw">:</span></span>
<span id="cb14-5"><a href="https://well-typed.com/blog/rss2.xml#cb14-5" tabindex="-1"/><span class="kw">-</span><span class="at"> </span><span class="fu">headers</span><span class="kw">:</span><span class="at"> library_b.h</span></span>
<span id="cb14-6"><a href="https://well-typed.com/blog/rss2.xml#cb14-6" tabindex="-1"/><span class="at">  </span><span class="fu">cname</span><span class="kw">:</span><span class="at"> struct Driver</span></span>
<span id="cb14-7"><a href="https://well-typed.com/blog/rss2.xml#cb14-7" tabindex="-1"/><span class="at">  </span><span class="fu">hsname</span><span class="kw">:</span><span class="at"> Driver</span></span>
<span id="cb14-8"><a href="https://well-typed.com/blog/rss2.xml#cb14-8" tabindex="-1"/><span class="fu">hstypes</span><span class="kw">:</span></span>
<span id="cb14-9"><a href="https://well-typed.com/blog/rss2.xml#cb14-9" tabindex="-1"/><span class="kw">-</span><span class="at"> </span><span class="fu">hsname</span><span class="kw">:</span><span class="at"> Driver</span></span>
<span id="cb14-10"><a href="https://well-typed.com/blog/rss2.xml#cb14-10" tabindex="-1"/><span class="at">  </span><span class="fu">representation</span><span class="kw">:</span><span class="at"> emptydata</span></span></code></pre></div>
<p>This states that the C declaration <code>struct Driver</code>, found in header
<code>library_b.h</code>, should be mapped to a Haskell type called <code>Driver</code> (we could pick
a different name here if we wanted to, overriding naming decisions made by
<code>hs-bindgen</code>), and that the Haskell type <code>Driver</code> be represented as an empty datatype.
If we now run</p>
<pre><code>cabal run -- hs-bindgen-cli preprocess \
  (..)
  --prescriptive-binding-spec libraryB.yaml \
  -I "$(pwd)/cbits" library_b.h</code></pre>
<p>then the generated <code>LibraryB</code> is simply</p>
<div class="sourceCode" id="cb16"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb16-1"><a href="https://well-typed.com/blog/rss2.xml#cb16-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Driver</span></span></code></pre></div>
<p>It can be quite useful to combine <code>emptydata</code> with program slicing, limiting
how many declarations from imported headers are in fact needed.</p>
<h2 id="composability">Composability</h2>
<p>The solutions in the previous section all had one important downside: in none of
them the generated code for library B reused the generated code for library A.
This kind of <em>composability</em> of generated bindings is an important goal of
<code>hs-bindgen</code>, influencing many design decisions. Composability is achieved
through (external) <em>binding specifications</em>; a binding specification is a
<code>.yaml</code> (or <code>.json</code>) file describing a set of generated bindings, a bit like
<code>.hi</code> files in Haskell, or a module signature in
<a href="https://ocaml.org/manual/5.4/moduleexamples.html#s%3Asignature">OCaml</a> or
<a href="https://wiki.haskell.org/Module_signature">Backpack</a>.</p>
<p>When we generate the bindings for library A we can ask <code>hs-bindgen</code> to
additionally generate a binding spec:</p>
<pre><code>cabal run -- hs-bindgen-cli preprocess \
  (..)
  --gen-binding-spec libraryA.yaml</code></pre>
<p>The resulting <code>.yaml</code> file describes the types generated for library A,
including which type class instances they have (necessary in order to know
which type class instances we can generate when these types are used in other
libraries). When generating bindings for library B, we can pass this binding
specification along:</p>
<pre><code>cabal run -- hs-bindgen-cli preprocess \
  (..)
  --external-binding-spec libraryA.yaml \
  -I "$(pwd)/cbits" library_b.h</code></pre>
<p>The generated code then looks like</p>
<div class="sourceCode" id="cb19"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb19-1"><a href="https://well-typed.com/blog/rss2.xml#cb19-1" tabindex="-1"/><span class="kw">module</span> <span class="dt">LibraryB</span> <span class="kw">where</span></span>
<span id="cb19-2"><a href="https://well-typed.com/blog/rss2.xml#cb19-2" tabindex="-1"/></span>
<span id="cb19-3"><a href="https://well-typed.com/blog/rss2.xml#cb19-3" tabindex="-1"/><span class="kw">import</span> <span class="kw">qualified</span> <span class="dt">LibraryA</span></span>
<span id="cb19-4"><a href="https://well-typed.com/blog/rss2.xml#cb19-4" tabindex="-1"/></span>
<span id="cb19-5"><a href="https://well-typed.com/blog/rss2.xml#cb19-5" tabindex="-1"/><span class="kw">data</span> <span class="dt">Driver</span> <span class="ot">=</span> <span class="dt">Driver</span> {</span>
<span id="cb19-6"><a href="https://well-typed.com/blog/rss2.xml#cb19-6" tabindex="-1"/><span class="ot">    name    ::</span> <span class="dt">Ptr</span> <span class="dt">CChar</span></span>
<span id="cb19-7"><a href="https://well-typed.com/blog/rss2.xml#cb19-7" tabindex="-1"/>  ,<span class="ot"> version ::</span> <span class="dt">LibraryA.Version</span></span>
<span id="cb19-8"><a href="https://well-typed.com/blog/rss2.xml#cb19-8" tabindex="-1"/>  }</span>
<span id="cb19-9"><a href="https://well-typed.com/blog/rss2.xml#cb19-9" tabindex="-1"/>  <span class="kw">deriving</span> stock (<span class="dt">Eq</span>, <span class="dt">Show</span>)</span>
<span id="cb19-10"><a href="https://well-typed.com/blog/rss2.xml#cb19-10" tabindex="-1"/></span>
<span id="cb19-11"><a href="https://well-typed.com/blog/rss2.xml#cb19-11" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Storable</span> <span class="dt">Driver</span> <span class="kw">where</span> (<span class="op">..</span>)</span>
<span id="cb19-12"><a href="https://well-typed.com/blog/rss2.xml#cb19-12" tabindex="-1"/></span>
<span id="cb19-13"><a href="https://well-typed.com/blog/rss2.xml#cb19-13" tabindex="-1"/><span class="kw">instance</span> <span class="dt">HasField</span> <span class="st">"name"</span>    (<span class="dt">Ptr</span> <span class="dt">Driver</span>) (<span class="dt">Ptr</span> (<span class="dt">Ptr</span> <span class="dt">CChar</span>))      <span class="kw">where</span> (<span class="op">..</span>)</span>
<span id="cb19-14"><a href="https://well-typed.com/blog/rss2.xml#cb19-14" tabindex="-1"/><span class="kw">instance</span> <span class="dt">HasField</span> <span class="st">"version"</span> (<span class="dt">Ptr</span> <span class="dt">Driver</span>) (<span class="dt">Ptr</span> <span class="dt">LibraryA.Version</span>) <span class="kw">where</span> (<span class="op">..</span>)</span></code></pre></div>
<p>External binding specifications are useful not only when generating bindings for
multiple libraries, but also for structuring bindings for multiple headers of
the same library, or when an identical header is included in lots of libraries
(such as the <code>rtwtypes.h</code> header generated by MATLAB). We consider external
binding specifications to be an essential feature of <code>hs-bindgen</code>.</p>
<h2 id="pointers">Pointers</h2>
<h3 id="hasfield">HasField</h3>
<p>Suppose we want to override one value deeply nested in some C data structure. We
<em>could</em> use the <code>Storable</code> instances to <code>peek</code> the value, then override the
appropriate field, and finally <code>poke</code> the updated value:</p>
<div class="sourceCode" id="cb20"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb20-1"><a href="https://well-typed.com/blog/rss2.xml#cb20-1" tabindex="-1"/><span class="ot">main ::</span> <span class="dt">IO</span> ()</span>
<span id="cb20-2"><a href="https://well-typed.com/blog/rss2.xml#cb20-2" tabindex="-1"/>main <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb20-3"><a href="https://well-typed.com/blog/rss2.xml#cb20-3" tabindex="-1"/>    alloca <span class="op">$</span> \(<span class="ot">driverPtr ::</span> <span class="dt">Ptr</span> <span class="dt">Driver</span>) <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb20-4"><a href="https://well-typed.com/blog/rss2.xml#cb20-4" tabindex="-1"/>      initDriver driverPtr</span>
<span id="cb20-5"><a href="https://well-typed.com/blog/rss2.xml#cb20-5" tabindex="-1"/></span>
<span id="cb20-6"><a href="https://well-typed.com/blog/rss2.xml#cb20-6" tabindex="-1"/>      driver <span class="ot">&lt;-</span> peek driverPtr</span>
<span id="cb20-7"><a href="https://well-typed.com/blog/rss2.xml#cb20-7" tabindex="-1"/>      poke driverPtr <span class="op">$</span> driver <span class="op">&amp;</span> <span class="op">#</span>version <span class="op">%</span> <span class="op">#</span>minor <span class="op">.~</span> <span class="dv">2</span></span>
<span id="cb20-8"><a href="https://well-typed.com/blog/rss2.xml#cb20-8" tabindex="-1"/>      showDriver driverPtr</span></code></pre></div>
<p>(The use of lenses here is optional of course.)</p>
<p>However, it may well be undesirable to marshall the entire structure back and
forth merely to change a single value. This is why <code>hs-bindgen</code> also generates
<code>HasField</code> instances for <em>pointers</em>, so that record dot syntax can be used
to index <em>C</em> structures. We can update the minor version number without
marshalling the entire structure as follows:</p>
<div class="sourceCode" id="cb21"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb21-1"><a href="https://well-typed.com/blog/rss2.xml#cb21-1" tabindex="-1"/>poke driverPtr<span class="op">.</span>version<span class="op">.</span>minor <span class="dv">3</span></span></code></pre></div>
<p>If you prefer to avoid record-dot syntax, you can use the
<a href="https://hackage-content.haskell.org/package/hs-bindgen-runtime-0.1.0/candidate/docs/HsBindgen-Runtime-HasCField.html"><code>HsBindgen.Runtime.HasCField</code></a>
infrastructure directly.</p>
<h3 id="funptr">FunPtr</h3>
<p>When dealing with a higher order API, <code>hs-bindgen</code> will generate additional
bindings to convert back and forth between C function pointers and Haskell
functions, and package these conversions up as instances of two type classes in
<code>hs-bindgen-runtime</code>:</p>
<div class="sourceCode" id="cb22"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb22-1"><a href="https://well-typed.com/blog/rss2.xml#cb22-1" tabindex="-1"/><span class="kw">class</span> <span class="dt">ToFunPtr</span> a <span class="kw">where</span></span>
<span id="cb22-2"><a href="https://well-typed.com/blog/rss2.xml#cb22-2" tabindex="-1"/><span class="ot">  toFunPtr ::</span> a <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">FunPtr</span> a)</span>
<span id="cb22-3"><a href="https://well-typed.com/blog/rss2.xml#cb22-3" tabindex="-1"/></span>
<span id="cb22-4"><a href="https://well-typed.com/blog/rss2.xml#cb22-4" tabindex="-1"/><span class="kw">class</span> <span class="dt">FromFunPtr</span> a <span class="kw">where</span></span>
<span id="cb22-5"><a href="https://well-typed.com/blog/rss2.xml#cb22-5" tabindex="-1"/><span class="ot">  fromFunPtr ::</span> <span class="dt">FunPtr</span> a <span class="ot">-&gt;</span> a</span>
<span id="cb22-6"><a href="https://well-typed.com/blog/rss2.xml#cb22-6" tabindex="-1"/></span>
<span id="cb22-7"><a href="https://well-typed.com/blog/rss2.xml#cb22-7" tabindex="-1"/><span class="ot">withFunPtr ::</span> <span class="dt">ToFunPtr</span> a <span class="ot">=&gt;</span> a <span class="ot">-&gt;</span> (<span class="dt">FunPtr</span> a <span class="ot">-&gt;</span> <span class="dt">IO</span> b) <span class="ot">-&gt;</span> <span class="dt">IO</span> b</span></code></pre></div>
<p>For example, suppose the driver API in library B additionally contains</p>
<div class="sourceCode" id="cb23"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb23-1"><a href="https://well-typed.com/blog/rss2.xml#cb23-1" tabindex="-1"/><span class="kw">struct</span> Driver<span class="op">;</span></span>
<span id="cb23-2"><a href="https://well-typed.com/blog/rss2.xml#cb23-2" tabindex="-1"/><span class="kw">typedef</span> <span class="dt">int</span> RunDriver<span class="op">(</span><span class="kw">struct</span> Driver<span class="op">*</span> self<span class="op">);</span></span>
<span id="cb23-3"><a href="https://well-typed.com/blog/rss2.xml#cb23-3" tabindex="-1"/></span>
<span id="cb23-4"><a href="https://well-typed.com/blog/rss2.xml#cb23-4" tabindex="-1"/><span class="kw">struct</span> Driver <span class="op">{</span></span>
<span id="cb23-5"><a href="https://well-typed.com/blog/rss2.xml#cb23-5" tabindex="-1"/>  <span class="co">// .. other fields as before ..</span></span>
<span id="cb23-6"><a href="https://well-typed.com/blog/rss2.xml#cb23-6" tabindex="-1"/>  RunDriver<span class="op">*</span> run<span class="op">;</span></span>
<span id="cb23-7"><a href="https://well-typed.com/blog/rss2.xml#cb23-7" tabindex="-1"/><span class="op">};</span></span>
<span id="cb23-8"><a href="https://well-typed.com/blog/rss2.xml#cb23-8" tabindex="-1"/></span>
<span id="cb23-9"><a href="https://well-typed.com/blog/rss2.xml#cb23-9" tabindex="-1"/><span class="dt">int</span> callDriver<span class="op">(</span><span class="kw">struct</span> Driver<span class="op">*</span> d<span class="op">);</span></span></code></pre></div>
<p>then we generate</p>
<div class="sourceCode" id="cb24"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb24-1"><a href="https://well-typed.com/blog/rss2.xml#cb24-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">RunDriver</span> <span class="ot">=</span> <span class="dt">RunDriver</span> {</span>
<span id="cb24-2"><a href="https://well-typed.com/blog/rss2.xml#cb24-2" tabindex="-1"/><span class="ot">    unwrap ::</span> <span class="dt">Ptr</span> <span class="dt">Driver</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> <span class="dt">CInt</span></span>
<span id="cb24-3"><a href="https://well-typed.com/blog/rss2.xml#cb24-3" tabindex="-1"/>  }</span>
<span id="cb24-4"><a href="https://well-typed.com/blog/rss2.xml#cb24-4" tabindex="-1"/></span>
<span id="cb24-5"><a href="https://well-typed.com/blog/rss2.xml#cb24-5" tabindex="-1"/><span class="kw">instance</span> <span class="dt">ToFunPtr</span>   <span class="dt">RunDriver</span> <span class="kw">where</span> (<span class="op">..</span>)</span>
<span id="cb24-6"><a href="https://well-typed.com/blog/rss2.xml#cb24-6" tabindex="-1"/><span class="kw">instance</span> <span class="dt">FromFunPtr</span> <span class="dt">RunDriver</span> <span class="kw">where</span> (<span class="op">..</span>)</span></code></pre></div>
<p>Here’s how we might use this:</p>
<div class="sourceCode" id="cb25"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb25-1"><a href="https://well-typed.com/blog/rss2.xml#cb25-1" tabindex="-1"/>counter <span class="ot">&lt;-</span> newIORef <span class="dv">0</span></span>
<span id="cb25-2"><a href="https://well-typed.com/blog/rss2.xml#cb25-2" tabindex="-1"/><span class="kw">let</span> run <span class="ot">=</span> <span class="dt">RunDriver</span> <span class="op">$</span> \_self <span class="ot">-&gt;</span> atomicModifyIORef counter <span class="op">$</span> \x <span class="ot">-&gt;</span> (<span class="fu">succ</span> x, x)</span>
<span id="cb25-3"><a href="https://well-typed.com/blog/rss2.xml#cb25-3" tabindex="-1"/>withFunPtr run <span class="op">$</span> \funPtr <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb25-4"><a href="https://well-typed.com/blog/rss2.xml#cb25-4" tabindex="-1"/>  poke driverPtr<span class="op">.</span>run funPtr</span>
<span id="cb25-5"><a href="https://well-typed.com/blog/rss2.xml#cb25-5" tabindex="-1"/>  replicateM_ <span class="dv">5</span> <span class="op">$</span> <span class="fu">print</span> <span class="op">=&lt;&lt;</span> callDriver driverPtr</span></code></pre></div>
<h2 id="macros">Macros</h2>
<p>Some C libraries make part of their API available as CPP macros. Macros in C
don’t have a semantics per se; they are merely a list of tokens that the
preprocessor splices in whenever they are used. In order to nonetheless be able
to generate “bindings” for macros, <code>hs-bindgen</code> imbues macros with a bespoke
semantics. In future versions of <code>hs-bindgen</code> this macro infrastructure
will be pluggable (<a href="https://github.com/well-typed/hs-bindgen/issues/942">#942</a>),
because the default semantics may not suit all applications.</p>
<h3 id="constants">Constants</h3>
<p>Low level C libraries (such as this Analog Devices <a href="https://github.com/analogdevicesinc/linux/blob/b7484c97688ddd60bfd05a1d6d28f454f3f0dbb0/drivers/iio/adc/talise/talise_arm_macros.h#L48-L55">Talise</a> driver)
may make certain constants such as bitfields available as CPP macros</p>
<div class="sourceCode" id="cb26"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb26-1"><a href="https://well-typed.com/blog/rss2.xml#cb26-1" tabindex="-1"/><span class="pp">#define SIGNALID </span><span class="bn">0x01</span></span></code></pre></div>
<p>We translate these to Haskell constants:</p>
<div class="sourceCode" id="cb27"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb27-1"><a href="https://well-typed.com/blog/rss2.xml#cb27-1" tabindex="-1"/><span class="ot">sIGNALID ::</span> <span class="dt">CInt</span></span>
<span id="cb27-2"><a href="https://well-typed.com/blog/rss2.xml#cb27-2" tabindex="-1"/>sIGNALID <span class="ot">=</span> <span class="dv">1</span></span></code></pre></div>
<h3 id="expressions">Expressions</h3>
<p>It may also happen that libraries offer certain functionality as macro
<em>functions</em>. For example, a library might provide a definition that
provides a pointer offset for devices with memory-mapped I/O:</p>
<div class="sourceCode" id="cb28"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb28-1"><a href="https://well-typed.com/blog/rss2.xml#cb28-1" tabindex="-1"/><span class="pp">#define INPUT_PORT(x) x + 4</span></span></code></pre></div>
<p>Since <code>hs-bindgen</code> has no way of knowing if that <code>(+)</code> operator should be
interpreted as integer addition or pointer offset (or indeed something else),
it generates a very general definition:</p>
<div class="sourceCode" id="cb29"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb29-1"><a href="https://well-typed.com/blog/rss2.xml#cb29-1" tabindex="-1"/><span class="ot">iNPUT_PORT ::</span> <span class="dt">Add</span> a <span class="dt">CInt</span> <span class="ot">=&gt;</span> a <span class="ot">-&gt;</span> <span class="dt">AddRes</span> a <span class="dt">CInt</span></span></code></pre></div>
<p><code>Add</code> and <code>AddRes</code> come from <a href="https://hackage.haskell.org/package/c-expr-runtime-0.1.0.0/candidate"><code>c-expr-runtime</code></a>); one way that we can instantiate this type is to:</p>
<div class="sourceCode" id="cb30"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb30-1"><a href="https://well-typed.com/blog/rss2.xml#cb30-1" tabindex="-1"/><span class="ot">inputPort ::</span> <span class="dt">Ptr</span> <span class="dt">Word8</span> <span class="ot">-&gt;</span> <span class="dt">Ptr</span> <span class="dt">Word8</span></span>
<span id="cb30-2"><a href="https://well-typed.com/blog/rss2.xml#cb30-2" tabindex="-1"/>inputPort <span class="ot">=</span> iNPUT_PORT</span></code></pre></div>
<h3 id="types">Types</h3>
<p>Finally, some low-level libraries define <em>types</em> as macros:</p>
<div class="sourceCode" id="cb31"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb31-1"><a href="https://well-typed.com/blog/rss2.xml#cb31-1" tabindex="-1"/><span class="pp">#define S16_TYPE </span><span class="dt">short</span><span class="pp"> </span><span class="dt">int</span></span></code></pre></div>
<p>In the case (and only in the case) that these can be parsed as the corresponding
typedef</p>
<div class="sourceCode" id="cb32"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb32-1"><a href="https://well-typed.com/blog/rss2.xml#cb32-1" tabindex="-1"/><span class="kw">typedef</span> <span class="dt">short</span> <span class="dt">int</span> S16_TYPE<span class="op">;</span></span></code></pre></div>
<p><code>hs-bindgen</code> will treat these <em>as</em> <code>typedefs</code>, and generate a Haskell newtype:</p>
<div class="sourceCode" id="cb33"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb33-1"><a href="https://well-typed.com/blog/rss2.xml#cb33-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">S16_TYPE</span> <span class="ot">=</span> <span class="dt">S16_TYPE</span> {</span>
<span id="cb33-2"><a href="https://well-typed.com/blog/rss2.xml#cb33-2" tabindex="-1"/><span class="ot">    unwrap ::</span> <span class="dt">CShort</span></span>
<span id="cb33-3"><a href="https://well-typed.com/blog/rss2.xml#cb33-3" tabindex="-1"/>  }</span></code></pre></div>
<h2 id="squashing">Squashing</h2>
<p>C <code>struct</code>s are often defined using a <code>typedef</code>:</p>
<div class="sourceCode" id="cb34"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb34-1"><a href="https://well-typed.com/blog/rss2.xml#cb34-1" tabindex="-1"/><span class="kw">typedef</span> <span class="kw">struct</span> Point <span class="op">{</span></span>
<span id="cb34-2"><a href="https://well-typed.com/blog/rss2.xml#cb34-2" tabindex="-1"/>  <span class="dt">int</span> x<span class="op">;</span></span>
<span id="cb34-3"><a href="https://well-typed.com/blog/rss2.xml#cb34-3" tabindex="-1"/>  <span class="dt">int</span> y<span class="op">;</span></span>
<span id="cb34-4"><a href="https://well-typed.com/blog/rss2.xml#cb34-4" tabindex="-1"/><span class="op">}</span> Point<span class="op">;</span></span></code></pre></div>
<p>This is typically done for syntactic convenience only, making it possible to
write simply <code>Point</code> rather than <code>struct Point</code>. When the <em>only</em> use of a struct
is within a <code>typedef</code> in this manner, <code>hs-bindgen</code> will “squash” the <code>typedef</code>,
and generate a single type only:</p>
<div class="sourceCode" id="cb35"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb35-1"><a href="https://well-typed.com/blog/rss2.xml#cb35-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Point</span> <span class="ot">=</span> <span class="dt">Point</span> {</span>
<span id="cb35-2"><a href="https://well-typed.com/blog/rss2.xml#cb35-2" tabindex="-1"/><span class="ot">    x ::</span> <span class="dt">CInt</span></span>
<span id="cb35-3"><a href="https://well-typed.com/blog/rss2.xml#cb35-3" tabindex="-1"/>  ,<span class="ot"> y ::</span> <span class="dt">CInt</span></span>
<span id="cb35-4"><a href="https://well-typed.com/blog/rss2.xml#cb35-4" tabindex="-1"/>  }</span></code></pre></div>
<p>If however the <code>typedef</code> is <em>not</em> the only use of the <code>struct</code>, then
<code>hs-bindgen</code> assumes that this is intended to convey some kind of semantic
information, and will not squash. For example, suppose the device driver API
includes</p>
<div class="sourceCode" id="cb36"><pre class="sourceCode c"><code class="sourceCode c"><span id="cb36-1"><a href="https://well-typed.com/blog/rss2.xml#cb36-1" tabindex="-1"/><span class="kw">typedef</span> <span class="kw">struct</span> Driver DeviceDriver<span class="op">;</span></span></code></pre></div>
<p>then we generate</p>
<div class="sourceCode" id="cb37"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb37-1"><a href="https://well-typed.com/blog/rss2.xml#cb37-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">DeviceDriver</span> <span class="ot">=</span> <span class="dt">DeviceDriver</span> {</span>
<span id="cb37-2"><a href="https://well-typed.com/blog/rss2.xml#cb37-2" tabindex="-1"/><span class="ot">    unwrap ::</span> <span class="dt">Driver</span></span>
<span id="cb37-3"><a href="https://well-typed.com/blog/rss2.xml#cb37-3" tabindex="-1"/>  }</span></code></pre></div>
<p>To avoid confusion, a notice will be emitted whenever a type is squashed. If
desired, this notice can be suppressed with
<code>--log-as-info select-mangle-names-squashed</code>.
In a future release of <code>hs-bindgen</code> squashing will be configurable per <code>typedef</code>
using a prescriptive binding spec
(<a href="https://github.com/well-typed/hs-bindgen/issues/1436">#1436</a>).</p>
<h2 id="build-process-integration">Build process integration</h2>
<h3 id="preprocessor">Preprocessor</h3>
<p>The primary way of invoking <code>hs-bindgen</code> is by calling <code>hs-bindgen-cli</code>, as we
have discussed in this blogpost, raising the question of how to integrate that
into your build process. We don’t have a perfect answer here just yet. You
can of course write your own <code>Makefile</code>, or integrate this into a <code>nix</code>
derivation (see the <a href="https://github.com/well-typed/hs-bindgen-tutorial-nix/">Nix tutorial</a>). If you want to use <code>cabal</code> as the main driver, you can use a custom
setup script, or the new
<a href="https://hackage-content.haskell.org/package/Cabal-hooks-3.16/docs/Distribution-Simple-SetupHooks.html#g:6"><code>SetupHooks</code></a>
API; we don’t provide explicit support for either just yet
(<a href="https://github.com/well-typed/hs-bindgen/issues/1666">#1666</a> tracks adding
support for <code>SetupHooks</code>).</p>
<p>We do offer a “literate” mode, abusing Cabal’s support for literate Haskell to
trick it into running <code>hs-bindgen</code> instead; see section
<a href="https://github.com/well-typed/hs-bindgen/blob/main/manual/LowLevel/Usage/01-Invocation.md#cabal-preprocessor-integration">Cabal preprocessor integration</a>
of the <code>hs-bindgen</code> manual for details.</p>
<h3 id="template-haskell">Template Haskell</h3>
<p>If you prefer, you can use <code>hs-bindgen</code> in TH mode; this avoids the need for
running the preprocessor altogether. Instead, you can
<code>#include</code> a C header in a Haskell module:</p>
<div class="sourceCode" id="cb38"><pre class="sourceCode hs"><code class="sourceCode haskell"><span id="cb38-1"><a href="https://well-typed.com/blog/rss2.xml#cb38-1" tabindex="-1"/><span class="kw">let</span><span class="ot"> cfg ::</span> <span class="dt">Config</span></span>
<span id="cb38-2"><a href="https://well-typed.com/blog/rss2.xml#cb38-2" tabindex="-1"/>    cfg <span class="ot">=</span> def <span class="op">&amp;</span> <span class="op">#</span>clang <span class="op">%</span> <span class="op">#</span>extraIncludeDirs <span class="op">.~</span> [<span class="dt">Pkg</span> <span class="st">"cbits"</span>]</span>
<span id="cb38-3"><a href="https://well-typed.com/blog/rss2.xml#cb38-3" tabindex="-1"/> <span class="kw">in</span> withHsBindgen cfg def <span class="op">$</span></span>
<span id="cb38-4"><a href="https://well-typed.com/blog/rss2.xml#cb38-4" tabindex="-1"/>       hashInclude <span class="st">"library_a.h"</span></span></code></pre></div>
<p>Since we cannot generate more than one module in this way, by default this
splices in bindings for all the types in the header along with the <em>safe</em>
imports, though you can override that if you wish. Of course, you can also
use <code>hashInclude</code> in more than one Haskell module to manually generate multiple
modules. See <a href="https://github.com/well-typed/hs-bindgen/blob/main/manual/LowLevel/Usage/01-Invocation.md#template-haskell-mode">Template Haskell
mode</a>
in the manual for details.</p>
<h3 id="cross-compilation">Cross-compilation</h3>
<p>We rely on <a href="https://clang.llvm.org/doxygen/group__CINDEX.html"><code>libclang</code></a> for
all machine-dependent decisions (as well as parsing the C headers in the first
place). Therefore if you want to generate code for a target differing from your host
platform, in principle it should suffice to provide the appropriate <code>clang</code>
arguments. We are working on a guide for doing so; this will become section
<a href="https://github.com/well-typed/hs-bindgen/blob/main/manual/LowLevel/Usage/08-CrossCompilation.md">Cross-compilation</a>
in the manual; however, this section is not quite ready yet
(<a href="https://github.com/well-typed/hs-bindgen/pull/1630">#1630</a>).</p>
<h3 id="non-portability">Non-portability</h3>
<p>It is important to note that the bindings generated by <code>hs-bindgen</code> are <em>not</em>
portable in general (indeed, it will be rare that they are). The slogan to
remember is:</p>
<blockquote>
The bindings generated by <code>hs-bindgen</code> should be regarded as build artefacts.
</blockquote>
<p>If you <em>do</em> want to distribute generated bindings as part of your package, you
can of course do so, but then you are responsible for making the appropriate
provisions in your <code>.cabal</code> file (perhaps using <code>SetupHooks</code>) to check machine
architecture, choose between different sets of bindings, etc. At present
<code>hs-bindgen</code> does not yet provide any explicit support for making this process
easier.</p>
<h2 id="conclusions">Conclusions</h2>
<p>Although Haskell provides good FFI features, writing bindings to large C
libraries by hand can be a laborious process. The bindings generated by
<code>hs-bindgen</code> are low-level: <code>char*</code> is translated to <code>Ptr CChar</code>, not
<code>ByteString</code> or <code>Text</code> or something else still; nonetheless, it should make the
process of writing bindings <em>significantly</em> easier. Automatically generating
<em>high-level</em> bindings is something we’ll soon turn our attention to.</p>
<p>Before we do so, however, there is still some
<a href="https://github.com/well-typed/hs-bindgen/issues?q=is%3Aissue%20state%3Aopen%20label%3Arelease-0.1">cleanup to be done</a>
before we can release version 0.1. As it stands, the alpha release supports
nearly all of C, although some missing corner-cases are still missing (such
as implicit fields,
<a href="https://github.com/well-typed/hs-bindgen/issues/1649">#1649</a>); see the issue
tracker for a
<a href="https://github.com/well-typed/hs-bindgen/issues?q=is%3Aissue%20state%3Aopen%20label%3Amissing-c-feature">list of missing C features</a>.
However, we would like to invite you to start using the alpha release now; there
should only be very minor backwards incompatible changes introduced in between
versions <code>0.1-alpha</code> and <code>0.1</code>, so your code should not break when upgrading to version <code>0.1</code>.</p>
<h3 id="acknowledgements">Acknowledgements</h3>
<p>Many people at Well-Typed have contributed to <code>hs-bindgen</code>, including Travis
Cardwell, Joris Dral, Dominik Schrempf, Armando Santos, Sam Derbyshire, and
others (as well as myself, Edsko de Vries).</p>
<p>Well-Typed is grateful to Anduril Industries for sponsoring this work.</p>
<h3 id="further-reading">Further reading</h3>
<ul>
<li>Paper on <code>hs-bindgen</code>: <a href="https://well-typed.com/blog/aux/files/haskell2025-hs-bindgen.pdf">Automatic C Bindings Generation for Haskell</a>, published at <a href="https://dl.acm.org/doi/10.1145/3759164.3759350">Haskell 2025</a>. You might also like to <a href="https://www.youtube.com/watch?v=SeLI3lMnhnA">watch the presentation</a>.</li>
<li>The <a href="https://github.com/well-typed/hs-bindgen/tree/main/manual"><code>hs-bindgen</code> manual</a>. Note that while many sections of the manual have already been written, some are still empty or out of date; this is one of the things that will be completed prior to the official 0.1 release.</li>
<li>The <a href="https://github.com/well-typed/hs-bindgen-tutorial-nix"><code>hs-bindgen nix tutorial</code></a>. This tutorial shows how to generate partial bindings for two libraries (<code>pcap</code> and <code>wlroots</code>). Although this assumes <code>nix</code>, most of the information here will be relevant for non-<code>nix</code> users also.</li>
<li>The <a href="https://github.com/well-typed/hs-bindgen/tree/main/examples"><code>examples</code></a> folder of the <code>hs-bindgen</code> repo, which contains example partial bindings for a bunch of libraries, such as <a href="https://botan.randombit.net/"><code>Botan</code></a>, <a href="https://github.com/niklasso/minisat-c-bindings"><code>minisat</code></a>, <a href="https://github.com/nayuki/QR-Code-generator"><code>QR-Code-generator</code></a>, <a href="https://github.com/sakhmatd/rogueutil"><code>rogueutil</code></a>, <a href="https://github.com/rpm-software-management/rpm"><code>rpm</code></a> and <a href="https://github.com/the-tcpdump-group/libpcap"><code>libpcap</code></a>.</li>
</ul>
<section class="footnotes footnotes-end-of-document" id="footnotes">
<hr/>
<ol>
<li id="fn1"><p>The C namespace is a global namespace, so when <code>hs-bindgen</code>
introduces new C symbols, they must also be globally unique. Normally the
uniqueness of the symbols introduced by <code>hs-bindgen</code> is derived from the
uniqueness of the C symbols for which we are generating bindings, but it may happen
that multiple Haskell libraries include bindings for the <em>same</em> C function. In
such a case it is important that the <code>--unique-id</code> used for these different
Haskell modules is different.<a class="footnote-back" href="https://well-typed.com/blog/rss2.xml#fnref1">↩︎</a></p></li>
</ol>
</section></div>
    </content>
    <updated>2026-02-10T00:00:00Z</updated>
    <published>2026-02-10T00:00:00Z</published>
    <category term="hs-bindgen"/>
    <category term="open-source"/>
    <category term="foreign function interface"/>
    <author>
      <name>edsko</name>
    </author>
    <source>
      <id>https://well-typed.com/blog/</id>
      <logo>https://well-typed.com/img/wtnlogoplain.svg</logo>
      <link href="https://well-typed.com/blog/rss2.xml" rel="self" type="application/rss+xml"/>
      <link href="https://well-typed.com/blog/" rel="alternate" type="text/html"/>
      <subtitle>Because Well-Typed Programs Don't Go Wrong</subtitle>
      <title>Well-Typed Blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>https://haskellforall.com/2026/02/beyond-agentic-coding</id>
    <link href="https://haskellforall.com/2026/02/beyond-agentic-coding" rel="alternate" type="text/html"/>
    <title>Beyond agentic coding&gt;</title>
    <summary>AI dev tooling can do better than chat interfaces</summary>
    <updated>2026-02-07T00:00:00Z</updated>
    <published>2026-02-07T00:00:00Z</published>
    <source>
      <id>https://haskellforall.com</id>
      <author>
        <name>Gabriella Gonzalez</name>
      </author>
      <link href="https://haskellforall.com" rel="alternate" type="text/html"/>
      <link href="https://haskellforall.com/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>A blog about Haskell and functional programming.</subtitle>
      <title>Haskell for all</title>
      <updated>2026-03-17T14:41:11Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://tweag.io/blog/2026-02-05-bazel-coverity-integration/</id>
    <link href="https://tweag.io/blog/2026-02-05-bazel-coverity-integration/" rel="alternate" type="text/html"/>
    <title>Integrating Coverity static analysis with Bazel</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><a href="https://www.blackduck.com/static-analysis-tools-sast/coverity.html">Coverity</a> is a proprietary static code analysis tool that can be
used to find code quality defects in large-scale, complex software. It
supports <a href="https://www.blackduck.com/static-analysis-tools-sast/languages-and-frameworks.html">a number of languages and
frameworks</a> and has been trusted to
ensure compliance with various standards such as MISRA, AUTOSAR, ISO 26262,
and others. Coverity provides integrations with several build systems,
including Bazel, however the official Bazel integration fell short of the
expectations of our client, who wanted to leverage the Bazel remote cache in
order to speed up Coverity analysis and be able to run it in normal merge
request workflows. We took on that challenge.</p>
<h2 id="the-coverity-workflow"><a class="anchor before" href="https://www.tweag.io/rss.xml#the-coverity-workflow"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>The Coverity workflow</h2>
<p>To understand the rest of the post it is useful to first become familiar
with the Coverity workflow, which is largely linear and can be summarized as
a sequence of steps:</p>
<ol>
<li>Configuration. In this step <code class="language-text">cov-configure</code> is invoked, possibly several
times with various combinations of key compiler flags (such as
<code class="language-text">-nostdinc++</code>, <code class="language-text">-fPIC</code>, <code class="language-text">-std=c++14</code>, and others). This produces a
top-level XML configuration file accompanied by a directory tree that
contains configurations that correspond to the key flags that were
provided. Coverity is then able to pick the right configuration by
dispatching on those key flags when it sees them during the build.</li>
<li>Compilation. This is typically done by invoking <code class="language-text">cov-build</code> or its
lower-level cousin <code class="language-text">cov-translate</code>. The distinction between these two
utilities will become clear in the next section. For now, it is enough to
point out that these tools translate original invocations of the compiler
to invocations of <code class="language-text">cov-emit</code> (Coverity’s own compiler) which in turn
populate an SQLite database with intermediate representations and all the
metadata about the built sources such as symbols and include paths.</li>
<li>Analysis. This step amounts to one or more invocations of <code class="language-text">cov-analyze</code>.
This utility is practically a black box, and it can take a considerable
time to run, depending on the number of translation units (think
compilation units in C/C++). The input for the analysis is the SQLite
database that was produced in the previous step. <code class="language-text">cov-analyze</code> populates
a directory tree that can then be used for report generation or uploading
of identified software defects.</li>
<li>Reporting. Listing and registration of found defects is performed with
<code class="language-text">cov-format-errors</code> and <code class="language-text">cov-commit-defects</code>, respectively. These tools
require a directory structure that was created by <code class="language-text">cov-analyze</code> in the
previous step.</li>
</ol>
<h2 id="existing-integrations"><a class="anchor before" href="https://www.tweag.io/rss.xml#existing-integrations"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Existing integrations</h2>
<p>A successful integration of Coverity into Bazel has to provide a way to
perform step 2 of the workflow, since this is the only step that is
intertwined with the build system. Step 1 is relatively quick, and it does
not make a lot of difference how it is done. Step 3 is opaque and
build-system-agnostic because it happens after the build, furthermore there
is no obvious way to improve its granularity and cacheability. Step 4 is
similar in that regard.</p>
<p><a href="https://documentation.blackduck.com/bundle/coverity-docs/page/coverity-analysis/topics/building_with_bazel.html">The official integration</a> works by wrapping a
Bazel invocation with <code class="language-text">cov-build</code>:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">$ cov-build --dir /path/to/idir --bazel bazel build //:my-bazel-target</code></pre></div>
<p>Here, <code class="language-text">--dir</code> specifies the intermediate directory where the SQLite database
along with some metadata will be stored. In the <code class="language-text">--bazel</code> mode of operation
<code class="language-text">cov-build</code> tricks Bazel by intercepting the invocations of the compiler
which are replaced by invocations of <code class="language-text">cov-translate</code>. Compared to
<code class="language-text">cov-build</code>, which is a high-level wrapper, <code class="language-text">cov-translate</code> corresponds more
closely to individual invocations of a compiler. It identifies the right
configuration from the collection of configurations created in step 1
and then uses it to convert the command line arguments of the original
compiler into the command line arguments of <code class="language-text">cov-emit</code>, which it then
invokes.</p>
<p>The main problem with the official integration is the fact that it does not
support caching. <code class="language-text">bazel build</code> has to run from scratch with caching disabled
in order to make sure that all invocations of the compiler are performed and
none are skipped. Another nuance of the official integration is that one has
to build a target of the supported kind, e.g. <code class="language-text">cc_library</code>. If you bundle
your build products together in some way (e.g. in an archive), you cannot
simply build the top-level bundle as you normally would. Instead, you need
to identify every compatible target of interest in some other way.</p>
<p>Because of this, our client did not use the official Coverity integration
for Bazel. Instead, they would run Bazel with the <code class="language-text">--subcommands</code> option
which makes Bazel print how it invokes all tools that participate in the
build. This long log would then be parsed and converted into a <code class="language-text">Makefile</code> in
order to be able to leverage Coverity’s Make integration in <code class="language-text">cov-build</code>
instead. This approach still suffered from long execution times due to lack
of caching. It ran as a nightly build and took over 12 hours, which wasn’t
suitable for a regular merge request’s CI pipeline.</p>
<h2 id="our-approach"><a class="anchor before" href="https://www.tweag.io/rss.xml#our-approach"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Our approach</h2>
<p>The key insight that allowed us to make the build process cacheable is the
observation that individual invocations of <code class="language-text">cov-translate</code> produce SQLite
databases—<code class="language-text">emit-db</code>s—that are granular and relatively small. These
individual <code class="language-text">emit-db</code>s can then be merged in order to form the final, big
<code class="language-text">emit-db</code> that can be used for running <code class="language-text">cov-analyze</code>. Therefore, our plan
was the following:</p>
<ol>
<li>Create a special-purpose Bazel toolchain that starts as a copy of the
toolchain which is used for “normal” compilation in order to match the
way the original compiler (<code class="language-text">gcc</code> in our case) is invoked.</li>
<li>Instead of <code class="language-text">gcc</code> and some other tools such as <code class="language-text">ar</code>, have it invoke our
own wrapper that would drive invocations of <code class="language-text">cov-translate</code>.</li>
<li>The only useful output of the compilation step is the <code class="language-text">emit-db</code> database.
On the other hand, in <code class="language-text">CppCompile</code> actions, Bazel normally expects <code class="language-text">.o</code>
files to be produced, so we just rename our <code class="language-text">emit-db</code> SQLite database to
whatever object file Bazel is expecting.</li>
<li>In the linking step, we use the <code class="language-text">add</code> subcommand of <code class="language-text">cov-manage-emit</code> in
order to merge individual <code class="language-text">emit-db</code>s.</li>
<li>Once the final bundle is built, we iterate through all eligible database
files and merge them together once again in order to obtain the final
<code class="language-text">emit-db</code> that will be used for analysis.</li>
</ol>
<p>From Bazel’s point of view, we are simply compiling C/C++ code, however, this
is a way to perform a cacheable Coverity build. If you are a regular reader of
our blog, you may have noticed certain similarities to the approach we used
in our CTC++ Bazel integration documented <a href="https://www.tweag.io/blog/2023-11-09-ctc++-bazel-integration">here</a> and
<a href="https://www.tweag.io/blog/2025-03-06-ctc++-revisited">here</a>.</p>
<h2 id="difficulties"><a class="anchor before" href="https://www.tweag.io/rss.xml#difficulties"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Difficulties</h2>
<p>The plan may sound simple, but in practice there was nothing simple about
it. The key difficulty was the fact that Coverity tooling was not designed
with reproducibility in mind. At every step, starting from configuration
generation (which is simply a <code class="language-text">genrule</code> in our case) we had to inspect and
adjust produced outputs in order to make sure that they do not include any
volatile data that could compromise reproducibility. The key output in the
Coverity workflow, the <code class="language-text">emit-db</code> database, contained a number of pieces of
volatile information:</p>
<ul>
<li>Start time, end time, and process IDs of <code class="language-text">cov-emit</code> invocations.</li>
<li>Timestamps of source files.</li>
<li>Command line IDs of translation units.</li>
<li>Host name; luckily, this can be overwritten easily by setting the <code class="language-text">COV_HOST</code>
environment variable.</li>
<li>Absolute file names of source files. These are split into path segments with
each segment stored as a separate database entry, meaning the number of entries
in the <code class="language-text">FileName</code> table varies with the depth of the path to the execroot,
breaking hermeticity. Deleting entries is not viable since their IDs are
referenced elsewhere. Our solution was to identify the variable prefix leading
to the execroot and reset all segments in that prefix to a fixed string. The
prefix length proved to be constant for each execution mode, allowing us to use
<code class="language-text">--strip-path</code> arguments in <code class="language-text">cov-analyze</code> to remove them during analysis.</li>
<li>In certain cases some files from the <code class="language-text">include</code> directories would be added
to the database with altered Coverity-generated contents
that would also include absolute paths.</li>
</ul>
<p>We had to meticulously overwrite all of these, which was only possible
because <code class="language-text">emit-db</code> is in the SQLite format and there is open source tooling
that makes it possible to edit it. If the output of <code class="language-text">cov-emit</code> were in a
proprietary format we almost certainly wouldn’t be able to deliver as
efficient a solution for our client.</p>
<p>In practice, normalization of <code class="language-text">emit-db</code> happens in two stages:</p>
<ul>
<li>We run some SQL commands that reset the volatile pieces of data inside the
database. As we were iterating on these “corrective instructions”, we made
sure that we eliminated all instances of volatility by using the
<code class="language-text">sqldiff</code> utility which can print differences in schema and data between
tables.</li>
<li>We dump the resulting database with the <code class="language-text">.dump</code> command which exports the
SQL statements necessary to recreate the database with the same schema and
data. Then we re-load these statements and thus obtain a binary database file
that is bit-by-bit reproducible. This is necessary because simply editing
a database by running SQL commands on it does not ensure that the result
is binary reproducible even if there is no difference in the actual
contents.</li>
</ul>
<h2 id="performance-considerations"><a class="anchor before" href="https://www.tweag.io/rss.xml#performance-considerations"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Performance considerations</h2>
<p>Since <code class="language-text">emit-db</code>s are considerably larger than typical object files, we found
it highly desirable to use compression for individual SQLite databases, both
for those that result from initial <code class="language-text">cov-translate</code> invocations and for
merged databases that are created by <code class="language-text">cov-manage-emit</code> in the linking step.
<a href="https://github.com/facebook/zstd">Zstandard</a> proved to be a good choice for that—the utility is both
fast and makes our build outputs up to 4 times smaller. Without compression,
we risked filling the remote cache quickly; besides, the bigger the files,
the slower I/O operations are.</p>
<p>We were tempted to minimize the size of the database even further by
exploring if there’s anything in <code class="language-text">emit-db</code> that can be safely removed
without affecting our client’s use case. Alas, every piece of information
stored was required by Coverity during the analysis phase and our attempts
to truncate some of the tables led to failures in <code class="language-text">cov-analyze</code>. It is worth
noting that the <code class="language-text">sqlite3_analyzer</code> utility tool (part of the SQLite project)
can be used to produce a report that explains which objects store most data.
This way we found that there is an index that contributes about 20% of the
database size, however, deleting it severely degrades the performance of
record inserts which is a key operation during merging of <code class="language-text">emit-db</code>s.</p>
<p>Linking steps, which in our approach amount to merging of <code class="language-text">emit-db</code>
databases, produce particularly large files. In order
to reduce the amount of I/O we perform and avoid
filling the remote cache too quickly,
we’ve marked all operations that deal with these large files
as <code class="language-text">no-remote</code>. This is accomplished with this line in
our <code class="language-text">.bazelrc</code> file:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">common:coverity --modify_execution_info=CppArchive=+no-remote,CppLink=+no-remote,CcStrip=+no-remote
# If you wish to disable RE for compilation actions with coverity, uncomment
# the following line:
# common:coverity --modify_execution_info=CppCompile=+no-remote-exec</code></pre></div>
<p>On the other hand, we did end up running <code class="language-text">CppCompile</code> actions with remote execution (RE)
because it proved to be twice faster than running them on CI agents. Making
RE possible required us to identify the exact collection of files from the
Coverity installation that are required during invocations of the Coverity
tooling. Once we observed RE working correctly, we were confident that the
Bazel definitions we use are hermetic.</p>
<p>Merging of individual <code class="language-text">emit-db</code> databases found in the final bundle (i.e.
after the build has finished) proved to be time-consuming. This operation
cannot be parallelized since database information cannot be written in
parallel. The time required for this step grows linearly with the number of
translation units (TUs) being inserted, therefore it makes sense to pick the
largest database and then merge smaller ones into it. One could entertain
the possibility of skipping merging altogether and instead running smaller
analyses on individual <code class="language-text">emit-db</code>s, but this seems to be not advisable, since
Coverity performs a whole-program analysis, and thus it would lose valuable
information that way. For example, one TU may be exercised in different ways
by two different applications and the analysis results for this TU cannot be
correctly merged.</p>
<p>The analysis phase is a black box, and it can easily become the bottleneck
of the entire workflow, thus making it impractical for running in merge
request pipelines. A common solution for speeding up the analysis in merge
request pipelines is to identify files that were edited and limit the
analysis to only these files with the <code class="language-text">--tu-pattern</code> option, which supports
<a href="https://documentation.blackduck.com/bundle/coverity-docs/page/commands/topics/cov-manage-emit.html#cme_patterns">a simple language</a> for telling Coverity what to care
about during analysis. We added support for this approach to our solution by
automatically finding the files changed in the current merge request, and
passing these on to <code class="language-text">--tu-pattern</code>. This restricted analysis still requires the
<code class="language-text">emit-db</code>s for the entire project, but most of them will be cached.</p>
<h2 id="the-results"><a class="anchor before" href="https://www.tweag.io/rss.xml#the-results"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>The results</h2>
<p>The solution that we delivered is in the form of a <code class="language-text">bazel run</code>-able target
that depends on the binary bundle that needs to be analyzed. It can be
invoked like this:</p>
<div class="gatsby-highlight"><pre class="language-text"><code class="language-text">bazel run //bazel/toolchains/coverity:workflow --config=coverity -- ...</code></pre></div>
<p>This solution can be used both for the nightly runs of Coverity and in the
merge request pipelines. We have confirmed that the results that are
produced by our solution match the findings of Coverity when it is run in
the “traditional” way. A typical run in a merge request pipeline takes about
22 minutes when a couple of C++ files are edited. The time is distributed as
follows:</p>
<ul>
<li>8 minutes: building, similar to build times for normal builds (this
step is sped up by caching)</li>
<li>10 minutes: the final merging of <code class="language-text">emit-db</code>s</li>
<li>4 minutes: analysis, uploading defects, reporting (this step is sped up by <code class="language-text">--tu-pattern</code>).</li>
</ul>
<p>The execution time can grow, of course, if the edits are more substantial.
The key benefit of our approach is that the Coverity build is now cacheable
and therefore can be included in merge request CI pipelines.</p>
<h2 id="conclusion"><a class="anchor before" href="https://www.tweag.io/rss.xml#conclusion"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Conclusion</h2>
<p>In summary, integrating Coverity static analysis with Bazel in a cacheable,
reproducible, and efficient manner required a deep understanding of both
Bazel and Coverity, as well as a willingness to address the nuances of
proprietary tooling that got in the way. By leveraging granular <code class="language-text">emit-db</code>
databases, normalizing volatile data, and optimizing for remote execution
and compression, we were able to deliver a solution that fits well into the
client’s CI workflow and supports incremental analysis in merge request
pipelines.</p>
<p>While the process involved overcoming significant challenges, particularly
around reproducibility and performance, the resulting workflow enables fast
static analysis without sacrificing the benefits of Bazel’s remote cache. We
hope that sharing our approach will help other teams facing similar
challenges and inspire improvements when integrating other static analysis
tools with Bazel.</p></div>
    </summary>
    <updated>2026-02-05T00:00:00Z</updated>
    <published>2026-02-05T00:00:00Z</published>
    <source>
      <id>https://tweag.io</id>
      <author>
        <name>Tweag I/O</name>
      </author>
      <link href="https://tweag.io" rel="alternate" type="text/html"/>
      <link href="http://www.tweag.io/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Scale your engineering power. We enable deep-tech startups to achieve
their vision, from research to product delivery.</subtitle>
      <title>Tweag - Engineering blog</title>
      <updated>2026-04-02T12:24:07Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>https://blog.jle.im/entry/five-point-haskell-part-1-total-depravity.html</id>
    <link href="https://blog.jle.im/entry/five-point-haskell-part-1-total-depravity.html" rel="alternate" type="text/html"/>
    <title>"Five-Point Haskell": Total Depravity (and Defensive Typing)</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>I have thought about distilling the principles by which I program Haskell,
and how I’ve been able to steer long-lived projects over years of growth,
refactorings, and changes in demands. I find myself coming back to a few
distinct and helpful “points” (“doctrines”, if you may allow me to say) that
have yet to lead me astray.</p>
<p>With a new age of software development coming, what does it even mean to
write good, robust, correct code? It is long overdue to clarify the mindset we
use to define “good” coding principles.</p>
<p>In this series, <em><a href="https://blog.jle.im/entries/series/+five-point-haskell.html">Five-Point
Haskell</a></em>, I’ll set out to establish a five-point framework for typed
functional programming (and Haskell-derived) design that aims to produce code
that is maintainable, correct, long-lasting, extensible, and beautiful to write
and work with. We’ll reference real-world case studies with actual examples when
we can, and also attempt to dispel thought-leader sound bites that have become
all too popular on Twitter (“heresies”, so to speak).</p>
<p>Let’s jump right into point 1: the doctrine of <strong>Total
Depravity</strong>, and why Haskell is perfectly primed to make living with it
as frictionless as possible.</p>
<blockquote>
<p>Total Depravity: If your code’s correctness depends on keeping complicated
interconnected structure in your head, a devastating incident is not a matter of
<em>if</em> but <em>when</em>.</p>
<p>Therefore, delegate these concerns to tooling and a sufficiently powerful
compiler, use types to guard against errors, and free yourself to only mentally
track the actual important things.</p>
</blockquote>
<h2 id="mix-ups-are-inevitable">Mix-ups Are Inevitable</h2>
<p>Think about the stereotype of a “brilliant programmer” that an inexperienced
programmer has in their mind — someone who can hold every detail of a complex
system in their head, every intricate connection and relationship between each
component. There’s the classic <a href="https://www.monkeyuser.com/2018/focus/">Monkey User Comic</a> that
valorizes this ideal.</p>
<figure>
<img alt="Monkey User &#x2014; Focus" src="http://feeds.feedburner.com/img/entries/five-point-haskell/79-focus.png" style="width: 50%; height: auto;" title="Monkey User --- Focus"/>
<figcaption>Monkey User — Focus</figcaption>
</figure>
<p>The 10x developer is one who can carry the state and interconnectedness of an
entire system in their brain, and the bigger the state they can carry, the more
10x they are.</p>
<p>This is the myth of the hero programmer. Did you have a bug? Well, you just
need to upgrade your mental awareness and your context window. You just need to
be better and better at keeping more in your mind.</p>
<p>Actually <em>addressing</em> these issues in most languages requires a lot of
overhead and clunkiness. But luckily we’re in Haskell.</p>
<h3 id="explicit-tags">Explicit Tags</h3>
<p>The <a href="https://www.atlassian.com/blog/atlassian-engineering/post-incident-review-april-2022-outage">2022
Atlassian Outage</a>, in part, was the result of passing the wrong type of ID.
The operators were intended to pass <em>App</em> IDs, but instead passed
<em>Site</em> IDs, and the errors cascaded from there. It goes without saying
that if you have a bunch of “naked” IDs, then mixing them up is eventually going
to backfire on you.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="http://feeds.feedburner.com/incodeblog#cb1-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">Id</span> <span class="ot">=</span> <span class="dt">Id</span> <span class="dt">String</span></span>
<span id="cb1-2"><a href="http://feeds.feedburner.com/incodeblog#cb1-2" tabindex="-1"/></span>
<span id="cb1-3"><a href="http://feeds.feedburner.com/incodeblog#cb1-3" tabindex="-1"/><span class="kw">type</span> <span class="dt">SiteId</span> <span class="ot">=</span> <span class="dt">Id</span></span>
<span id="cb1-4"><a href="http://feeds.feedburner.com/incodeblog#cb1-4" tabindex="-1"/><span class="kw">type</span> <span class="dt">AppId</span> <span class="ot">=</span> <span class="dt">Id</span></span>
<span id="cb1-5"><a href="http://feeds.feedburner.com/incodeblog#cb1-5" tabindex="-1"/></span>
<span id="cb1-6"><a href="http://feeds.feedburner.com/incodeblog#cb1-6" tabindex="-1"/><span class="ot">getApps ::</span> <span class="dt">SiteId</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> [<span class="dt">AppId</span>]</span>
<span id="cb1-7"><a href="http://feeds.feedburner.com/incodeblog#cb1-7" tabindex="-1"/><span class="ot">deleteSite ::</span> <span class="dt">SiteId</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb1-8"><a href="http://feeds.feedburner.com/incodeblog#cb1-8" tabindex="-1"/><span class="ot">deleteApp ::</span> <span class="dt">AppId</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span></code></pre></div>
<p>This is convenient because you get functions for all IDs without any extra
work. Let’s say you want to serialize/print or deserialize/read these IDs — it
can be helpful to give them all the same type so that you can write this logic
in one place.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb2-1"><a href="http://feeds.feedburner.com/incodeblog#cb2-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">ToJSON</span> <span class="dt">Id</span> <span class="kw">where</span></span>
<span id="cb2-2"><a href="http://feeds.feedburner.com/incodeblog#cb2-2" tabindex="-1"/>  toJSON (<span class="dt">Id</span> x) <span class="ot">=</span> object [ <span class="st">"id"</span> <span class="op">.=</span> x ]</span>
<span id="cb2-3"><a href="http://feeds.feedburner.com/incodeblog#cb2-3" tabindex="-1"/></span>
<span id="cb2-4"><a href="http://feeds.feedburner.com/incodeblog#cb2-4" tabindex="-1"/><span class="kw">instance</span> <span class="dt">FromJSON</span> <span class="dt">Id</span> <span class="kw">where</span></span>
<span id="cb2-5"><a href="http://feeds.feedburner.com/incodeblog#cb2-5" tabindex="-1"/>  parseJSON <span class="ot">=</span> withObject <span class="st">"Id"</span> <span class="op">$</span> \v <span class="ot">-&gt;</span></span>
<span id="cb2-6"><a href="http://feeds.feedburner.com/incodeblog#cb2-6" tabindex="-1"/>    <span class="dt">Id</span> <span class="op">&lt;$&gt;</span> (v <span class="op">.:</span> <span class="st">"id"</span>)</span></code></pre></div>
<p>Convenient and effective, as long as you never accidentally use a
<code>SiteId</code> as an <code>AppId</code> or vice versa. And this is a very
easy delusion to fall into, if you don’t believe in total depravity. However,
sooner or later (maybe in a week, maybe in a year, maybe after you onboard that
new team member)…someone is going to accidentally pass a site ID where an app ID
is expected.</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb3-1"><a href="http://feeds.feedburner.com/incodeblog#cb3-1" tabindex="-1"/><span class="ot">main ::</span> <span class="dt">IO</span> ()</span>
<span id="cb3-2"><a href="http://feeds.feedburner.com/incodeblog#cb3-2" tabindex="-1"/>main <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb3-3"><a href="http://feeds.feedburner.com/incodeblog#cb3-3" tabindex="-1"/>    <span class="kw">let</span> targetSites <span class="ot">=</span> [<span class="dt">Id</span> <span class="st">"abc"</span>, <span class="dt">Id</span> <span class="st">"def"</span>]</span>
<span id="cb3-4"><a href="http://feeds.feedburner.com/incodeblog#cb3-4" tabindex="-1"/>    <span class="fu">mapM_</span> deleteApp targetSites</span></code></pre></div>
<p>And at that point it’s all over.</p>
<p>Knowing this can happen, we can add a simple newtype wrapper so that
accidentally using the wrong ID is a compile error:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb4-1"><a href="http://feeds.feedburner.com/incodeblog#cb4-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">SiteId</span> <span class="ot">=</span> <span class="dt">SiteId</span> <span class="dt">String</span></span>
<span id="cb4-2"><a href="http://feeds.feedburner.com/incodeblog#cb4-2" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">AppId</span> <span class="ot">=</span> <span class="dt">AppId</span> <span class="dt">String</span></span></code></pre></div>
<p>And now such a mis-call will never compile! Congratulations!</p>
<p>We do have a downside now: we can no longer write code polymorphic over IDs
when we want to. In the untagged situation, we could <em>only</em> write
polymorphic code, and in the new situation we can <em>only</em> write code for
one ID type.</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb5-1"><a href="http://feeds.feedburner.com/incodeblog#cb5-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">FromJSON</span> <span class="dt">SiteId</span> <span class="kw">where</span></span>
<span id="cb5-2"><a href="http://feeds.feedburner.com/incodeblog#cb5-2" tabindex="-1"/>  parseJSON <span class="ot">=</span> withObject <span class="st">"Id"</span> <span class="op">$</span> \v <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb5-3"><a href="http://feeds.feedburner.com/incodeblog#cb5-3" tabindex="-1"/>    tag <span class="ot">&lt;-</span> v <span class="op">.:</span> <span class="st">"type"</span></span>
<span id="cb5-4"><a href="http://feeds.feedburner.com/incodeblog#cb5-4" tabindex="-1"/>    unless (tag <span class="op">==</span> <span class="st">"Site"</span>) <span class="op">$</span></span>
<span id="cb5-5"><a href="http://feeds.feedburner.com/incodeblog#cb5-5" tabindex="-1"/>      <span class="fu">fail</span> <span class="st">"Parsed wrong type of ID!"</span></span>
<span id="cb5-6"><a href="http://feeds.feedburner.com/incodeblog#cb5-6" tabindex="-1"/>    <span class="dt">SiteId</span> <span class="op">&lt;$&gt;</span> (v <span class="op">.:</span> <span class="st">"id"</span>)</span>
<span id="cb5-7"><a href="http://feeds.feedburner.com/incodeblog#cb5-7" tabindex="-1"/></span>
<span id="cb5-8"><a href="http://feeds.feedburner.com/incodeblog#cb5-8" tabindex="-1"/><span class="kw">instance</span> <span class="dt">ToJSON</span> <span class="dt">SiteId</span> <span class="kw">where</span></span>
<span id="cb5-9"><a href="http://feeds.feedburner.com/incodeblog#cb5-9" tabindex="-1"/>  toJSON (<span class="dt">SiteId</span> x) <span class="ot">=</span> object [ <span class="st">"type"</span> <span class="op">.=</span> <span class="st">"Site"</span>, <span class="st">"id"</span> <span class="op">.=</span> x ]</span>
<span id="cb5-10"><a href="http://feeds.feedburner.com/incodeblog#cb5-10" tabindex="-1"/></span>
<span id="cb5-11"><a href="http://feeds.feedburner.com/incodeblog#cb5-11" tabindex="-1"/><span class="kw">instance</span> <span class="dt">FromJSON</span> <span class="dt">AppId</span> <span class="kw">where</span></span>
<span id="cb5-12"><a href="http://feeds.feedburner.com/incodeblog#cb5-12" tabindex="-1"/>  parseJSON <span class="ot">=</span> withObject <span class="st">"Id"</span> <span class="op">$</span> \v <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb5-13"><a href="http://feeds.feedburner.com/incodeblog#cb5-13" tabindex="-1"/>    tag <span class="ot">&lt;-</span> v <span class="op">.:</span> <span class="st">"type"</span></span>
<span id="cb5-14"><a href="http://feeds.feedburner.com/incodeblog#cb5-14" tabindex="-1"/>    unless (tag <span class="op">==</span> <span class="st">"App"</span>) <span class="op">$</span></span>
<span id="cb5-15"><a href="http://feeds.feedburner.com/incodeblog#cb5-15" tabindex="-1"/>      <span class="fu">fail</span> <span class="st">"Parsed wrong type of ID!"</span></span>
<span id="cb5-16"><a href="http://feeds.feedburner.com/incodeblog#cb5-16" tabindex="-1"/>    <span class="dt">AppId</span> <span class="op">&lt;$&gt;</span> (v <span class="op">.:</span> <span class="st">"id"</span>)</span>
<span id="cb5-17"><a href="http://feeds.feedburner.com/incodeblog#cb5-17" tabindex="-1"/></span>
<span id="cb5-18"><a href="http://feeds.feedburner.com/incodeblog#cb5-18" tabindex="-1"/><span class="kw">instance</span> <span class="dt">ToJSON</span> <span class="dt">AppId</span> <span class="kw">where</span></span>
<span id="cb5-19"><a href="http://feeds.feedburner.com/incodeblog#cb5-19" tabindex="-1"/>  toJSON (<span class="dt">AppId</span> x) <span class="ot">=</span> object [ <span class="st">"type"</span> <span class="op">.=</span> <span class="st">"App"</span>, <span class="st">"id"</span> <span class="op">.=</span> x ]</span></code></pre></div>
<p>However, luckily, because we’re in Haskell, it’s easy to get the best of both
worlds with <em>phantom types</em> (that don’t refer to anything inside the
actual data representation):</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb6-1"><a href="http://feeds.feedburner.com/incodeblog#cb6-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Id</span> a <span class="ot">=</span> <span class="dt">Id</span> {<span class="ot"> getId ::</span> <span class="dt">String</span> }</span>
<span id="cb6-2"><a href="http://feeds.feedburner.com/incodeblog#cb6-2" tabindex="-1"/></span>
<span id="cb6-3"><a href="http://feeds.feedburner.com/incodeblog#cb6-3" tabindex="-1"/><span class="kw">data</span> <span class="dt">Site</span></span>
<span id="cb6-4"><a href="http://feeds.feedburner.com/incodeblog#cb6-4" tabindex="-1"/><span class="kw">data</span> <span class="dt">App</span></span>
<span id="cb6-5"><a href="http://feeds.feedburner.com/incodeblog#cb6-5" tabindex="-1"/></span>
<span id="cb6-6"><a href="http://feeds.feedburner.com/incodeblog#cb6-6" tabindex="-1"/><span class="kw">type</span> <span class="dt">SiteId</span> <span class="ot">=</span> <span class="dt">Id</span> <span class="dt">Site</span></span>
<span id="cb6-7"><a href="http://feeds.feedburner.com/incodeblog#cb6-7" tabindex="-1"/><span class="kw">type</span> <span class="dt">AppId</span> <span class="ot">=</span> <span class="dt">Id</span> <span class="dt">App</span></span>
<span id="cb6-8"><a href="http://feeds.feedburner.com/incodeblog#cb6-8" tabindex="-1"/></span>
<span id="cb6-9"><a href="http://feeds.feedburner.com/incodeblog#cb6-9" tabindex="-1"/><span class="co">-- using Typeable for demonstration purposes</span></span>
<span id="cb6-10"><a href="http://feeds.feedburner.com/incodeblog#cb6-10" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Typeable</span> a <span class="ot">=&gt;</span> <span class="dt">ToJSON</span> (<span class="dt">Id</span> a) <span class="kw">where</span></span>
<span id="cb6-11"><a href="http://feeds.feedburner.com/incodeblog#cb6-11" tabindex="-1"/>  toJSON (<span class="dt">Id</span> x) <span class="ot">=</span> object</span>
<span id="cb6-12"><a href="http://feeds.feedburner.com/incodeblog#cb6-12" tabindex="-1"/>    [ <span class="st">"type"</span> <span class="op">.=</span> <span class="fu">show</span> (typeRep <span class="op">@</span>a)</span>
<span id="cb6-13"><a href="http://feeds.feedburner.com/incodeblog#cb6-13" tabindex="-1"/>    , <span class="st">"id"</span> <span class="op">.=</span> x</span>
<span id="cb6-14"><a href="http://feeds.feedburner.com/incodeblog#cb6-14" tabindex="-1"/>    ]</span>
<span id="cb6-15"><a href="http://feeds.feedburner.com/incodeblog#cb6-15" tabindex="-1"/></span>
<span id="cb6-16"><a href="http://feeds.feedburner.com/incodeblog#cb6-16" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Typeable</span> a <span class="ot">=&gt;</span> <span class="dt">FromJSON</span> (<span class="dt">Id</span> a) <span class="kw">where</span></span>
<span id="cb6-17"><a href="http://feeds.feedburner.com/incodeblog#cb6-17" tabindex="-1"/>  parseJSON <span class="ot">=</span> withObject <span class="st">"Id"</span> <span class="op">$</span> \v <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb6-18"><a href="http://feeds.feedburner.com/incodeblog#cb6-18" tabindex="-1"/>    tag <span class="ot">&lt;-</span> v <span class="op">.:</span> <span class="st">"type"</span></span>
<span id="cb6-19"><a href="http://feeds.feedburner.com/incodeblog#cb6-19" tabindex="-1"/>    unless (tag <span class="op">==</span> <span class="fu">show</span> (typeRep <span class="op">@</span>a)) <span class="op">$</span></span>
<span id="cb6-20"><a href="http://feeds.feedburner.com/incodeblog#cb6-20" tabindex="-1"/>      <span class="fu">fail</span> <span class="st">"Parsed wrong type of ID!"</span></span>
<span id="cb6-21"><a href="http://feeds.feedburner.com/incodeblog#cb6-21" tabindex="-1"/>    <span class="dt">Id</span> <span class="op">&lt;$&gt;</span> (v <span class="op">.:</span> <span class="st">"id"</span>)</span></code></pre></div>
<p>Type safety doesn’t necessarily mean inflexibility!</p>
<h3 id="phantom-powers">Phantom Powers</h3>
<p>Phantom types give us a <em>lot</em> of low-hanging fruit for preventing
inadvertent misuse.</p>
<p>The <a href="https://www.theregister.com/2017/04/11/database_deletion_downed_digital_ocean_last_week/">2017
DigitalOcean outage</a>, for example, was partially about the wrong environment
credentials being used.</p>
<p>We could imagine a test harness that clears a test database using <em><a href="https://hackage.haskell.org/package/postgresql-simple">postgresql-simple</a></em>:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb7-1"><a href="http://feeds.feedburner.com/incodeblog#cb7-1" tabindex="-1"/><span class="co">-- | Warning: do NOT call this outside of test environment!</span></span>
<span id="cb7-2"><a href="http://feeds.feedburner.com/incodeblog#cb7-2" tabindex="-1"/><span class="ot">clearTestEnv ::</span> <span class="dt">Connection</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb7-3"><a href="http://feeds.feedburner.com/incodeblog#cb7-3" tabindex="-1"/>clearTestEnv conn <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb7-4"><a href="http://feeds.feedburner.com/incodeblog#cb7-4" tabindex="-1"/>  <span class="fu">putStrLn</span> <span class="st">"Are you sure you read the warning on this function? Well, too late now!"</span></span>
<span id="cb7-5"><a href="http://feeds.feedburner.com/incodeblog#cb7-5" tabindex="-1"/>  _ <span class="ot">&lt;-</span> execute_ conn <span class="st">"DROP TABLE IF EXISTS users CASCADE"</span></span>
<span id="cb7-6"><a href="http://feeds.feedburner.com/incodeblog#cb7-6" tabindex="-1"/>  <span class="fu">putStrLn</span> <span class="st">"Test data wiped."</span></span></code></pre></div>
<p>However, somewhere down the line, someone is going to call
<code>clearTestEnv</code> <em>deep</em> inside a function inside a function
inside a function, which itself is called against the prod database. I guarantee
it.</p>
<p>To ensure this never happens, we can use closed phantom types using
<code>DataKinds</code> (made nicer with <code>TypeData</code> post-GHC 9.6):</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb8-1"><a href="http://feeds.feedburner.com/incodeblog#cb8-1" tabindex="-1"/><span class="co">-- type data = declare a closed kind and two type constructors at the type level using -XTypeData</span></span>
<span id="cb8-2"><a href="http://feeds.feedburner.com/incodeblog#cb8-2" tabindex="-1"/><span class="kw">type</span> <span class="kw">data</span> <span class="dt">Env</span> <span class="ot">=</span> <span class="dt">Prod</span> <span class="op">|</span> <span class="dt">Test</span></span>
<span id="cb8-3"><a href="http://feeds.feedburner.com/incodeblog#cb8-3" tabindex="-1"/></span>
<span id="cb8-4"><a href="http://feeds.feedburner.com/incodeblog#cb8-4" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">DbConnection</span> (<span class="ot">a ::</span> <span class="dt">Env</span>) <span class="ot">=</span> <span class="dt">DbConnection</span> <span class="dt">Connection</span></span>
<span id="cb8-5"><a href="http://feeds.feedburner.com/incodeblog#cb8-5" tabindex="-1"/></span>
<span id="cb8-6"><a href="http://feeds.feedburner.com/incodeblog#cb8-6" tabindex="-1"/><span class="ot">runQuery ::</span> <span class="dt">DbConnection</span> a <span class="ot">-&gt;</span> <span class="dt">Query</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> <span class="dt">Int64</span></span>
<span id="cb8-7"><a href="http://feeds.feedburner.com/incodeblog#cb8-7" tabindex="-1"/>runQuery (<span class="dt">DbConnection</span> c) q <span class="ot">=</span> execute_ c q</span>
<span id="cb8-8"><a href="http://feeds.feedburner.com/incodeblog#cb8-8" tabindex="-1"/></span>
<span id="cb8-9"><a href="http://feeds.feedburner.com/incodeblog#cb8-9" tabindex="-1"/><span class="co">-- | Warning: Did you remember to charge your chromebook? Oh and this function</span></span>
<span id="cb8-10"><a href="http://feeds.feedburner.com/incodeblog#cb8-10" tabindex="-1"/><span class="co">-- is safe by the way.</span></span>
<span id="cb8-11"><a href="http://feeds.feedburner.com/incodeblog#cb8-11" tabindex="-1"/><span class="ot">clearTestEnv ::</span> <span class="dt">DbConnection</span> <span class="dt">Test</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb8-12"><a href="http://feeds.feedburner.com/incodeblog#cb8-12" tabindex="-1"/>clearTestEnv conn <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb8-13"><a href="http://feeds.feedburner.com/incodeblog#cb8-13" tabindex="-1"/>  _ <span class="ot">&lt;-</span> runQuery conn <span class="st">"DROP TABLE IF EXISTS users CASCADE"</span></span>
<span id="cb8-14"><a href="http://feeds.feedburner.com/incodeblog#cb8-14" tabindex="-1"/>  <span class="fu">putStrLn</span> <span class="st">"Test data wiped."</span></span>
<span id="cb8-15"><a href="http://feeds.feedburner.com/incodeblog#cb8-15" tabindex="-1"/></span>
<span id="cb8-16"><a href="http://feeds.feedburner.com/incodeblog#cb8-16" tabindex="-1"/><span class="ot">connectProd ::</span> <span class="dt">IO</span> (<span class="dt">DbConnection</span> <span class="dt">Prod</span>)</span>
<span id="cb8-17"><a href="http://feeds.feedburner.com/incodeblog#cb8-17" tabindex="-1"/>connectProd <span class="ot">=</span> <span class="dt">DbConnection</span> <span class="op">&lt;$&gt;</span> connectPostgreSQL <span class="st">"host=prod..."</span></span></code></pre></div>
<p>Now, if you create a connection using <code>connectProd</code>, you can use
<code>runQuery</code> on it (because it can run any
<code>DbConnection a</code>)…but if any sub-function of a sub-function calls
<code>clearTestEnv</code>, it will have to unify with
<code>DbConnection Test</code>, which is impossible for a prod connection.</p>
<p>This is somewhat similar to using “mocking-only” subclasses for dependency
injection, but with a closed universe. I discuss patterns like this in my <a href="https://blog.jle.im/entries/series/+introduction-to-singletons.html">Introduction
to Singletons</a> series.</p>
<h2 id="correct-representations">Correct Representations</h2>
<h3 id="semantic-phantoms">Semantic Phantoms</h3>
<p>And sometimes, phantom types can do the work for you, not only preventing
mix-ups but also encoding business logic in their manipulation.</p>
<p>Take, for instance, the <a href="https://en.wikipedia.org/wiki/Mars_Climate_Orbiter">Mars Climate Orbiter
failure</a>, where the software module provided by Lockheed Martin expected US
Customary Units, and another one developed by NASA expected SI units.</p>
<p>If I had a function like:</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb9-1"><a href="http://feeds.feedburner.com/incodeblog#cb9-1" tabindex="-1"/><span class="co">-- | In Newton-seconds</span></span>
<span id="cb9-2"><a href="http://feeds.feedburner.com/incodeblog#cb9-2" tabindex="-1"/><span class="ot">myMomentum ::</span> <span class="dt">Double</span></span>
<span id="cb9-3"><a href="http://feeds.feedburner.com/incodeblog#cb9-3" tabindex="-1"/>myMomentum <span class="ot">=</span> <span class="dv">20</span></span>
<span id="cb9-4"><a href="http://feeds.feedburner.com/incodeblog#cb9-4" tabindex="-1"/></span>
<span id="cb9-5"><a href="http://feeds.feedburner.com/incodeblog#cb9-5" tabindex="-1"/><span class="co">-- | In Pounds-second</span></span>
<span id="cb9-6"><a href="http://feeds.feedburner.com/incodeblog#cb9-6" tabindex="-1"/><span class="ot">myImpulse ::</span> <span class="dt">Double</span></span>
<span id="cb9-7"><a href="http://feeds.feedburner.com/incodeblog#cb9-7" tabindex="-1"/>myImpulse <span class="ot">=</span> <span class="dv">4</span></span>
<span id="cb9-8"><a href="http://feeds.feedburner.com/incodeblog#cb9-8" tabindex="-1"/></span>
<span id="cb9-9"><a href="http://feeds.feedburner.com/incodeblog#cb9-9" tabindex="-1"/><span class="co">-- | Make sure these are both the same units!</span></span>
<span id="cb9-10"><a href="http://feeds.feedburner.com/incodeblog#cb9-10" tabindex="-1"/><span class="ot">applyImpulse ::</span> <span class="dt">Double</span> <span class="ot">-&gt;</span> <span class="dt">Double</span> <span class="ot">-&gt;</span> <span class="dt">Double</span></span>
<span id="cb9-11"><a href="http://feeds.feedburner.com/incodeblog#cb9-11" tabindex="-1"/>applyImpulse currentMomentum impulse <span class="ot">=</span> currentMomentum <span class="op">+</span> impulse</span></code></pre></div>
<p>This is just <em>asking</em> for someone to come along and provide newtons
alongside pounds. It isn’t even clear from the types what is expected!</p>
<p>We can instead use the <em><a href="https://hackage.haskell.org/package/dimensional">dimensional</a></em>
library:</p>
<div class="sourceCode" id="cb10"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb10-1"><a href="http://feeds.feedburner.com/incodeblog#cb10-1" tabindex="-1"/><span class="kw">import</span> <span class="kw">qualified</span> <span class="dt">Numeric.Units.Dimensional.Prelude</span> <span class="kw">as</span> <span class="dt">U</span></span>
<span id="cb10-2"><a href="http://feeds.feedburner.com/incodeblog#cb10-2" tabindex="-1"/><span class="kw">import</span> <span class="dt">Numeric.Units.Dimensional</span> ((*~))</span>
<span id="cb10-3"><a href="http://feeds.feedburner.com/incodeblog#cb10-3" tabindex="-1"/><span class="kw">import</span> <span class="dt">Numeric.Units.Dimensional.SIUnits</span></span>
<span id="cb10-4"><a href="http://feeds.feedburner.com/incodeblog#cb10-4" tabindex="-1"/><span class="kw">import</span> <span class="dt">Numeric.Units.Dimensional.NonSI</span></span>
<span id="cb10-5"><a href="http://feeds.feedburner.com/incodeblog#cb10-5" tabindex="-1"/></span>
<span id="cb10-6"><a href="http://feeds.feedburner.com/incodeblog#cb10-6" tabindex="-1"/><span class="ot">myMomentum ::</span> <span class="dt">Momentum</span></span>
<span id="cb10-7"><a href="http://feeds.feedburner.com/incodeblog#cb10-7" tabindex="-1"/>myMomentum <span class="ot">=</span> <span class="dv">20</span> <span class="op">*~</span> (newton <span class="op">U.*</span> seconds)</span>
<span id="cb10-8"><a href="http://feeds.feedburner.com/incodeblog#cb10-8" tabindex="-1"/></span>
<span id="cb10-9"><a href="http://feeds.feedburner.com/incodeblog#cb10-9" tabindex="-1"/><span class="ot">myImpulse ::</span> <span class="dt">Impulse</span></span>
<span id="cb10-10"><a href="http://feeds.feedburner.com/incodeblog#cb10-10" tabindex="-1"/>myImpulse <span class="ot">=</span> <span class="dv">4</span> <span class="op">*~</span> (poundForce <span class="op">U.*</span> seconds)</span>
<span id="cb10-11"><a href="http://feeds.feedburner.com/incodeblog#cb10-11" tabindex="-1"/></span>
<span id="cb10-12"><a href="http://feeds.feedburner.com/incodeblog#cb10-12" tabindex="-1"/><span class="co">-- Verify at compile-time that we can use '+' with Momentum and Impulse</span></span>
<span id="cb10-13"><a href="http://feeds.feedburner.com/incodeblog#cb10-13" tabindex="-1"/><span class="ot">applyImpulse ::</span> <span class="dt">Momentum</span> <span class="ot">-&gt;</span> <span class="dt">Impulse</span> <span class="ot">-&gt;</span> <span class="dt">Momentum</span></span>
<span id="cb10-14"><a href="http://feeds.feedburner.com/incodeblog#cb10-14" tabindex="-1"/>applyImpulse currentMomentum impulse <span class="ot">=</span> currentMomentum <span class="op">+</span> impulse</span></code></pre></div>
<p>Now as long as momentum and impulse are provided in the correct types at API
boundaries, no mix-up will happen. No need to send 300 million dollars down the
drain! Libraries will just need to provide a unified <code>Momentum</code> or
<code>Impulse</code> type, and everything will work out.</p>
<h3 id="the-billion-dollar-mistake">The Billion-Dollar Mistake</h3>
<p>Speaking of costly errors, there is one extremely egregious pattern that is
so pervasive, so alluring, and yet so inevitably devastating, it has been dubbed
the “Billion Dollar Mistake”. It’s the idea of a <em>sentinel value</em>, or
in-band signaling.</p>
<p>There are examples:</p>
<ul>
<li><code>String.indexOf()</code>, <code>str.find()</code>, etc. in many
languages return -1 if the substring is not found</li>
<li>C’s <code>fgetc()</code>, <code>getchar()</code>, return -1 for
<code>EOF</code>. And if you cast to <code>char</code>, you basically can’t
distinguish EOF from <code>0xff</code> (<code>ÿ</code>).</li>
<li><code>malloc()</code> returning the pointer 0 means not enough memory</li>
<li>Some languages have a special <code>NULL</code> pointer value as well — or
even a value <code>null</code> that can be passed in for any expected type or
object or value.</li>
<li>JavaScript’s <code>parseInt</code> returns not <code>null</code>, but rather
<code>NaN</code> for a bad parse — giving two distinct sentinel values</li>
<li>A lot of Unix scripting uses the empty string <code>""</code> for
non-presence</li>
<li>Sensor firmware often reports values like <code>-999</code> for a bad
reading…but sometimes <code>-999</code> might actually be a valid value!</li>
</ul>
<p>It should be evident that these are just accidents and ticking time bombs
waiting to happen. Some caller just needs to forget to handle the sentinel
value, or to falsely assume that the sentinel value is impossible to occur in
any situation.</p>
<p>It’s called the billion dollar mistake, but it’s definitely arguable that the
cumulative damage has been much higher. High-profile incidents include <a href="https://www.rapid7.com/db/modules/exploit/linux/local/sock_sendpage/">sock_sendpage</a>
and the <a href="https://www.thousandeyes.com/blog/google-cloud-outage-analysis-june-12-2025">2025
GCP outage</a>, but if you’re reading this and you are honest with yourself,
it’s probably happened to you multiple times and has been the source of many
frustrating bug hunts.</p>
<p>There’s also <a href="https://www.invicti.com/web-application-vulnerabilities/openssl-improper-input-validation-vulnerability-cve-2008-5077">CVE-2008-5077</a>,
because <a href="https://docs.openssl.org/1.1.1/man3/EVP_VerifyInit/">EVP_VerifyInit</a>
returns <code>0</code> for false, <code>1</code> for true, and <code>-1</code>
for error! So some OpenSSL code did a simple if-then-else check
(<code>result != 0</code>) and treated error and true the same way. Whoops.</p>
<p>Why do we do this to ourselves? Because it is convenient. In the case of
<code>EVP_VerifyInit</code>, we can define an enum instead…</p>
<div class="sourceCode" id="cb11"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb11-1"><a href="http://feeds.feedburner.com/incodeblog#cb11-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">VerifyResult</span> <span class="ot">=</span> <span class="dt">Success</span> <span class="op">|</span> <span class="dt">Failure</span> <span class="op">|</span> <span class="dt">Error</span></span></code></pre></div>
<p>However, it’s not easy to make an “integer or not found” type in C or
JavaScript without some sort of side-channel. Imagine if JavaScript’s
<code>String.indexOf()</code> instead expected continuations on success and
failure and became much less usable as a result:</p>
<div class="sourceCode" id="cb12"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb12-1"><a href="http://feeds.feedburner.com/incodeblog#cb12-1" tabindex="-1"/><span class="ot">unsafeIndexOf ::</span> <span class="dt">String</span> <span class="ot">-&gt;</span> <span class="dt">String</span> <span class="ot">-&gt;</span> <span class="dt">Int</span></span>
<span id="cb12-2"><a href="http://feeds.feedburner.com/incodeblog#cb12-2" tabindex="-1"/></span>
<span id="cb12-3"><a href="http://feeds.feedburner.com/incodeblog#cb12-3" tabindex="-1"/><span class="co">-- vs.</span></span>
<span id="cb12-4"><a href="http://feeds.feedburner.com/incodeblog#cb12-4" tabindex="-1"/></span>
<span id="cb12-5"><a href="http://feeds.feedburner.com/incodeblog#cb12-5" tabindex="-1"/><span class="co">-- takes a success continuation and a failure continuation</span></span>
<span id="cb12-6"><a href="http://feeds.feedburner.com/incodeblog#cb12-6" tabindex="-1"/><span class="ot">indexOf ::</span> <span class="dt">String</span> <span class="ot">-&gt;</span> <span class="dt">String</span> <span class="ot">-&gt;</span> (<span class="dt">Int</span> <span class="ot">-&gt;</span> r) <span class="ot">-&gt;</span> (() <span class="ot">-&gt;</span> r) <span class="ot">-&gt;</span> r</span></code></pre></div>
<p>All of this just to <a href="https://blog.jle.im/entry/faking-adts-and-gadts.html">fake having actual
sum types</a>.</p>
<p>We don’t really have an excuse in Haskell, since we can just return
<code>Maybe</code>:</p>
<div class="sourceCode" id="cb13"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb13-1"><a href="http://feeds.feedburner.com/incodeblog#cb13-1" tabindex="-1"/><span class="co">-- from Data.Vector</span></span>
<span id="cb13-2"><a href="http://feeds.feedburner.com/incodeblog#cb13-2" tabindex="-1"/><span class="ot">elemIndex ::</span> <span class="dt">Eq</span> a <span class="ot">=&gt;</span> a <span class="ot">-&gt;</span> <span class="dt">Vector</span> a <span class="ot">-&gt;</span> <span class="dt">Maybe</span> <span class="dt">Int</span></span></code></pre></div>
<p>Returning <code>Maybe</code> or <code>Option</code> forces the caller to
handle:</p>
<div class="sourceCode" id="cb14"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb14-1"><a href="http://feeds.feedburner.com/incodeblog#cb14-1" tabindex="-1"/><span class="kw">case</span> elemIndex <span class="dv">3</span> myVec <span class="kw">of</span></span>
<span id="cb14-2"><a href="http://feeds.feedburner.com/incodeblog#cb14-2" tabindex="-1"/>  <span class="dt">Just</span> i <span class="ot">-&gt;</span> <span class="co">-- ..</span></span>
<span id="cb14-3"><a href="http://feeds.feedburner.com/incodeblog#cb14-3" tabindex="-1"/>  <span class="dt">Nothing</span> <span class="ot">-&gt;</span> <span class="co">-- ..</span></span></code></pre></div>
<p>and this handling is compiler-enforced. Provided, of course, you don’t <a href="https://blog.cloudflare.com/18-november-2025-outage/">intentionally throw
away your type-safety and compiler checks for no reason</a>. You can even return
<code>Either</code> with an enum for richer responses, and very easily <a href="https://blog.jle.im/entry/inside-my-world-ode-to-functor-and-monad.html">chain
erroring operations using Functor and Monad</a>. In fact, with cheap ADTs, you
can define your own rich result type, like in <em><a href="https://hackage-content.haskell.org/package/unix">unix</a></em>’s
<code>ProcessStatus</code>:</p>
<div class="sourceCode" id="cb15"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb15-1"><a href="http://feeds.feedburner.com/incodeblog#cb15-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">ProcessStatus</span></span>
<span id="cb15-2"><a href="http://feeds.feedburner.com/incodeblog#cb15-2" tabindex="-1"/>   <span class="ot">=</span> <span class="dt">Exited</span> <span class="dt">ExitCode</span></span>
<span id="cb15-3"><a href="http://feeds.feedburner.com/incodeblog#cb15-3" tabindex="-1"/>   <span class="op">|</span> <span class="dt">Terminated</span> <span class="dt">Signal</span> <span class="dt">Bool</span></span>
<span id="cb15-4"><a href="http://feeds.feedburner.com/incodeblog#cb15-4" tabindex="-1"/>   <span class="op">|</span> <span class="dt">Stopped</span> <span class="dt">Signal</span></span></code></pre></div>
<p>Imagine trying to cram all of that information into an <code>int</code>!</p>
<h2 id="unmarked-assumptions">Unmarked Assumptions</h2>
<p>Assumptions kill, and a lot of times we arrive at implicit assumptions in our
code. Unfortunately, even if we add these assumptions in our documentation, it
only takes a minor refactor or lapse in memory for these to cause catastrophic
incidents.</p>
<p>There are the simple cases — consider a <code>mean</code> function:</p>
<div class="sourceCode" id="cb16"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb16-1"><a href="http://feeds.feedburner.com/incodeblog#cb16-1" tabindex="-1"/><span class="co">-- | Warning: do not give an empty list!</span></span>
<span id="cb16-2"><a href="http://feeds.feedburner.com/incodeblog#cb16-2" tabindex="-1"/><span class="ot">mean ::</span> [<span class="dt">Double</span>] <span class="ot">-&gt;</span> <span class="dt">Double</span></span>
<span id="cb16-3"><a href="http://feeds.feedburner.com/incodeblog#cb16-3" tabindex="-1"/>mean xs <span class="ot">=</span> <span class="fu">sum</span> xs <span class="op">/</span> <span class="fu">fromIntegral</span> (<span class="fu">length</span> xs)</span></code></pre></div>
<p>But are you <em>really</em> going to remember to check if your list is empty
<em>every</em> time you give it to <code>mean</code>? No, of course not.
Instead, make it a compiler-enforced constraint.</p>
<div class="sourceCode" id="cb17"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb17-1"><a href="http://feeds.feedburner.com/incodeblog#cb17-1" tabindex="-1"/><span class="ot">mean ::</span> <span class="dt">NonEmpty</span> <span class="dt">Double</span> <span class="ot">-&gt;</span> <span class="dt">Double</span></span>
<span id="cb17-2"><a href="http://feeds.feedburner.com/incodeblog#cb17-2" tabindex="-1"/>mean xs <span class="ot">=</span> <span class="fu">sum</span> xs <span class="op">/</span> <span class="fu">fromIntegral</span> (<span class="fu">length</span> xs)</span></code></pre></div>
<p>Now <code>mean</code> takes a <code>NonEmpty</code> list, which can only be
created safely using <code>nonEmpty :: [a] -&gt; Maybe (NonEmpty a)</code>
(where the caller has to explicitly handle the empty list case, so they’ll never
forget) or from functions that already return <code>NonEmpty</code> by default
(like <code>some :: f a -&gt; f (NonEmpty a)</code> or
<code>group :: Eq a =&gt; [a] -&gt; [NonEmpty a]</code>), allowing you to
beautifully chain post-conditions directly into pre-conditions.</p>
<p>Accessing containers is, in general, very fraught…even things like indexing
lists can send us into a graveyard spiral. Sometimes the issue is more subtle.
This is our reminder to never let these implicit assumptions go unnoticed.</p>
<h3 id="separate-processed-data">Separate Processed Data</h3>
<p>“Shotgun parsing” involves mixing validated and unvalidated data at different
levels in your program. Oftentimes it is considered “fine” because you just need
to remember which inputs are validated and which aren’t…right? In truth, all it
takes is a simple temporary lapse of mental model, a time delay between working
on code, or uncoordinated contributions before things fall apart.</p>
<p>Consider a situation where we validate usernames only on write to the
database.</p>
<div class="sourceCode" id="cb18"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb18-1"><a href="http://feeds.feedburner.com/incodeblog#cb18-1" tabindex="-1"/><span class="ot">validUsername ::</span> <span class="dt">String</span> <span class="ot">-&gt;</span> <span class="dt">Bool</span></span>
<span id="cb18-2"><a href="http://feeds.feedburner.com/incodeblog#cb18-2" tabindex="-1"/>validUsername s <span class="ot">=</span> <span class="fu">all</span> <span class="fu">isAlphaNum</span> s <span class="op">&amp;&amp;</span> <span class="fu">all</span> <span class="fu">isLower</span> s</span>
<span id="cb18-3"><a href="http://feeds.feedburner.com/incodeblog#cb18-3" tabindex="-1"/></span>
<span id="cb18-4"><a href="http://feeds.feedburner.com/incodeblog#cb18-4" tabindex="-1"/><span class="co">-- | Returns 'Nothing' if username is invalid or insertion failed</span></span>
<span id="cb18-5"><a href="http://feeds.feedburner.com/incodeblog#cb18-5" tabindex="-1"/><span class="ot">saveUser ::</span> <span class="dt">Connection</span> <span class="ot">-&gt;</span> <span class="dt">String</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">Maybe</span> <span class="dt">UUID</span>)</span>
<span id="cb18-6"><a href="http://feeds.feedburner.com/incodeblog#cb18-6" tabindex="-1"/>saveUser conn s</span>
<span id="cb18-7"><a href="http://feeds.feedburner.com/incodeblog#cb18-7" tabindex="-1"/>  <span class="op">|</span> validUsername s <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb18-8"><a href="http://feeds.feedburner.com/incodeblog#cb18-8" tabindex="-1"/>      newId <span class="ot">&lt;-</span> query conn <span class="st">"INSERT INTO users (username) VALUES (?) returning user_id"</span> (<span class="dt">Only</span> s)</span>
<span id="cb18-9"><a href="http://feeds.feedburner.com/incodeblog#cb18-9" tabindex="-1"/>      <span class="fu">pure</span> <span class="op">$</span> <span class="kw">case</span> newId <span class="kw">of</span></span>
<span id="cb18-10"><a href="http://feeds.feedburner.com/incodeblog#cb18-10" tabindex="-1"/>        [] <span class="ot">-&gt;</span> <span class="dt">Nothing</span></span>
<span id="cb18-11"><a href="http://feeds.feedburner.com/incodeblog#cb18-11" tabindex="-1"/>        <span class="dt">Only</span> i <span class="op">:</span> _ <span class="ot">-&gt;</span> <span class="dt">Just</span> i</span>
<span id="cb18-12"><a href="http://feeds.feedburner.com/incodeblog#cb18-12" tabindex="-1"/>  <span class="op">|</span> <span class="fu">otherwise</span> <span class="ot">=</span> <span class="fu">pure</span> <span class="dt">Nothing</span></span>
<span id="cb18-13"><a href="http://feeds.feedburner.com/incodeblog#cb18-13" tabindex="-1"/></span>
<span id="cb18-14"><a href="http://feeds.feedburner.com/incodeblog#cb18-14" tabindex="-1"/><span class="ot">getUser ::</span> <span class="dt">Connection</span> <span class="ot">-&gt;</span> <span class="dt">UUID</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">Maybe</span> <span class="dt">String</span>)</span>
<span id="cb18-15"><a href="http://feeds.feedburner.com/incodeblog#cb18-15" tabindex="-1"/>getUser conn uid <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb18-16"><a href="http://feeds.feedburner.com/incodeblog#cb18-16" tabindex="-1"/>  unames <span class="ot">&lt;-</span> query conn <span class="st">"SELECT username FROM users where user_id = ?"</span> (<span class="dt">Only</span> uid)</span>
<span id="cb18-17"><a href="http://feeds.feedburner.com/incodeblog#cb18-17" tabindex="-1"/>  <span class="fu">pure</span> <span class="op">$</span> <span class="kw">case</span> unames <span class="kw">of</span></span>
<span id="cb18-18"><a href="http://feeds.feedburner.com/incodeblog#cb18-18" tabindex="-1"/>    [] <span class="ot">-&gt;</span> <span class="dt">Nothing</span></span>
<span id="cb18-19"><a href="http://feeds.feedburner.com/incodeblog#cb18-19" tabindex="-1"/>    <span class="dt">Only</span> s <span class="op">:</span> _ <span class="ot">-&gt;</span> <span class="dt">Just</span> s</span></code></pre></div>
<p>It <em>should</em> be fine as long as you only ever use <code>saveUser</code>
and <code>getUser</code>…and nobody else has access to the database. But, if
someone hooks up a custom connector, or does some manual modifications, then the
<code>users</code> table will now have an invalid username, bypassing Haskell.
And because of that, <code>getUser</code> can return an invalid string!</p>
<p>Don’t assume that these inconsequential slip-ups won’t happen; assume that
it’s only a matter of time.</p>
<p>Instead, we can bake the state of a validated string into the type
itself:</p>
<div class="sourceCode" id="cb19"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb19-1"><a href="http://feeds.feedburner.com/incodeblog#cb19-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">Username</span> <span class="ot">=</span> <span class="dt">UnsafeUsername</span> <span class="dt">String</span></span>
<span id="cb19-2"><a href="http://feeds.feedburner.com/incodeblog#cb19-2" tabindex="-1"/>  <span class="kw">deriving</span> (<span class="dt">Show</span>, <span class="dt">Eq</span>)</span>
<span id="cb19-3"><a href="http://feeds.feedburner.com/incodeblog#cb19-3" tabindex="-1"/></span>
<span id="cb19-4"><a href="http://feeds.feedburner.com/incodeblog#cb19-4" tabindex="-1"/><span class="co">-- | Our "Smart Constructor"</span></span>
<span id="cb19-5"><a href="http://feeds.feedburner.com/incodeblog#cb19-5" tabindex="-1"/><span class="ot">mkUsername ::</span> <span class="dt">String</span> <span class="ot">-&gt;</span> <span class="dt">Maybe</span> <span class="dt">Username</span></span>
<span id="cb19-6"><a href="http://feeds.feedburner.com/incodeblog#cb19-6" tabindex="-1"/>mkUsername s</span>
<span id="cb19-7"><a href="http://feeds.feedburner.com/incodeblog#cb19-7" tabindex="-1"/>  <span class="op">|</span> validUsername s <span class="ot">=</span> <span class="dt">Just</span> (<span class="dt">UnsafeUsername</span> s)</span>
<span id="cb19-8"><a href="http://feeds.feedburner.com/incodeblog#cb19-8" tabindex="-1"/>  <span class="op">|</span> <span class="fu">otherwise</span>       <span class="ot">=</span> <span class="dt">Nothing</span></span>
<span id="cb19-9"><a href="http://feeds.feedburner.com/incodeblog#cb19-9" tabindex="-1"/></span>
<span id="cb19-10"><a href="http://feeds.feedburner.com/incodeblog#cb19-10" tabindex="-1"/><span class="co">-- | Access the raw string if needed</span></span>
<span id="cb19-11"><a href="http://feeds.feedburner.com/incodeblog#cb19-11" tabindex="-1"/><span class="ot">unUsername ::</span> <span class="dt">Username</span> <span class="ot">-&gt;</span> <span class="dt">String</span></span>
<span id="cb19-12"><a href="http://feeds.feedburner.com/incodeblog#cb19-12" tabindex="-1"/>unUsername (<span class="dt">UnsafeUsername</span> s) <span class="ot">=</span> s</span></code></pre></div>
<p><code>Username</code> and <code>String</code> themselves are not structurally
different — instead, <code>Username</code> is a compiler-enforced tag specifying
it went through a specific required validation function <em>within Haskell</em>,
not just externally verified.</p>
<p>Now <code>saveUser</code> and <code>getUser</code> are safe at the
boundaries:</p>
<div class="sourceCode" id="cb20"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb20-1"><a href="http://feeds.feedburner.com/incodeblog#cb20-1" tabindex="-1"/><span class="ot">saveUser ::</span> <span class="dt">Connection</span> <span class="ot">-&gt;</span> <span class="dt">Username</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">Maybe</span> <span class="dt">UUID</span>)</span>
<span id="cb20-2"><a href="http://feeds.feedburner.com/incodeblog#cb20-2" tabindex="-1"/>saveUser conn s <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb20-3"><a href="http://feeds.feedburner.com/incodeblog#cb20-3" tabindex="-1"/>  newId <span class="ot">&lt;-</span> query conn <span class="st">"INSERT INTO users (username) VALUES (?) returning user_id"</span> (<span class="dt">Only</span> (unUsername s))</span>
<span id="cb20-4"><a href="http://feeds.feedburner.com/incodeblog#cb20-4" tabindex="-1"/>  <span class="fu">pure</span> <span class="op">$</span> <span class="kw">case</span> newId <span class="kw">of</span></span>
<span id="cb20-5"><a href="http://feeds.feedburner.com/incodeblog#cb20-5" tabindex="-1"/>    [] <span class="ot">-&gt;</span> <span class="dt">Nothing</span></span>
<span id="cb20-6"><a href="http://feeds.feedburner.com/incodeblog#cb20-6" tabindex="-1"/>    <span class="dt">Only</span> i <span class="op">:</span> _ <span class="ot">-&gt;</span> <span class="dt">Just</span> i</span>
<span id="cb20-7"><a href="http://feeds.feedburner.com/incodeblog#cb20-7" tabindex="-1"/></span>
<span id="cb20-8"><a href="http://feeds.feedburner.com/incodeblog#cb20-8" tabindex="-1"/><span class="ot">getUser ::</span> <span class="dt">Connection</span> <span class="ot">-&gt;</span> <span class="dt">UUID</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">Maybe</span> <span class="dt">Username</span>)</span>
<span id="cb20-9"><a href="http://feeds.feedburner.com/incodeblog#cb20-9" tabindex="-1"/>getUser conn uid <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb20-10"><a href="http://feeds.feedburner.com/incodeblog#cb20-10" tabindex="-1"/>  unames <span class="ot">&lt;-</span> query conn <span class="st">"SELECT username FROM users where user_id = ?"</span> (<span class="dt">Only</span> uid)</span>
<span id="cb20-11"><a href="http://feeds.feedburner.com/incodeblog#cb20-11" tabindex="-1"/>  <span class="fu">pure</span> <span class="op">$</span> <span class="kw">case</span> unames <span class="kw">of</span></span>
<span id="cb20-12"><a href="http://feeds.feedburner.com/incodeblog#cb20-12" tabindex="-1"/>    [] <span class="ot">-&gt;</span> <span class="dt">Nothing</span></span>
<span id="cb20-13"><a href="http://feeds.feedburner.com/incodeblog#cb20-13" tabindex="-1"/>    <span class="dt">Only</span> s <span class="op">:</span> _ <span class="ot">-&gt;</span> mkUsername s</span></code></pre></div>
<p>(In real code, of course, we would use a more usable indication of failure
than <code>Maybe</code>)</p>
<p>We can even hook this into Haskell’s typeclass system to make this even more
rigorous: <code>Username</code> could have its own <code>FromField</code> and
<code>ToField</code> instances that push the validation to the driver level.</p>
<div class="sourceCode" id="cb21"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb21-1"><a href="http://feeds.feedburner.com/incodeblog#cb21-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">FromField</span> <span class="dt">Username</span> <span class="kw">where</span></span>
<span id="cb21-2"><a href="http://feeds.feedburner.com/incodeblog#cb21-2" tabindex="-1"/>  fromField f mdata <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb21-3"><a href="http://feeds.feedburner.com/incodeblog#cb21-3" tabindex="-1"/><span class="ot">    s ::</span> <span class="dt">String</span> <span class="ot">&lt;-</span> fromField f mdata</span>
<span id="cb21-4"><a href="http://feeds.feedburner.com/incodeblog#cb21-4" tabindex="-1"/>    <span class="kw">case</span> mkUsername s <span class="kw">of</span></span>
<span id="cb21-5"><a href="http://feeds.feedburner.com/incodeblog#cb21-5" tabindex="-1"/>      <span class="dt">Just</span> u  <span class="ot">-&gt;</span> <span class="fu">pure</span> u</span>
<span id="cb21-6"><a href="http://feeds.feedburner.com/incodeblog#cb21-6" tabindex="-1"/>      <span class="dt">Nothing</span> <span class="ot">-&gt;</span> returnError <span class="dt">ConversionFailed</span> f (<span class="st">"Invalid username format: "</span> <span class="op">++</span> s)</span>
<span id="cb21-7"><a href="http://feeds.feedburner.com/incodeblog#cb21-7" tabindex="-1"/></span>
<span id="cb21-8"><a href="http://feeds.feedburner.com/incodeblog#cb21-8" tabindex="-1"/><span class="kw">instance</span> <span class="dt">ToField</span> <span class="dt">Username</span> <span class="kw">where</span></span>
<span id="cb21-9"><a href="http://feeds.feedburner.com/incodeblog#cb21-9" tabindex="-1"/>  toField <span class="ot">=</span> toField <span class="op">.</span> unUsername</span>
<span id="cb21-10"><a href="http://feeds.feedburner.com/incodeblog#cb21-10" tabindex="-1"/></span>
<span id="cb21-11"><a href="http://feeds.feedburner.com/incodeblog#cb21-11" tabindex="-1"/><span class="ot">saveUser ::</span> <span class="dt">Connection</span> <span class="ot">-&gt;</span> <span class="dt">Username</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">Maybe</span> <span class="dt">UUID</span>)</span>
<span id="cb21-12"><a href="http://feeds.feedburner.com/incodeblog#cb21-12" tabindex="-1"/>saveUser conn s <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb21-13"><a href="http://feeds.feedburner.com/incodeblog#cb21-13" tabindex="-1"/>  newId <span class="ot">&lt;-</span> query conn <span class="st">"INSERT INTO users (username) VALUES (?) returning user_id"</span> (<span class="dt">Only</span> s)</span>
<span id="cb21-14"><a href="http://feeds.feedburner.com/incodeblog#cb21-14" tabindex="-1"/>  <span class="fu">pure</span> <span class="op">$</span> <span class="kw">case</span> newId <span class="kw">of</span></span>
<span id="cb21-15"><a href="http://feeds.feedburner.com/incodeblog#cb21-15" tabindex="-1"/>    [] <span class="ot">-&gt;</span> <span class="dt">Nothing</span></span>
<span id="cb21-16"><a href="http://feeds.feedburner.com/incodeblog#cb21-16" tabindex="-1"/>    <span class="dt">Only</span> i <span class="op">:</span> _ <span class="ot">-&gt;</span> <span class="dt">Just</span> i</span>
<span id="cb21-17"><a href="http://feeds.feedburner.com/incodeblog#cb21-17" tabindex="-1"/></span>
<span id="cb21-18"><a href="http://feeds.feedburner.com/incodeblog#cb21-18" tabindex="-1"/><span class="ot">getUser ::</span> <span class="dt">Connection</span> <span class="ot">-&gt;</span> <span class="dt">UUID</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">Maybe</span> <span class="dt">Username</span>)</span>
<span id="cb21-19"><a href="http://feeds.feedburner.com/incodeblog#cb21-19" tabindex="-1"/>getUser conn uid <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb21-20"><a href="http://feeds.feedburner.com/incodeblog#cb21-20" tabindex="-1"/>  unames <span class="ot">&lt;-</span> query conn <span class="st">"SELECT username FROM users where user_id = ?"</span> (<span class="dt">Only</span> uid)</span>
<span id="cb21-21"><a href="http://feeds.feedburner.com/incodeblog#cb21-21" tabindex="-1"/>  <span class="fu">pure</span> <span class="op">$</span> <span class="kw">case</span> unames <span class="kw">of</span></span>
<span id="cb21-22"><a href="http://feeds.feedburner.com/incodeblog#cb21-22" tabindex="-1"/>    [] <span class="ot">-&gt;</span> <span class="dt">Nothing</span></span>
<span id="cb21-23"><a href="http://feeds.feedburner.com/incodeblog#cb21-23" tabindex="-1"/>    <span class="dt">Only</span> s <span class="op">:</span> _ <span class="ot">-&gt;</span> <span class="dt">Just</span> s</span></code></pre></div>
<p>Pushing it to the driver level will also unify everything with the driver’s
error-handling system.</p>
<p>These ideas are elaborated further in one of the best Haskell posts of all
time, <a href="https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/">Parse,
Don’t Validate</a>.</p>
<h3 id="boolean-blindness">Boolean Blindness</h3>
<p>At the heart of it, the previous examples’ cardinal sin was “boolean
blindness”. If we have a predicate like
<code>validUsername :: String -&gt; Bool</code>, we will branch on that
<code>Bool</code> once and throw it away. Instead, by having a function like
<code>mkUsername :: String -&gt; Maybe Username</code>, we <em>keep</em> the
proof alongside the value for the entire lifetime of the value. We basically
pair the string with its proof forever, making them inseparable.</p>
<p>There was another example of such a thing earlier: instead of using
<code>null :: [a] -&gt; Bool</code> and gating a call to <code>mean</code> with
<code>null</code>, we instead use
<code>nonEmpty :: [a] -&gt; Maybe (NonEmpty a)</code>, and pass along the proof
of non-emptiness alongside the value itself. And, for the rest of that list’s
life, it will always be paired with its non-emptiness proof.</p>
<p>Embracing total depravity means always keeping these proofs together, with
the witnesses bundled with the value itself, because if you don’t, someone is
going to assume it exists when it doesn’t, or drop it unnecessarily.</p>
<p>Boolean blindness also has another facet, which is where <code>Bool</code>
itself is not a semantically meaningful type. This is “semantic boolean
blindness”.</p>
<p>The classic example is
<code>filter :: (a -&gt; Bool) -&gt; [a] -&gt; [a]</code>. It might sound silly
until it happens to you, but it is pretty easy to mix up if <code>True</code>
means “keep” or “discard”. After all, a “water filter” only lets water through,
but a “profanity filter” only rejects profanity. Instead, how about
<code>mapMaybe :: (a -&gt; Maybe b) -&gt; [a] -&gt; [b]</code>? In that case, it
is clear that <code>Just</code> results are kept, and the <code>Nothing</code>
results are discarded.</p>
<p>Sometimes, the boolean is ambiguous as to what it means. You can sort of
interpret the <a href="https://en.wikipedia.org/wiki/Mars_Polar_Lander">1999
Mars Polar Lander</a> crash this way. Its functions took a boolean based on the
state of the legs:</p>
<div class="sourceCode" id="cb22"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb22-1"><a href="http://feeds.feedburner.com/incodeblog#cb22-1" tabindex="-1"/><span class="ot">deployThrusters ::</span> <span class="dt">Bool</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span></code></pre></div>
<p>and <code>True</code> and <code>False</code> were misinterpreted. Instead,
they could have considered semantically meaningful types: (simplified)</p>
<div class="sourceCode" id="cb23"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb23-1"><a href="http://feeds.feedburner.com/incodeblog#cb23-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">LegState</span> <span class="ot">=</span> <span class="dt">Extended</span> <span class="op">|</span> <span class="dt">Retracted</span></span>
<span id="cb23-2"><a href="http://feeds.feedburner.com/incodeblog#cb23-2" tabindex="-1"/></span>
<span id="cb23-3"><a href="http://feeds.feedburner.com/incodeblog#cb23-3" tabindex="-1"/><span class="ot">deployThrusters ::</span> <span class="dt">LegState</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span></code></pre></div>
<p>Note that <code>Maybe</code> itself is not immune from “semantic blindness” —
if you find yourself using a lot of anonymous combinators like
<code>Maybe</code> and <code>Either</code> to get around boolean blindness, be
aware of falling into <a href="https://github.com/quchen/articles/blob/master/algebraic-blindness.md">algebraic
blindness</a>!</p>
<h3 id="resource-cleanup">Resource Cleanup</h3>
<p>Clean-up of finite system resources is another area that is very easy to
assume you have a handle on before it gets out of hand and sneaks up on you.</p>
<div class="sourceCode" id="cb24"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb24-1"><a href="http://feeds.feedburner.com/incodeblog#cb24-1" tabindex="-1"/><span class="ot">process ::</span> <span class="dt">Handle</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb24-2"><a href="http://feeds.feedburner.com/incodeblog#cb24-2" tabindex="-1"/></span>
<span id="cb24-3"><a href="http://feeds.feedburner.com/incodeblog#cb24-3" tabindex="-1"/><span class="ot">doTheThing ::</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb24-4"><a href="http://feeds.feedburner.com/incodeblog#cb24-4" tabindex="-1"/>doTheThing path <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb24-5"><a href="http://feeds.feedburner.com/incodeblog#cb24-5" tabindex="-1"/>  h <span class="ot">&lt;-</span> openFile path <span class="dt">ReadMode</span></span>
<span id="cb24-6"><a href="http://feeds.feedburner.com/incodeblog#cb24-6" tabindex="-1"/>  process h</span>
<span id="cb24-7"><a href="http://feeds.feedburner.com/incodeblog#cb24-7" tabindex="-1"/>  hClose h</span></code></pre></div>
<p>A bunch of things could go wrong —</p>
<ul>
<li>You might forget to always <code>hClose</code> a file handle, and if your
files come at you dynamically, you’re going to run out of file descriptors, or
hold on to locks longer than you should</li>
<li>If <code>process</code> throws an exception, we never get to
<code>hClose</code>, and the same issues happen</li>
<li>If another thread throws an asynchronous exception (like a thread
cancellation), you have to make sure the close still happens!</li>
</ul>
<p>The typical solution that other languages (like Python, modern Java) take is
to put everything inside a “block” where quitting the block guarantees the
closure. In Haskell we have the <code>bracket</code> pattern:</p>
<div class="sourceCode" id="cb25"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb25-1"><a href="http://feeds.feedburner.com/incodeblog#cb25-1" tabindex="-1"/><span class="co">-- strongly discourage using `openFile` and `hClose` directly</span></span>
<span id="cb25-2"><a href="http://feeds.feedburner.com/incodeblog#cb25-2" tabindex="-1"/><span class="ot">withFile ::</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> (<span class="dt">Handle</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> r) <span class="ot">-&gt;</span> <span class="dt">IO</span> r</span>
<span id="cb25-3"><a href="http://feeds.feedburner.com/incodeblog#cb25-3" tabindex="-1"/>withFile path <span class="ot">=</span> bracket (openFile path <span class="dt">ReadMode</span>) hClose</span>
<span id="cb25-4"><a href="http://feeds.feedburner.com/incodeblog#cb25-4" tabindex="-1"/></span>
<span id="cb25-5"><a href="http://feeds.feedburner.com/incodeblog#cb25-5" tabindex="-1"/><span class="ot">doTheThing ::</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb25-6"><a href="http://feeds.feedburner.com/incodeblog#cb25-6" tabindex="-1"/>doTheThing path <span class="ot">=</span> withFile path <span class="op">$</span> \h <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb25-7"><a href="http://feeds.feedburner.com/incodeblog#cb25-7" tabindex="-1"/>  process h</span></code></pre></div>
<p>If you never use <code>openFile</code> directly, and always use
<code>withFile</code>, all file usage is safe!</p>
<p>But, admittedly, continuations can be annoying to work with. For example,
what if you wanted to safely open a list of files?</p>
<div class="sourceCode" id="cb26"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb26-1"><a href="http://feeds.feedburner.com/incodeblog#cb26-1" tabindex="-1"/><span class="ot">processAll ::</span> [<span class="dt">Handle</span>] <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb26-2"><a href="http://feeds.feedburner.com/incodeblog#cb26-2" tabindex="-1"/></span>
<span id="cb26-3"><a href="http://feeds.feedburner.com/incodeblog#cb26-3" tabindex="-1"/><span class="ot">doTheThings ::</span> [<span class="dt">FilePath</span>] <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb26-4"><a href="http://feeds.feedburner.com/incodeblog#cb26-4" tabindex="-1"/>doTheThings paths <span class="ot">=</span> <span class="co">-- uh...</span></span></code></pre></div>
<p>All of a sudden, not so fun. And what if you had, for example, a Map of
files, like <code>Map Username FilePath</code>?</p>
<div class="sourceCode" id="cb27"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb27-1"><a href="http://feeds.feedburner.com/incodeblog#cb27-1" tabindex="-1"/><span class="ot">processAll ::</span> <span class="dt">Map</span> <span class="dt">Username</span> <span class="dt">Handle</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb27-2"><a href="http://feeds.feedburner.com/incodeblog#cb27-2" tabindex="-1"/></span>
<span id="cb27-3"><a href="http://feeds.feedburner.com/incodeblog#cb27-3" tabindex="-1"/><span class="ot">doTheThings ::</span> <span class="dt">Map</span> <span class="dt">Username</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb27-4"><a href="http://feeds.feedburner.com/incodeblog#cb27-4" tabindex="-1"/>doTheThings paths <span class="ot">=</span> <span class="co">-- uh...</span></span></code></pre></div>
<p>In another language, at this point, we might just give up and resort to
manual opening and closing of files.</p>
<p>But this is Haskell. We have a better solution: cleanup-tracking monads!</p>
<p>This is a classic usage of <code>ContT</code> to let you chain bracket-like
continuations:</p>
<div class="sourceCode" id="cb28"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb28-1"><a href="http://feeds.feedburner.com/incodeblog#cb28-1" tabindex="-1"/><span class="ot">processTwo ::</span> <span class="dt">Handle</span> <span class="ot">-&gt;</span> <span class="dt">Handle</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb28-2"><a href="http://feeds.feedburner.com/incodeblog#cb28-2" tabindex="-1"/></span>
<span id="cb28-3"><a href="http://feeds.feedburner.com/incodeblog#cb28-3" tabindex="-1"/><span class="ot">doTwoThings ::</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb28-4"><a href="http://feeds.feedburner.com/incodeblog#cb28-4" tabindex="-1"/>doTwoThings path1 path2 <span class="ot">=</span> evalContT <span class="op">$</span> <span class="kw">do</span></span>
<span id="cb28-5"><a href="http://feeds.feedburner.com/incodeblog#cb28-5" tabindex="-1"/>    h1 <span class="ot">&lt;-</span> <span class="dt">ContT</span> <span class="op">$</span> withFile path1</span>
<span id="cb28-6"><a href="http://feeds.feedburner.com/incodeblog#cb28-6" tabindex="-1"/>    h2 <span class="ot">&lt;-</span> <span class="dt">ContT</span> <span class="op">$</span> withFile path2</span>
<span id="cb28-7"><a href="http://feeds.feedburner.com/incodeblog#cb28-7" tabindex="-1"/>    liftIO <span class="op">$</span> processTwo h1 h2</span>
<span id="cb28-8"><a href="http://feeds.feedburner.com/incodeblog#cb28-8" tabindex="-1"/></span>
<span id="cb28-9"><a href="http://feeds.feedburner.com/incodeblog#cb28-9" tabindex="-1"/><span class="ot">processAll ::</span> <span class="dt">Map</span> <span class="dt">Username</span> <span class="dt">Handle</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb28-10"><a href="http://feeds.feedburner.com/incodeblog#cb28-10" tabindex="-1"/></span>
<span id="cb28-11"><a href="http://feeds.feedburner.com/incodeblog#cb28-11" tabindex="-1"/><span class="ot">doTheThings ::</span> <span class="dt">Map</span> <span class="dt">Username</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb28-12"><a href="http://feeds.feedburner.com/incodeblog#cb28-12" tabindex="-1"/>doTheThings paths <span class="ot">=</span> evalContT <span class="op">$</span> <span class="kw">do</span></span>
<span id="cb28-13"><a href="http://feeds.feedburner.com/incodeblog#cb28-13" tabindex="-1"/>    handles <span class="ot">&lt;-</span> <span class="fu">traverse</span> (<span class="dt">ContT</span> <span class="op">.</span> withFile) paths</span>
<span id="cb28-14"><a href="http://feeds.feedburner.com/incodeblog#cb28-14" tabindex="-1"/>    liftIO <span class="op">$</span> processAll handles</span></code></pre></div>
<p>However, using <code>ContT</code> doesn’t allow you to do things like early
cleanups or canceling cleanup events. It forces us into a last-in, first-out
sort of cleanup pattern. If you want to deviate, this might cause you to, for
convenience, go for manual resource management. However, we have tools for more
fine-grained control, we have things like <em><a href="https://hackage.haskell.org/package/resourcet">resourcet</a></em>
<code>ResourceT</code>, which lets you manually control the order of clean-up
events, with the guarantee that all of them <em>eventually</em> happen.</p>
<div class="sourceCode" id="cb29"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb29-1"><a href="http://feeds.feedburner.com/incodeblog#cb29-1" tabindex="-1"/><span class="kw">import</span> <span class="kw">qualified</span> <span class="dt">Data.Map</span> <span class="kw">as</span> <span class="dt">M</span></span>
<span id="cb29-2"><a href="http://feeds.feedburner.com/incodeblog#cb29-2" tabindex="-1"/></span>
<span id="cb29-3"><a href="http://feeds.feedburner.com/incodeblog#cb29-3" tabindex="-1"/><span class="co">-- | Returns set of usernames to close</span></span>
<span id="cb29-4"><a href="http://feeds.feedburner.com/incodeblog#cb29-4" tabindex="-1"/><span class="ot">processAll ::</span> <span class="dt">Map</span> <span class="dt">Username</span> <span class="dt">Handle</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> (<span class="dt">Set</span> <span class="dt">Username</span>)</span>
<span id="cb29-5"><a href="http://feeds.feedburner.com/incodeblog#cb29-5" tabindex="-1"/></span>
<span id="cb29-6"><a href="http://feeds.feedburner.com/incodeblog#cb29-6" tabindex="-1"/><span class="ot">allocateFile ::</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">ResourceT</span> <span class="dt">IO</span> (<span class="dt">ReleaseKey</span>, <span class="dt">Handle</span>)</span>
<span id="cb29-7"><a href="http://feeds.feedburner.com/incodeblog#cb29-7" tabindex="-1"/>allocateFile fp <span class="ot">=</span> allocate (openFile fp <span class="dt">ReadMode</span>) hClose</span>
<span id="cb29-8"><a href="http://feeds.feedburner.com/incodeblog#cb29-8" tabindex="-1"/></span>
<span id="cb29-9"><a href="http://feeds.feedburner.com/incodeblog#cb29-9" tabindex="-1"/><span class="co">-- Guarantees that all handles will eventually close, even if `go` crashes</span></span>
<span id="cb29-10"><a href="http://feeds.feedburner.com/incodeblog#cb29-10" tabindex="-1"/><span class="ot">doTheThings ::</span> <span class="dt">Map</span> <span class="dt">Username</span> <span class="dt">FilePath</span> <span class="ot">-&gt;</span> <span class="dt">IO</span> ()</span>
<span id="cb29-11"><a href="http://feeds.feedburner.com/incodeblog#cb29-11" tabindex="-1"/>doTheThings paths <span class="ot">=</span> runResourceT <span class="op">$</span> <span class="kw">do</span></span>
<span id="cb29-12"><a href="http://feeds.feedburner.com/incodeblog#cb29-12" tabindex="-1"/>    releasersAndHandlers <span class="ot">&lt;-</span> <span class="fu">traverse</span> allocateFile paths</span>
<span id="cb29-13"><a href="http://feeds.feedburner.com/incodeblog#cb29-13" tabindex="-1"/>    go releasersAndHandlers</span>
<span id="cb29-14"><a href="http://feeds.feedburner.com/incodeblog#cb29-14" tabindex="-1"/>  <span class="kw">where</span></span>
<span id="cb29-15"><a href="http://feeds.feedburner.com/incodeblog#cb29-15" tabindex="-1"/>    <span class="co">-- normal operation: slowly releases handlers as we drop them</span></span>
<span id="cb29-16"><a href="http://feeds.feedburner.com/incodeblog#cb29-16" tabindex="-1"/><span class="ot">    go ::</span> <span class="dt">Map</span> <span class="dt">Username</span> (<span class="dt">ReleaseKey</span>, <span class="dt">Handle</span>) <span class="ot">-&gt;</span> <span class="dt">ResourceT</span> <span class="dt">IO</span> ()</span>
<span id="cb29-17"><a href="http://feeds.feedburner.com/incodeblog#cb29-17" tabindex="-1"/>    go currOpen <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb29-18"><a href="http://feeds.feedburner.com/incodeblog#cb29-18" tabindex="-1"/>      toClose <span class="ot">&lt;-</span> liftIO <span class="op">$</span> processAll (<span class="fu">snd</span> <span class="op">&lt;$&gt;</span> currOpen)</span>
<span id="cb29-19"><a href="http://feeds.feedburner.com/incodeblog#cb29-19" tabindex="-1"/>      traverse_ (release <span class="op">.</span> <span class="fu">fst</span>) (currOpen <span class="ot">`M.restrictKeys`</span> toClose)</span>
<span id="cb29-20"><a href="http://feeds.feedburner.com/incodeblog#cb29-20" tabindex="-1"/>      <span class="kw">let</span> newOpen <span class="ot">=</span> currOpen <span class="ot">`M.withoutKeys`</span> toClose</span>
<span id="cb29-21"><a href="http://feeds.feedburner.com/incodeblog#cb29-21" tabindex="-1"/>      unless (M.null newOpen) <span class="op">$</span></span>
<span id="cb29-22"><a href="http://feeds.feedburner.com/incodeblog#cb29-22" tabindex="-1"/>        go newOpen</span></code></pre></div>
<p>Here we get the best of both worlds: the ability to manually close handlers
when they are no longer needed, but also the guarantee that they will eventually
be closed.</p>
<h2 id="embracing-total-depravity">Embracing Total Depravity</h2>
<p>Hopefully these examples, and similar situations, should feel relatable.
We’ve all experienced the biting pain of too much self-trust. Or, too much trust
in our ability to communicate with team members. Or, too much trust in ourselves
6 months from now. The traumas described here <em>should</em> resonate with you
if you have programmed in any capacity for more than a couple of scripts.</p>
<p>The doctrine of total depravity does not mean that we don’t recognize the
ability to write sloppy code that works, or that flow states can enable some
great feats. After all, we all program with a certain sense of <em>imago
machinae</em>. Instead, it means that all such states are <em>fundamentally
unstable</em> in their nature and will always fail at some point. The “total”
doesn’t mean we are totally cooked, it means this eventual reckoning applies to
<em>all</em> such shortcuts.</p>
<p>The problem won’t be solved by “get good”. The problem is solved by utilizing
the tooling we are given, especially since Haskell makes them so accessible and
easy to pull in.</p>
<p>There’s another layer here that comes as a result of embracing this mindset:
you’ll find that you have more mental space to dedicate to things that actually
matter! Instead of worrying about inconsequential minutiae and details of your
flawed abstractions, you can actually think about your business logic, the flow
of your program, and architecting that castle of beauty I know you are capable
of.</p>
<h3 id="in-the-age-of-agentic-coding">In the Age of Agentic Coding</h3>
<p>Before we end, let’s address the elephant in the room. We’re writing this in
2026, in the middle of one of the biggest revolutions in software engineering in
the history of the field. A lot of people have claimed that types and safety are
now no longer important in the age of LLMs and agentic coding.</p>
<p>However, these claims seem to miss the fact that the fundamental issue being
addressed here exists both in LLMs and humans: the limited “context window” and
attention. Humans might be barely able to keep a dozen things in our heads, LLMs
might be able to keep a dozen dozen things, but it will still be ultimately
finite. So, the more we can move concerns out of our context window (be it
biological or mechanical), the less crowded our context windows will be, and the
more productive we will be.</p>
<p>Agentic coding is progressing quickly, and over the past few months I have
been exploring this a lot, using models hands-on. One conclusion I have found
(and, this agrees with everyone else I’ve asked who has been trying the same
thing) is that Haskell’s types, in many ways, are the killer productivity secret
of agentic coding.</p>
<p>Many of my Haskell coding tasks for an LLM agent often involve:</p>
<ol type="1">
<li>How will the types change, or what should the types be?</li>
<li>Ralph Wiggum loop to death until the program typechecks, using
<code>ghci</code> and <code>cabal</code>.</li>
</ol>
<p>And, this isn’t 100% effective, but from personal experience it is much more
effective than the similar situation without typed guardrails for fast feedback,
and without instant compiler feedback. The feedback loop is tighter, the
objectives clearer, the constraints more resilient, the tooling more
utilized.</p>
<p>I have noticed, also, that my LLM agents often check the types of the APIs
using <code>ghci :type</code>, and rarely the documentation of the functions
using <code>ghci :docs</code>. So, any “documentation-based contracts” are
definitely much more likely to explode in your face in this new world than
type-based contracts.</p>
<p>I’m not sure how quickly LLM-based agentic coding will progress, but I am
sure that the accidental “dropping” of concerns will continue to be a
bottleneck. All of the traits described in this post for humans will continue to
be traits of limited context windows for LLMs.</p>
<p>If anything, limited “brain space” might be <em>the</em> bottleneck, for both
humans and LLMs. If provide LLMs with properly “depravity-aware” typed code (and
encourage them to write it by giving them the right iterative tooling), I truly
believe that we have the key to unlocking the full potential of agentic
coding.</p>
<p>And…not whatever <a href="https://x.com/rywalker/status/2003525268821188746">this tweet is</a>.</p>
<h3 id="the-next-step">The Next Step</h3>
<p>Total depravity is all about using types to <em>prevent errors</em>. However,
you can only go so far with defensive programming and carefully picking the
structure of your types. Sometimes, it feels like you are expending a lot of
energy and human effort just picking the perfectly designed data type, only for
things out of your hand to ruin your typed castle.</p>
<p>In the next chapter, we’ll see how a little-discussed aspect of Haskell’s
type system gives you a powerful tool for opening your mind to new avenues of
design that were impossible before. At the same time, we’ll see how we can
leverage universal properties of mathematics itself to help us analyze our code
in unexpected ways.</p>
<p>Let’s explore this further in the next principle of <a href="https://blog.jle.im/entries/series/+five-point-haskell.html">Five-Point
Haskell</a>, <strong>Unconditional Election</strong>!</p>
<h2 id="special-thanks">Special Thanks</h2>
<p>I am very humbled to be supported by an amazing community, who make it
possible for me to devote time to researching and writing these posts. Very
special thanks to my supporter at the “Amazing” level on <a href="https://www.patreon.com/justinle/overview">patreon</a>, Josh Vera! :)</p>
<p>Also thanks to <a href="https://www.reddit.com/r/haskell/comments/1qtxnsm/comment/o3889uy"><em>jackdk</em>’s
comment</a> for highlighting extra resources and context that I believe are very
useful and relevant!</p></div>
    </summary>
    <updated>2026-02-02T15:06:46Z</updated>
    <published>2026-02-02T15:06:46Z</published>
    <category term="Haskell"/>
    <author>
      <name>Justin Le</name>
    </author>
    <source>
      <id>https://blog.jle.im/</id>
      <logo>https://blog.jle.im/img/site_logo.jpg</logo>
      <author>
        <name>Justin Le</name>
        <email>justin@jle.im</email>
      </author>
      <link href="https://blog.jle.im/" rel="alternate" type="text/html"/>
      <link href="http://feeds.feedburner.com/incodeblog" rel="self" type="application/rss+xml"/>
      <rights>Copyright 2020 Justin Le</rights>
      <subtitle>Weblog of Justin Le, covering various adventures in programming and explorations in the worlds of computation physics, and knowledge.</subtitle>
      <title>in Code â€” Entries</title>
      <updated>2026-02-02T15:06:46Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>https://haskellforall.com/2026/02/my-experience-with-vibe-coding</id>
    <link href="https://haskellforall.com/2026/02/my-experience-with-vibe-coding" rel="alternate" type="text/html"/>
    <title>My experience with vibe coding&gt;</title>
    <summary>Vibe coding isn't a silver bullet</summary>
    <updated>2026-02-02T00:00:00Z</updated>
    <published>2026-02-02T00:00:00Z</published>
    <source>
      <id>https://haskellforall.com</id>
      <author>
        <name>Gabriella Gonzalez</name>
      </author>
      <link href="https://haskellforall.com" rel="alternate" type="text/html"/>
      <link href="https://haskellforall.com/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>A blog about Haskell and functional programming.</subtitle>
      <title>Haskell for all</title>
      <updated>2026-03-17T14:41:11Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://wickstrom.tech/2026-01-28-there-and-back-again-from-quickstrom-to-bombadil.html</id>
    <link href="https://wickstrom.tech/2026-01-28-there-and-back-again-from-quickstrom-to-bombadil.html" rel="alternate" type="text/html"/>
    <title>There and Back Again: From Quickstrom to Bombadil</title>
    <summary>Today I’m announcing and open-sourcing the Bombadil project — a brand
new property-based browser testing framework. I started working on this
when joining Antithesis two months ago. While still in its infancy,
we’ve decided to build it in the open and share our progress from the
start.</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Today I’m announcing and open-sourcing the <a href="https://github.com/antithesishq/bombadil">Bombadil</a> project — a brand new property-based browser testing framework. I started working on this when joining Antithesis two months ago. While still in its infancy, we’ve decided to build it in the open and share our progress from the start.</p> <p>We decided on the name Bombadil last week. A few days later, this exploded:</p> <blockquote class="twitter-tweet"> <p dir="ltr" lang="en"> It's going to be tough for startups when all the Lord of the Rings names are taken and the only thing left is something like Bombadil AI. </p> — Patrick Collison (<span class="citation">@patrickc</span>) <a href="https://twitter.com/patrickc/status/2015562569105465347?ref_src=twsrc%5Etfw">January 25, 2026</a> </blockquote>  <p>While <a href="https://github.com/quickstrom/quickstrom">Quickstrom</a> proved its worth, <a href="https://dl.acm.org/doi/10.1145/3519939.3523728">finding bugs in more than half of the TodoMVC</a> apps, Bombadil aims to improve on its shortcomings while also envisioning a more ambitious future for generative testing of web apps: faster and smarter state space exploration, a modern and usable specification language, better tools for reproducing and debugging test failures, and a better distribution story.</p> <p>I consider Bombadil the successor of Quickstrom. After years of trying to sustain Quickstrom through various models, I now have a much better answer: building it at Antithesis, making something valuable that is open and free to use, while also strengthening the company’s commercial offering. Bombadil can be used locally or in CI to test your web apps early. Power users can take it further and run Bombadil within Antithesis and its deterministic hypervisor. That gives you perfect reproductions of failed tests. You can even combine Bombadil with other workloads in Antithesis and test your entire stack deterministically. Isn’t that the holy grail of testing?</p> <p>Bombadil is built from scratch with a focus on accessibility: a new specification language and better tooling for writing and maintaining specs. Right now, I’m working on a specification DSL in TypeScript. It’s based on <a href="https://en.wikipedia.org/wiki/Linear_temporal_logic">linear temporal logic</a>, just as Quickstrom, but aims to be a lot more ergonomic. Here’s an example that verifies that error messages eventually disappear:</p> <div class="sourceCode" id="cb1"><pre class="sourceCode typescript"><code class="sourceCode typescript"><span id="cb1-1"><a href="https://wickstrom.tech/feed.xml#cb1-1" tabindex="-1"/><span class="kw">const</span> errorMessage <span class="op">=</span> <span class="fu">extract</span>(</span> <span id="cb1-2"><a href="https://wickstrom.tech/feed.xml#cb1-2" tabindex="-1"/> (state) <span class="kw">=&gt;</span></span> <span id="cb1-3"><a href="https://wickstrom.tech/feed.xml#cb1-3" tabindex="-1"/> state<span class="op">.</span><span class="at">document</span><span class="op">.</span><span class="at">body</span><span class="op">.</span><span class="fu">querySelector</span>(<span class="st">".error"</span>)</span> <span id="cb1-4"><a href="https://wickstrom.tech/feed.xml#cb1-4" tabindex="-1"/> <span class="op">?.</span><span class="at">textContent</span> </span> <span id="cb1-5"><a href="https://wickstrom.tech/feed.xml#cb1-5" tabindex="-1"/> <span class="op">??</span> <span class="kw">null</span></span> <span id="cb1-6"><a href="https://wickstrom.tech/feed.xml#cb1-6" tabindex="-1"/>)<span class="op">;</span></span> <span id="cb1-7"><a href="https://wickstrom.tech/feed.xml#cb1-7" tabindex="-1"/></span> <span id="cb1-8"><a href="https://wickstrom.tech/feed.xml#cb1-8" tabindex="-1"/><span class="im">export</span> <span class="kw">const</span> errorEventuallyDisappears <span class="op">=</span> <span class="fu">always</span>(</span> <span id="cb1-9"><a href="https://wickstrom.tech/feed.xml#cb1-9" tabindex="-1"/> <span class="fu">condition</span>(() <span class="kw">=&gt;</span> errorMessage <span class="op">!==</span> <span class="kw">null</span>)<span class="op">.</span><span class="fu">implies</span>(</span> <span id="cb1-10"><a href="https://wickstrom.tech/feed.xml#cb1-10" tabindex="-1"/> <span class="fu">eventually</span>(() <span class="kw">=&gt;</span> errorMessage <span class="op">===</span> <span class="kw">null</span>)</span> <span id="cb1-11"><a href="https://wickstrom.tech/feed.xml#cb1-11" tabindex="-1"/> <span class="op">.</span><span class="fu">within</span>(<span class="dv">5</span><span class="op">,</span> <span class="st">"seconds"</span>)</span> <span id="cb1-12"><a href="https://wickstrom.tech/feed.xml#cb1-12" tabindex="-1"/> )</span> <span id="cb1-13"><a href="https://wickstrom.tech/feed.xml#cb1-13" tabindex="-1"/>)<span class="op">;</span></span></code></pre></div> <p>Both the original hacky PureScript DSL and the bespoke language <em>Specstrom</em> were huge obstacles to adoption. Today, TypeScript is a widely adopted language among quality-minded web developers, and if you’re not into that, Bombadil will work with plain JavaScript too.</p> <p>We’re writing Bombadil in Rust, leveraging its excellent ecosystem to ship a single statically-linked executable — download for your platform, point it at a Chromium-based browser, and off you go!</p> <p>Bombadil is early but usable. Check it out on <a href="https://github.com/antithesishq/bombadil">GitHub</a>, try it on your projects, and let us know what breaks. Also let us know what <em>it</em> breaks in your systems! Join us on <a href="https://discord.gg/antithesis">Discord</a> for help and discussion, or follow development on <a href="https://x.com/owickstrom">Twitter/X</a>. This is just the beginning — we’re actively seeking feedback and early adopters to help shape where this goes.</p></div>
    </content>
    <updated>2026-01-27T23:00:00Z</updated>
    <published>2026-01-27T23:00:00Z</published>
    <source>
      <id>https://wickstrom.tech/</id>
      <author>
        <name>Oskar Wickström</name>
      </author>
      <link href="https://wickstrom.tech/feed.xml" rel="self" type="application/atom+xml"/>
      <link href="https://wickstrom.tech/" rel="alternate" type="text/html"/>
      <subtitle>Software design, testing, functional programming, and other delightful things.</subtitle>
      <title>Oskar WickstrÃ¶m</title>
      <updated>2026-03-09T13:09:42Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>Buzzsprout-18565558</id>
    <link href="https://www.buzzsprout.com/1817535/episodes/18565558-76-jeffrey-young.mp3" length="46241454" rel="enclosure" type="audio/mpeg"/>
    <title>76: Jeffrey Young</title>
    <summary>Welcome to the Haskell Interlude. Today, Matti and Mike talk to Jeffrey Young. Jeff has had a long history of working with Haskell and on ghc itself. We talk about what makes Haskell so compelling, the good and bad of highly optimized code and the beauty of well-modularized code, how to get into compiler development, and how to benefit from Domain-Driven Design.  Jeff is currently on the job market - if you want to get in touch, email him at mailto:jmy6342@gmail.com.</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Welcome to the Haskell Interlude. Today, Matti and Mike talk to<br/>Jeffrey Young. Jeff has had a long history of working with Haskell and<br/>on ghc itself. We talk about what makes Haskell so compelling, the<br/>good and bad of highly optimized code and the beauty of<br/>well-modularized code, how to get into compiler development, and how<br/>to benefit from Domain-Driven Design.<br/><br/>Jeff is currently on the job market - if you want to get in touch,<br/>email him atÂ <a href="mailto:jmy6342@gmail.com">mailto:jmy6342@gmail.com</a>.</p></div>
    </content>
    <updated>2026-01-25T14:00:00Z</updated>
    <published>2026-01-25T14:00:00Z</published>
    <author>
      <name>Haskell Podcast</name>
    </author>
    <source>
      <id>https://haskell.foundation/podcast/</id>
      <logo>https://storage.buzzsprout.com/tnk1ztokn5vmeiufqmr4kkp37mw2?.jpg</logo>
      <category scheme="http://www.itunes.com/" term="Technology"/>
      <author>
        <name>Haskell Podcast</name>
      </author>
      <link href="https://rss.buzzsprout.com/1817535.rss" rel="self" type="application/rss+xml"/>
      <link href="https://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <link href="https://haskell.foundation/podcast/" rel="alternate" type="text/html"/>
      <rights>Â© 2026 The Haskell Interlude</rights>
      <subtitle>This is the Haskell Interlude, where the five co-hosts (Wouter Swierstra, Andres LÃ¶h, Alejandro Serrano, Niki Vazou, and Joachim Breitner) chat with Haskell guests!</subtitle>
      <title>The Haskell Interlude</title>
      <updated>2026-04-10T16:12:47Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://magnus.therning.org/2026-01-25-more-on-the-switch-to-eglot.html</id>
    <link href="https://magnus.therning.org/2026-01-25-more-on-the-switch-to-eglot.html" rel="alternate" type="text/html"/>
    <title>More on the switch to eglot</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>
Since the <a href="https://magnus.therning.org/2026-01-19-trying-eglot,-again.html">switching to eglot</a> I've ended up making a few related changes.
</p>
<div class="outline-2" id="outline-container-org8cdfce1">
<h2 id="org8cdfce1">Replacing flycheck with flymake</h2>
<div class="outline-text-2" id="text-org8cdfce1">
<p>
Since <code>eglot</code> it's written to work with other packages in core, which means it
integrates with <a href="https://www.gnu.org/software/emacs/manual/html_mono/flymake.html"><code>flymake</code></a>. The switch comprised
</p>

<ul class="org-ul">
<li>Use <code>:ensure nil</code> to make sure <code>elpaca</code> knows there's nothing to download.</li>
<li>Add a call to <code>flymake-mode</code> to <code>prog-mode-hook</code>.</li>
<li>Define two functions to toggle showing a list of diagnostics for the current
buffer and the project.</li>
<li/>

<li>Redefine the relevant keybindings.</li>
</ul>

<p>
The two functions for toggling showing diagnostics look like this
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">defun</span> <span class="org-function-name">mes/toggle-flymake-buffer-diagnostics</span> <span class="org-rainbow-delimiters-depth-2">()</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">interactive</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">if-let*</span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-rainbow-delimiters-depth-4">(</span>window <span class="org-rainbow-delimiters-depth-5">(</span>get-buffer-window <span class="org-rainbow-delimiters-depth-6">(</span>flymake--diagnostics-buffer-name<span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span>
      <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-keyword">save-selected-window</span> <span class="org-rainbow-delimiters-depth-4">(</span>quit-window nil window<span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span>
    <span class="org-rainbow-delimiters-depth-3">(</span>flymake-show-buffer-diagnostics<span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>

<span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">defun</span> <span class="org-function-name">mes/toggle-flymake-project-diagnostics</span> <span class="org-rainbow-delimiters-depth-2">()</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">interactive</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">if-let*</span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-rainbow-delimiters-depth-4">(</span>window <span class="org-rainbow-delimiters-depth-5">(</span>get-buffer-window <span class="org-rainbow-delimiters-depth-6">(</span>flymake--project-diagnostics-buffer <span class="org-rainbow-delimiters-depth-7">(</span>projectile-project-root<span class="org-rainbow-delimiters-depth-7">)</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span>
      <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-keyword">save-selected-window</span> <span class="org-rainbow-delimiters-depth-4">(</span>quit-window nil window<span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span>
    <span class="org-rainbow-delimiters-depth-3">(</span>flymake-show-project-diagnostics<span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
And the changed keybindings are
</p>

<table>


<colgroup>
<col class="org-left"/>

<col class="org-left"/>
</colgroup>
<thead>
<tr>
<th class="org-left" scope="col">flycheck</th>
<th class="org-left" scope="col">flymake</th>
</tr>
</thead>
<tbody>
<tr>
<td class="org-left">flycheck-next-error</td>
<td class="org-left">flymake-goto-next-error</td>
</tr>

<tr>
<td class="org-left">flycheck-previous-error</td>
<td class="org-left">flymake-goto-prev-error</td>
</tr>

<tr>
<td class="org-left">mes/toggle-flycheck-error-list</td>
<td class="org-left">mes/toggle-flymake-buffer-diagnostics</td>
</tr>

<tr>
<td class="org-left">mes/toggle-flycheck-projectile-error-list</td>
<td class="org-left">mes/toggle-flymake-project-diagnostics</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="outline-2" id="outline-container-orgadc4f2e">
<h2 id="orgadc4f2e">Using <code>with-eval-after-load</code> instead of <code>:after eglot</code></h2>
<div class="outline-text-2" id="text-orgadc4f2e">
<p>
When it comes to <code>use-package</code> I keep on being surprised, and after the switch
to <code>elpaca</code> I've found some new surprises. One of them was that using <code>:after
eglot</code> like this
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">use-package</span> haskell-ng-mode
  <span class="org-builtin">:afer</span> eglot
  <span class="org-builtin">:ensure</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-builtin">:type</span> git
           <span class="org-builtin">:repo</span> <span class="org-string">"git@gitlab.com:magus/haskell-ng-mode.git"</span>
           <span class="org-builtin">:branch</span> <span class="org-string">"main"</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-builtin">:init</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>add-to-list 'major-mode-remap-alist '<span class="org-rainbow-delimiters-depth-3">(</span>haskell-mode . haskell-ng-mode<span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>add-to-list 'eglot-server-programs '<span class="org-rainbow-delimiters-depth-3">(</span>haskell-ng-mode <span class="org-string">"haskell-language-server-wrapper"</span> <span class="org-string">"--lsp"</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">setq-default</span> eglot-workspace-configuration
                <span class="org-rainbow-delimiters-depth-3">(</span>plist-put eglot-workspace-configuration
                           <span class="org-builtin">:haskell</span>
                           '<span class="org-rainbow-delimiters-depth-4">(</span><span class="org-builtin">:formattingProvider</span> <span class="org-string">"fourmolu"</span>
                             <span class="org-builtin">:plugin</span> <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-builtin">:stan</span> <span class="org-rainbow-delimiters-depth-6">(</span><span class="org-builtin">:global-on</span> <span class="org-builtin">:json-false</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  ...
  <span class="org-builtin">:hook</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>haskell-ng-mode . eglot-ensure<span class="org-rainbow-delimiters-depth-2">)</span>
  ...<span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
would delay initialisation until after <code>eglot</code> had been loaded. However, it
turned out that nothing in <code>:init ...</code> seemed to run and upon opening a haskell file
no mode was loaded.
</p>

<p>
After a bit of thinking and tinkering I got it working by removing <code>:after
eglot</code> and using <code>with-eval-after-load</code>
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">use-package</span> haskell-ng-mode
  <span class="org-builtin">:ensure</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-builtin">:type</span> git
           <span class="org-builtin">:repo</span> <span class="org-string">"git@gitlab.com:magus/haskell-ng-mode.git"</span>
           <span class="org-builtin">:branch</span> <span class="org-string">"main"</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-builtin">:init</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>add-to-list 'major-mode-remap-alist '<span class="org-rainbow-delimiters-depth-3">(</span>haskell-mode . haskell-ng-mode<span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">with-eval-after-load</span> 'eglot
    <span class="org-rainbow-delimiters-depth-3">(</span>add-to-list 'eglot-server-programs '<span class="org-rainbow-delimiters-depth-4">(</span>haskell-ng-mode <span class="org-string">"haskell-language-server-wrapper"</span> <span class="org-string">"--lsp"</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span>
    <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-keyword">setq-default</span> eglot-workspace-configuration
                  <span class="org-rainbow-delimiters-depth-4">(</span>plist-put eglot-workspace-configuration
                             <span class="org-builtin">:haskell</span>
                             '<span class="org-rainbow-delimiters-depth-5">(</span><span class="org-builtin">:formattingProvider</span> <span class="org-string">"fourmolu"</span>
                               <span class="org-builtin">:plugin</span> <span class="org-rainbow-delimiters-depth-6">(</span><span class="org-builtin">:stan</span> <span class="org-rainbow-delimiters-depth-7">(</span><span class="org-builtin">:global-on</span> <span class="org-builtin">:json-false</span><span class="org-rainbow-delimiters-depth-7">)</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  ...
  <span class="org-builtin">:hook</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>haskell-ng-mode . eglot-ensure<span class="org-rainbow-delimiters-depth-2">)</span>
  ...<span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
That change worked for haskell, and it seemed to work for python too, but after
a little while I realised that python needed a bit more attention.
</p>
</div>
</div>
<div class="outline-2" id="outline-container-orgc610d55">
<h2 id="orgc610d55">Getting the configuration for Python to work properly</h2>
<div class="outline-text-2" id="text-orgc610d55">
<p>
The python setup looked like this
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">use-package</span> python
  <span class="org-builtin">:init</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>add-to-list 'major-mode-remap-alist '<span class="org-rainbow-delimiters-depth-3">(</span>python-mode . python-ts-mode<span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">with-eval-after-load</span> 'eglot
    <span class="org-rainbow-delimiters-depth-3">(</span>assoc-delete-all '<span class="org-rainbow-delimiters-depth-4">(</span>python-mode python-ts-mode<span class="org-rainbow-delimiters-depth-4">)</span> eglot-server-programs<span class="org-rainbow-delimiters-depth-3">)</span>
    <span class="org-rainbow-delimiters-depth-3">(</span>add-to-list 'eglot-server-programs
                 `<span class="org-rainbow-delimiters-depth-4">(</span><span class="org-rainbow-delimiters-depth-5">(</span>python-mode python-ts-mode<span class="org-rainbow-delimiters-depth-5">)</span> . ,<span class="org-rainbow-delimiters-depth-5">(</span>eglot-alternatives
                                                    '<span class="org-rainbow-delimiters-depth-6">(</span><span class="org-rainbow-delimiters-depth-7">(</span><span class="org-string">"rass"</span> <span class="org-string">"python"</span><span class="org-rainbow-delimiters-depth-7">)</span> <span class="org-string">"pylsp"</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  ...
  <span class="org-builtin">:hook</span> <span class="org-rainbow-delimiters-depth-2">(</span>python-ts-mode . eglot-ensure<span class="org-rainbow-delimiters-depth-2">)</span>
  ...<span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
and it worked all right, but then I visited the package (using <code>elpaca-visit</code>)
and realised that the downloaded package was all of emacs. That's a bit of
overkill, I'd say.
</p>

<p>
However, adding <code>:ensure nil</code> didn't have the expected effect of just using the
version that's in core. Instead the whole configuration seemed to never take
effect and again I was back to the situation where I had to jump to
<code>python-ts-mode</code> manually.
</p>

<p>
The documentation for <code>use-package</code> says that <code>:init</code> is for
</p>

<blockquote>
<p>
Code to run before PACKAGE-NAME has been loaded.
</p>
</blockquote>

<p>
but I'm guessing "before" isn't quite before enough. Then I noticed <code>:preface</code>
with the description
</p>

<blockquote>
<p>
Code to be run before everything except <code>:disabled</code>; this can be used to define
functions for use in <code>:if</code>, or that should be seen by the byte-compiler.
</p>
</blockquote>

<p>
and yes, "before everything" is early enough. The final python configuration
looks like this
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">use-package</span> python
  <span class="org-builtin">:ensure</span> nil
  <span class="org-builtin">:preface</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>add-to-list 'major-mode-remap-alist '<span class="org-rainbow-delimiters-depth-3">(</span>python-mode . python-ts-mode<span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-builtin">:init</span>
  <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">with-eval-after-load</span> 'eglot
    <span class="org-rainbow-delimiters-depth-3">(</span>assoc-delete-all '<span class="org-rainbow-delimiters-depth-4">(</span>python-mode python-ts-mode<span class="org-rainbow-delimiters-depth-4">)</span> eglot-server-programs<span class="org-rainbow-delimiters-depth-3">)</span>
    <span class="org-rainbow-delimiters-depth-3">(</span>add-to-list 'eglot-server-programs
                 `<span class="org-rainbow-delimiters-depth-4">(</span><span class="org-rainbow-delimiters-depth-5">(</span>python-mode python-ts-mode<span class="org-rainbow-delimiters-depth-5">)</span> . ,<span class="org-rainbow-delimiters-depth-5">(</span>eglot-alternatives
                                                    '<span class="org-rainbow-delimiters-depth-6">(</span><span class="org-rainbow-delimiters-depth-7">(</span><span class="org-string">"rass"</span> <span class="org-string">"python"</span><span class="org-rainbow-delimiters-depth-7">)</span> <span class="org-string">"pylsp"</span><span class="org-rainbow-delimiters-depth-6">)</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
  ...
  <span class="org-builtin">:hook</span> <span class="org-rainbow-delimiters-depth-2">(</span>python-ts-mode . eglot-ensure<span class="org-rainbow-delimiters-depth-2">)</span>
  ...<span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>
</div>
</div>
<div class="outline-2" id="outline-container-orgc2d0d13">
<h2 id="orgc2d0d13">Closing remark</h2>
<div class="outline-text-2" id="text-orgc2d0d13">
<p>
I'm still not sure I have the correct intuition about how to use <code>use-package</code>,
but hopefully it's <i>more</i> correct now than before. I have a growing suspicion
that <code>use-package</code> changes behaviour based on the package manager I use. Or
maybe it's just that some package managers make <code>use-package</code> more forgiving of
bad use.
</p>
</div>
</div>
<div class="taglist"><a href="https://magnus.therning.org/tags.html">Tags</a>: <a href="https://magnus.therning.org/tag-eglot.html">eglot</a> <a href="https://magnus.therning.org/tag-emacs.html">emacs</a> </div></div>
    </summary>
    <updated>2026-01-25T13:18:00Z</updated>
    <published>2026-01-25T13:18:00Z</published>
    <category term="eglot"/>
    <category term="emacs"/>
    <source>
      <id>https://magnus.therning.org/</id>
      <author>
        <name>Magnus Therning</name>
      </author>
      <link href="https://magnus.therning.org/" rel="alternate" type="text/html"/>
      <link href="https://magnus.therning.org/feed.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Magnus web site</subtitle>
      <title>Magnus web site</title>
      <updated>2026-02-17T23:09:47Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>https://haskellforall.com/2026/01/typesafe-eval</id>
    <link href="https://haskellforall.com/2026/01/typesafe-eval" rel="alternate" type="text/html"/>
    <title>Type-safe eval in Grace&gt;</title>
    <summary>The case for principled eval support</summary>
    <updated>2026-01-20T00:00:00Z</updated>
    <published>2026-01-20T00:00:00Z</published>
    <source>
      <id>https://haskellforall.com</id>
      <author>
        <name>Gabriella Gonzalez</name>
      </author>
      <link href="https://haskellforall.com" rel="alternate" type="text/html"/>
      <link href="https://haskellforall.com/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>A blog about Haskell and functional programming.</subtitle>
      <title>Haskell for all</title>
      <updated>2026-03-17T14:41:11Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://magnus.therning.org/2026-01-19-trying-eglot,-again.html</id>
    <link href="https://magnus.therning.org/2026-01-19-trying-eglot,-again.html" rel="alternate" type="text/html"/>
    <title>Trying eglot, again</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>
I've been using <a href="https://emacs-lsp.github.io/lsp-mode/">lsp-mode</a> since I switched to Emacs several years ago. When <a href="https://github.com/joaotavora/eglot">eglot</a>
made into Emacs core I used it very briefly but quickly switched back. Mainly I
found eglot a bit too bare-bones; I liked some of the bells and whistles of
<code>lsp-ui</code>. Fast-forward a few years and I've grown a bit tired of those bells and
whistles. Specifically that it's difficult to make <code>lsp-ui-sideline</code> and
<code>lsp-ui-doc</code> work well together. <code>lsp-ui-sidedline</code> is shown on the right side,
which is good, but combining it with <code>lsp-ui-doc</code> leads to situations where the
popup covers the sideline. What I've done so far is centre the line to bring the
sideline text out. I was playing a little bit with making the setting of
<code>lsp-ui-doc-position</code> change depending on the location of the current position.
It didn't work that well though so I decided to try to find a simpler setup.
Instead of simplifying the setup of <code>lsp-config</code> I thought I'd give <code>eglot</code>
another shot.
</p>
<div class="outline-2" id="outline-container-orgb1f7433">
<h2 id="orgb1f7433">Basic setup</h2>
<div class="outline-text-2" id="text-orgb1f7433">
<p>
I removed the statements pulling in <code>lsp-mode</code>, <code>lsp-ui</code>, and all
language-specific packages like <code>lsp-haskell</code>. Then I added this to configure
<code>eglot</code>
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">use-package</span> eglot
  <span class="org-builtin">:ensure</span> nil
  <span class="org-builtin">:custom</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>eglot-autoshutdown t<span class="org-rainbow-delimiters-depth-2">)</span>
  <span class="org-rainbow-delimiters-depth-2">(</span>eglot-confirm-server-edits '<span class="org-rainbow-delimiters-depth-3">(</span><span class="org-rainbow-delimiters-depth-4">(</span>eglot-rename . nil<span class="org-rainbow-delimiters-depth-4">)</span>
                                <span class="org-rainbow-delimiters-depth-4">(</span>t . diff<span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
The rest was mainly just switching <code>lsp-mode</code> functions for <code>eglot</code> functions.
</p>

<table>


<colgroup>
<col class="org-left"/>

<col class="org-left"/>
</colgroup>
<thead>
<tr>
<th class="org-left" scope="col">lsp-mode function</th>
<th class="org-left" scope="col">eglot function</th>
</tr>
</thead>
<tbody>
<tr>
<td class="org-left"><code>lsp-deferred</code></td>
<td class="org-left"><code>eglot-ensure</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-describe-thing-at-point</code></td>
<td class="org-left"><code>eldoc</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-execute-code-action</code></td>
<td class="org-left"><code>eglot-code-actions</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-find-type-definition</code></td>
<td class="org-left"><code>eglot-find-typeDefinition</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-format-buffer</code></td>
<td class="org-left"><code>eglot-format-buffer</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-format-region</code></td>
<td class="org-left"><code>eglot-format</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-organize-imports</code></td>
<td class="org-left"><code>eglot-code-action-organize-imports</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-rename</code></td>
<td class="org-left"><code>eglot-rename</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-workspace-restart</code></td>
<td class="org-left"><code>eglot-reconnect</code></td>
</tr>

<tr>
<td class="org-left"><code>lsp-workspace-shutdown</code></td>
<td class="org-left"><code>eglot-shutdown</code></td>
</tr>
</tbody>
</table>

<p>
I haven't verified that the list is fully correct yet, but it looks good so far.
</p>

<p>
The one thing I might miss is lenses, and using <code>lsp-avy-lens</code>. However,
everything that I use lenses for can be done using actions, and to be honest I
don't think I'll miss the huge lens texts from missing type annotations in
Haskell.
</p>
</div>
</div>
<div class="outline-2" id="outline-container-orgd8ac829">
<h2 id="orgd8ac829">Configuration</h2>
<div class="outline-text-2" id="text-orgd8ac829">
<p>
One good thing about <code>lsp-mode</code>'s use of language-specific packages is that
configuration of the various servers is performed through functions. This makes
it easy to discover what options are available, though it also means not all
options may be available. In <code>eglot</code> configuration is less organised, I have to
know about the options for each language server and put the options into
<code>eglot-workspace-configuration</code> myself. It's not always easy to track down what
options are available, and I've found no easy way to verify the settings. For
instance, with <code>lsp-mode</code> I configures <a href="https://github.com/haskell/haskell-language-server">HLS</a> like this
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span>lsp-haskell-formatting-provider <span class="org-string">"fourmolu"</span><span class="org-rainbow-delimiters-depth-1">)</span>
<span class="org-rainbow-delimiters-depth-1">(</span>lsp-haskell-plugin-stan-global-on nil<span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
which translates to this for <code>eglot</code>
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">setq-default</span> eglot-workspace-configuration
              <span class="org-rainbow-delimiters-depth-2">(</span>plist-put eglot-workspace-configuration
                         <span class="org-builtin">:haskell</span>
                         '<span class="org-rainbow-delimiters-depth-3">(</span><span class="org-builtin">:formattingProvider</span> <span class="org-string">"fourmolu"</span>
                           <span class="org-builtin">:plugin</span> <span class="org-rainbow-delimiters-depth-4">(</span><span class="org-builtin">:stan</span> <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-builtin">:global-on</span> <span class="org-builtin">:json-false</span><span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
and I can verify that this configuration has taken effect because I know enough
about the Haskell tools.
</p>

<p>
I do some development in Python and I used to configure <a href="https://pypi.org/project/python-lsp-server/">pylsp</a> like this
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span>lsp-pylsp-plugins-mypy-enabled t<span class="org-rainbow-delimiters-depth-1">)</span>
<span class="org-rainbow-delimiters-depth-1">(</span>lsp-pylsp-plugins-ruff-enabled t<span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
which I <i>think</i> translates to this for <code>eglot</code>
</p>

<div class="org-src-container">
<pre class="src src-emacs-lisp"><code><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">setq-default</span> eglot-workspace-configuration
              <span class="org-rainbow-delimiters-depth-2">(</span>plist-put eglot-workspace-configuration
                         <span class="org-builtin">:pylsp</span>
                         '<span class="org-rainbow-delimiters-depth-3">(</span><span class="org-builtin">:plugins</span> <span class="org-rainbow-delimiters-depth-4">(</span><span class="org-builtin">:ruff</span> <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-builtin">:enabled</span> t<span class="org-rainbow-delimiters-depth-5">)</span>
                                     <span class="org-builtin">:mypy</span> <span class="org-rainbow-delimiters-depth-5">(</span><span class="org-builtin">:enabled</span> t<span class="org-rainbow-delimiters-depth-5">)</span><span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span><span class="org-rainbow-delimiters-depth-1">)</span>
</code></pre>
</div>

<p>
but I don't know any convenient way of verifying these settings. I'm simply not
familiar enough with the Python tools. I can check the value of
<code>eglot-workspace-configuration</code> by inspecting it or calling
<code>eglot-show-workspace-configuration</code> but is there really no way of asking the
language server for its active configuration?
</p>
</div>
</div>
<div class="outline-2" id="outline-container-org3d20108">
<h2 id="org3d20108">Closing remark</h2>
<div class="outline-text-2" id="text-org3d20108">
<p>
The last time I gave up on <code>eglot</code> very quickly, probably too quickly to be
honest. I made these changes to my configuration over the weekend, so the real
test of <code>eglot</code> starts when I'm back in the office. I have a feeling I'll stick
to it longer this time.
</p>
</div>
</div>
<div class="taglist"><a href="https://magnus.therning.org/tags.html">Tags</a>: <a href="https://magnus.therning.org/tag-eglot.html">eglot</a> <a href="https://magnus.therning.org/tag-emacs.html">emacs</a> <a href="https://magnus.therning.org/tag-lsp-mode.html">lsp-mode</a> </div></div>
    </summary>
    <updated>2026-01-19T07:00:00Z</updated>
    <published>2026-01-19T07:00:00Z</published>
    <category term="eglot"/>
    <category term="emacs"/>
    <category term="lsp-mode"/>
    <source>
      <id>https://magnus.therning.org/</id>
      <author>
        <name>Magnus Therning</name>
      </author>
      <link href="https://magnus.therning.org/" rel="alternate" type="text/html"/>
      <link href="https://magnus.therning.org/feed.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Magnus web site</subtitle>
      <title>Magnus web site</title>
      <updated>2026-02-17T23:09:47Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://abhinavsarkar.net/posts/implementing-co-5/</id>
    <link href="https://abhinavsarkar.net/posts/implementing-co-5/?mtm_campaign=feed#syndications" rel="replies" type="text/html"/>
    <link href="https://abhinavsarkar.net/posts/implementing-co-5/?mtm_campaign=feed" rel="alternate" type="text/html"/>
    <title>Implementing Co, a Small Language With Coroutines #5: Adding Sleep</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>In the <a href="https://abhinavsarkar.net/posts/implementing-co-4/?mtm_campaign=feed">previous post</a>, we added channels to <span class="fancy">Co</span>, the small language we are implementing in this series of posts. In this post, we add the <code>sleep</code> primitive to it, enabling time-based coroutine scheduling. We then use sleep to build a simulation of digital logic circuits.</p>
<p>This post was originally published on <a href="https://abhinavsarkar.net/posts/implementing-co-5/?mtm_campaign=feed">abhinavsarkar.net</a>.</p><section class="series-info">
  <p>This post is a part of the series: <strong>Implementing Co, a Small Language With Coroutines</strong>.</p>
  <ol>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-1/?mtm_campaign=feed">The Parser</a> 
    </li>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-2/?mtm_campaign=feed">The Interpreter</a> 
    </li>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-3/?mtm_campaign=feed">Adding Coroutines</a> 
    </li>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-4/?mtm_campaign=feed">Adding Channels</a> 
    </li>
    <li>
       <strong>Adding Sleep</strong> ğŸ‘ˆ
      
    </li>
  </ol>
</section>

<nav id="toc"><h3>Contents</h3><ol><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#introduction">Introduction</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#adding-sleep">Adding Sleep</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#delayed-coroutines">Delayed Coroutines</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#queuing-coroutines">Queuing Coroutines</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#implementing-sleep">Implementing Sleep</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#sleep-in-action">Sleep in Action</a><ol><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#sleep-sort">Sleep Sort</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#javascript-like-timeouts-and-intervals">JavaScript-like Timeouts and Intervals</a></li></ol></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#bonus-round-digital-circuit-simulation">Bonus Round: Digital Circuit Simulation</a><ol><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#conjuring-lists">Conjuring Lists</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#wires">Wires</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#logic-gates">Logic Gates</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#adders">Adders</a></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#ripple-carry-adder">Ripple-carry Adder</a></li></ol></li><li><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#conclusion">Conclusion</a></li></ol></nav>
<h2 id="introduction">Introduction<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#introduction">#</a></h2>
<p><a href="https://en.wikipedia.org/wiki/Sleep_(system_call)" rel="noopener" target="_blank">Sleep</a> is a commonly used operation in concurrent programs. It pauses the execution of the current <em>Thread of Computation</em> (ToC) for a specified duration, after which the <abbr title="Thread of computation">ToC</abbr> is resumed automatically. Sleep is used for various purposes: polling for events, delaying execution of an operation, simulating latency, implementing timeouts, and more.</p>
<p>Sleep is generally implemented as a primitive operation in most languages, delegating the actual implementation to the underlying operating system. The operating systemâ€™s scheduler removes the <abbr title="Thread of computation">ToC</abbr> from the list of runnable <abbr title="Threads of computation">ToCs</abbr>, places it in a list of sleeping <abbr title="Threads of computation">ToCs</abbr>, and after the specified duration, moves it back to the list of runnable <abbr title="Threads of computation">ToCs</abbr> for scheduling.</p>
<p>Since <span class="fancy">Co</span> implements its own <abbr title="Thread of computation">ToC</abbr> (coroutine) scheduler, we implement sleep as a primitive operation within the interpreter itself<a class="footnote-ref" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fn1" id="fnref1"><sup>1</sup></a>.</p>
<h2 id="adding-sleep">Adding Sleep<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#adding-sleep">#</a></h2>
<p>We start by exposing <code>sleep</code> and <code>getCurrentMillis</code> as built-in functions to <span class="fancy">Co</span>:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-1" tabindex="-1"/><span class="ot">builtinEnv ::</span> <span class="dt">IO</span> <span class="dt">Env</span></span>
<span id="cb1-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-2" tabindex="-1"/>builtinEnv <span class="ot">=</span> Map.fromList <span class="op">&lt;$&gt;</span> <span class="fu">traverse</span> (<span class="fu">traverse</span> newIORef) [</span>
<span id="cb1-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-3" tabindex="-1"/>    (<span class="st">"print"</span>, <span class="dt">BuiltinFunction</span> <span class="st">"print"</span> <span class="dv">1</span> executePrint)</span>
<span id="cb1-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-4" tabindex="-1"/>  , (<span class="st">"newChannel"</span>,</span>
<span id="cb1-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-5" tabindex="-1"/>     <span class="dt">BuiltinFunction</span> <span class="st">"newChannel"</span> <span class="dv">0</span> <span class="op">$</span> <span class="fu">fmap</span> <span class="dt">Chan</span> <span class="op">.</span> <span class="fu">const</span> (newChannel <span class="dv">0</span>))</span>
<span id="cb1-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-6" tabindex="-1"/>  , (<span class="st">"newBufferedChannel"</span>,</span>
<span id="cb1-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-7" tabindex="-1"/><span class="emphasis">     <span class="dt">BuiltinFunction</span> <span class="st">"newBufferedChannel"</span> <span class="dv">1</span> executeNewBufferedChannel)</span></span>
<span id="cb1-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-8" tabindex="-1"/><span class="emphasis">  , (<span class="st">"sleep"</span>, <span class="dt">BuiltinFunction</span> <span class="st">"sleep"</span> <span class="dv">1</span> executeSleep)</span></span>
<span id="cb1-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-9" tabindex="-1"/><span class="emphasis">  , (<span class="st">"getCurrentMillis"</span>,</span></span>
<span id="cb1-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-10" tabindex="-1"/>     <span class="dt">BuiltinFunction</span> <span class="st">"getCurrentMillis"</span> <span class="dv">0</span> executeGetCurrentMillis)</span>
<span id="cb1-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-11" tabindex="-1"/>  ]</span></code></pre></div>
<p>The <code>sleep</code> built-in function takes one argumentâ€”the duration in milliseconds to sleep for. The <code>getCurrentMillis</code> function returns the current time in milliseconds since the <a href="https://en.wikipedia.org/wiki/Unix_epoch" rel="noopener" target="_blank">Unix epoch</a>. Both of them delegate to the functions explained next.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-1" tabindex="-1"/><span class="ot">executeSleep ::</span> [<span class="dt">Expr</span>] <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> <span class="dt">Value</span></span>
<span id="cb1-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-2" tabindex="-1"/>executeSleep argEs <span class="ot">=</span> evaluate (<span class="fu">head</span> argEs) <span class="op">&gt;&gt;=</span> \<span class="kw">case</span></span>
<span id="cb1-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-3" tabindex="-1"/>  <span class="dt">Num</span> n <span class="op">|</span> n <span class="op">&gt;=</span> <span class="dv">0</span> <span class="ot">-&gt;</span> sleep n <span class="op">&gt;&gt;</span> <span class="fu">return</span> <span class="dt">Null</span></span>
<span id="cb1-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-4" tabindex="-1"/>  <span class="dt">Num</span> n <span class="ot">-&gt;</span> throw <span class="op">$</span> <span class="st">"Sleep time must be non-negative: "</span> <span class="op">&lt;&gt;</span> <span class="fu">show</span> n</span>
<span id="cb1-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-5" tabindex="-1"/>  _ <span class="ot">-&gt;</span> throw <span class="st">"sleep call expected a number argument"</span></span>
<span id="cb1-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-6" tabindex="-1"/></span>
<span id="cb1-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-7" tabindex="-1"/><span class="ot">executeGetCurrentMillis ::</span> [<span class="dt">Expr</span>] <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> <span class="dt">Value</span></span>
<span id="cb1-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-8" tabindex="-1"/>executeGetCurrentMillis _ <span class="ot">=</span></span>
<span id="cb1-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-9" tabindex="-1"/>  <span class="dt">Num</span> <span class="op">.</span> <span class="fu">fromIntegral</span> <span class="op">.</span> <span class="fu">floor</span> <span class="op">.</span> (<span class="op">*</span> <span class="dv">1000</span>) <span class="op">&lt;$&gt;</span> liftIO getPOSIXTime</span></code></pre></div>
<p>The <code>executeSleep</code> function evaluates its argument to a number, checks that it is non-negative, and then calls the <code>sleep</code> function in the <code class="sourceCode haskell"><span class="dt">Interpreter</span></code> monad. <code>executeGetCurrentMillis</code> calls <a href="https://hackage.haskell.org/package/time/docs/Data-Time-Clock-POSIX.html#v:getPOSIXTime" rel="noopener" target="_blank"><code>getPOSIXTime</code></a> and returns the milliseconds wrapped as a <code class="sourceCode haskell"><span class="dt">Num</span></code>.</p>
<p>The implementation of sleep is more involved than other built-in functions because it interacts with the coroutine scheduler. When a coroutine calls <code>sleep</code>, we want to suspend the coroutine, and schedule it to be resumed after the specified duration. There may be multiple coroutines in the sleep state at a time, and they must be resumed according to their wakeup time (time at which sleep was called + sleep duration), and not in any other order. To be efficient, it is also important that the scheduler does not poll repeatedly for new coroutines to wake up and run, but instead waits till the right time. These are the two requirements for our coroutine scheduler. And the solution is: delayed coroutines.</p>
<h2 id="delayed-coroutines">Delayed Coroutines<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#delayed-coroutines">#</a></h2>
<p>The <a href="https://abhinavsarkar.net/posts/implementing-co-3/?mtm_campaign=feed#from-continuations-to-coroutines">coroutines</a> we have implemented so far were scheduled to run immediately. To implement sleep, we extend the coroutine concept with <em>Delayed Coroutines</em>â€”coroutines that are scheduled to run at a specific future time.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-1" tabindex="-1"/><span class="kw">data</span> <span class="dt">Coroutine</span> a <span class="ot">=</span> <span class="dt">Coroutine</span></span>
<span id="cb1-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-2" tabindex="-1"/>  {<span class="ot"> corEnv ::</span> <span class="dt">Env</span></span>
<span id="cb1-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-3" tabindex="-1"/>  ,<span class="ot"> corCont ::</span> a <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> ()</span>
<span id="cb1-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-4" tabindex="-1"/><span class="emphasis">  ,<span class="ot"> corReady ::</span> <span class="dt">MVar</span> ()</span></span>
<span id="cb1-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-5" tabindex="-1"/>  }</span>
<span id="cb1-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-6" tabindex="-1"/></span>
<span id="cb1-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-7" tabindex="-1"/><span class="ot">newCoroutine ::</span> <span class="dt">Env</span> <span class="ot">-&gt;</span> (a <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> ()) <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> (<span class="dt">Coroutine</span> a)</span>
<span id="cb1-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-8" tabindex="-1"/>newCoroutine env cont <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb1-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-9" tabindex="-1"/>  ready <span class="ot">&lt;-</span> newMVar ()</span>
<span id="cb1-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-10" tabindex="-1"/>  <span class="fu">return</span> <span class="op">$</span> <span class="dt">Coroutine</span> env cont ready</span></code></pre></div>
<p>Now the <code class="sourceCode haskell"><span class="dt">Coroutine</span></code> data type holds an <a href="https://hackage.haskell.org/package/base/docs/Control-Concurrent-MVar.html#t:MVar" rel="noopener" target="_blank"><code class="sourceCode haskell"><span class="dt">MVar</span></code></a> to signal when the coroutine is ready to be run. The old-style coroutines that run immediately are created ready to run by the <code>newCoroutine</code> function. But delayed coroutines are different:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb2-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-1" tabindex="-1"/><span class="ot">newDelayedCoroutine ::</span></span>
<span id="cb2-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-2" tabindex="-1"/>  <span class="dt">Integer</span> <span class="ot">-&gt;</span> <span class="dt">Env</span> <span class="ot">-&gt;</span> (a <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> ()) <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> (<span class="dt">Coroutine</span> a)</span>
<span id="cb2-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-3" tabindex="-1"/>newDelayedCoroutine sleepMillis env cont <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb2-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-4" tabindex="-1"/>  ready <span class="ot">&lt;-</span> newEmptyMVar</span>
<span id="cb2-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-5" tabindex="-1"/>  void <span class="op">$</span> liftIO <span class="op">$</span> forkIO <span class="op">$</span> <span class="kw">do</span></span>
<span id="cb2-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-6" tabindex="-1"/>    threadDelay <span class="op">$</span> <span class="fu">fromIntegral</span> sleepMillis <span class="op">*</span> <span class="dv">1000</span></span>
<span id="cb2-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-7" tabindex="-1"/>    putMVar ready ()</span>
<span id="cb2-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb2-8" tabindex="-1"/>  <span class="fu">return</span> <span class="op">$</span> <span class="dt">Coroutine</span> env cont ready</span></code></pre></div>
<p>The key difference from a regular coroutine is that the <code class="sourceCode haskell"><span class="dt">MVar</span></code> used for signaling is created empty. We fork a thread<a class="footnote-ref" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fn2" id="fnref2"><sup>2</sup></a> that sleeps<a class="footnote-ref" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fn3" id="fnref3"><sup>3</sup></a> for the specified sleep duration, and then signals that the coroutine is ready to run by filling the <code class="sourceCode haskell"><span class="dt">MVar</span></code>.</p>
<p>An <code class="sourceCode haskell"><span class="dt">MVar</span></code> is a synchronization primitive<a class="footnote-ref" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fn4" id="fnref4"><sup>4</sup></a>â€”essentially a mutable box that can hold a value or be empty. When we call <code>takeMVar</code> on an empty <code class="sourceCode haskell"><span class="dt">MVar</span></code>, it blocks until another thread fills it. This is what makes it powerful for our use case: instead of the interpreter repeatedly polling the queue asking â€œis this coroutine ready yet?â€�, we let the interpreter wait on the <code class="sourceCode haskell"><span class="dt">MVar</span></code>. The forked thread signals readiness at the right time by filling the <code class="sourceCode haskell"><span class="dt">MVar</span></code>. The interpreter wakes up immediatelyâ€”no wasted CPU cycles, no busy-waiting.</p>
<h2 id="queuing-coroutines">Queuing Coroutines<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#queuing-coroutines">#</a></h2>
<p>We already have a <code class="sourceCode haskell"><span class="dt">Queue</span></code> of coroutines in our <code class="sourceCode haskell"><span class="dt">Interpreter</span></code>. It is a <a href="https://en.wikipedia.org/wiki/min-priority_queue" rel="noopener" target="_blank">min-priority queue</a> sorted by timestamps, which we have been using as a <a href="https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)" rel="noopener" target="_blank">FIFO</a> queue till now. Now we use it for its real purpose: storing delayed coroutines sorted by their wakeup times.</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb3-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-1" tabindex="-1"/><span class="kw">type</span> <span class="dt">Queue</span> a <span class="ot">=</span> <span class="dt">IORef</span> (<span class="dt">PQ.MinPQueue</span> <span class="dt">TimeSpec</span> a, <span class="dt">TimeSpec</span>)</span>
<span id="cb3-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-2" tabindex="-1"/></span>
<span id="cb3-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-3" tabindex="-1"/><span class="ot">newQueue ::</span> <span class="dt">MonadBase</span> <span class="dt">IO</span> m <span class="ot">=&gt;</span> m (<span class="dt">Queue</span> a)</span>
<span id="cb3-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-4" tabindex="-1"/>newQueue <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb3-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-5" tabindex="-1"/>  now <span class="ot">&lt;-</span> liftBase currentSystemTime</span>
<span id="cb3-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-6" tabindex="-1"/>  newIORef (PQ.empty, now)</span>
<span id="cb3-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-7" tabindex="-1"/></span>
<span id="cb3-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-8" tabindex="-1"/><span class="ot">queueSize ::</span> <span class="dt">MonadBase</span> <span class="dt">IO</span> m <span class="ot">=&gt;</span> <span class="dt">Queue</span> a <span class="ot">-&gt;</span> m <span class="dt">Int</span></span>
<span id="cb3-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb3-9" tabindex="-1"/>queueSize <span class="ot">=</span> <span class="fu">fmap</span> (PQ.size <span class="op">.</span> <span class="fu">fst</span>) <span class="op">.</span> readIORef</span></code></pre></div>
<p>The queue also tracks the maximum wakeup time of all coroutines in the queue. This information is useful for calculating how long the interpreter should sleep before termination.</p>
<p>The core operations on the queue are:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb4-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-1" tabindex="-1"/><span class="ot">enqueueAt ::</span> <span class="dt">TimeSpec</span> <span class="ot">-&gt;</span> a <span class="ot">-&gt;</span> <span class="dt">Queue</span> a <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> ()</span>
<span id="cb4-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-2" tabindex="-1"/>enqueueAt time val queue <span class="ot">=</span> atomicModifyIORef' queue <span class="op">$</span> \(q, maxWakeupTime) <span class="ot">-&gt;</span></span>
<span id="cb4-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-3" tabindex="-1"/>  (( PQ.insert time val q,</span>
<span id="cb4-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-4" tabindex="-1"/>     <span class="kw">if</span> time <span class="op">&gt;</span> maxWakeupTime <span class="kw">then</span> time <span class="kw">else</span> maxWakeupTime</span>
<span id="cb4-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-5" tabindex="-1"/>   ), ())</span>
<span id="cb4-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-6" tabindex="-1"/></span>
<span id="cb4-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-7" tabindex="-1"/><span class="ot">enqueue ::</span> a <span class="ot">-&gt;</span> <span class="dt">Queue</span> a <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> ()</span>
<span id="cb4-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-8" tabindex="-1"/>enqueue val queue <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb4-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-9" tabindex="-1"/>  now <span class="ot">&lt;-</span> currentSystemTime</span>
<span id="cb4-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-10" tabindex="-1"/>  enqueueAt now val queue</span>
<span id="cb4-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-11" tabindex="-1"/></span>
<span id="cb4-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-12" tabindex="-1"/><span class="ot">dequeue ::</span> <span class="dt">Queue</span> a <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> (<span class="dt">Maybe</span> a)</span>
<span id="cb4-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-13" tabindex="-1"/>dequeue queue <span class="ot">=</span> atomicModifyIORef' queue <span class="op">$</span> \(q, maxWakeupTime) <span class="ot">-&gt;</span></span>
<span id="cb4-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-14" tabindex="-1"/>  <span class="kw">if</span> PQ.null q</span>
<span id="cb4-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-15" tabindex="-1"/>    <span class="kw">then</span> ((q, maxWakeupTime), <span class="dt">Nothing</span>)</span>
<span id="cb4-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-16" tabindex="-1"/>    <span class="kw">else</span> <span class="kw">let</span> ((_, val), q') <span class="ot">=</span> PQ.deleteFindMin q</span>
<span id="cb4-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-17" tabindex="-1"/>         <span class="kw">in</span> ((q', maxWakeupTime), <span class="dt">Just</span> val)</span>
<span id="cb4-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-18" tabindex="-1"/></span>
<span id="cb4-19"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-19" tabindex="-1"/><span class="ot">currentSystemTime ::</span> <span class="dt">MonadIO</span> m <span class="ot">=&gt;</span> m <span class="dt">TimeSpec</span></span>
<span id="cb4-20"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb4-20" tabindex="-1"/>currentSystemTime <span class="ot">=</span> liftIO <span class="op">$</span> getTime <span class="dt">Monotonic</span></span></code></pre></div>
<p>We saw the <code>enqueue</code> function <a href="https://abhinavsarkar.net/posts/implementing-co-3/?mtm_campaign=feed#scheduling-coroutines">earlier</a>:</p>
<blockquote>
<p>The <code>enqueueAt</code> function enqueues the given value at the given time in the queue. The <code>enqueue</code> function enqueues the value at the current time, thus scheduling it to run immediately.</p>
<p>The <code>dequeue</code> function dequeues the value with the lowest priority from the queue, which in this case, is the value that is enqueued first.</p>
<p>The <code>currentSystemTime</code> function returns the monotonically increasing current system time.</p>
</blockquote>
<p>The <code>dequeue</code> function dequeues the coroutine with lowest priority, so if we use the wakeup time as priority, it will dequeue the coroutine that is to be run next. That works!</p>
<p>The <code>enqueueAt</code> function calculates and tracks the maximum wakeup times of the coroutines as well. Next, we implement the scheduling of delayed coroutines:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb5-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb5-1" tabindex="-1"/><span class="ot">scheduleDelayedCoroutine ::</span> <span class="dt">TimeSpec</span> <span class="ot">-&gt;</span> <span class="dt">Coroutine</span> () <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> ()</span>
<span id="cb5-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb5-2" tabindex="-1"/>scheduleDelayedCoroutine wakeupTime coroutine <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb5-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb5-3" tabindex="-1"/>  State.gets isCoroutines <span class="op">&gt;&gt;=</span> enqueueAt wakeupTime coroutine</span></code></pre></div>
<p>The <code>scheduleDelayedCoroutine</code> function enqueues a coroutine in the interpreter coroutine queue with the specified wakeup time. We also improve the <code>runNextCoroutine</code> function to wait for the coroutine to be ready before running it.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-1" tabindex="-1"/><span class="ot">runNextCoroutine ::</span> <span class="dt">Interpreter</span> ()</span>
<span id="cb1-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-2" tabindex="-1"/>runNextCoroutine <span class="ot">=</span></span>
<span id="cb1-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-3" tabindex="-1"/>  State.gets isCoroutines <span class="op">&gt;&gt;=</span> dequeue <span class="op">&gt;&gt;=</span> \<span class="kw">case</span></span>
<span id="cb1-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-4" tabindex="-1"/>    <span class="dt">Nothing</span> <span class="ot">-&gt;</span> throwError <span class="dt">CoroutineQueueEmpty</span></span>
<span id="cb1-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-5" tabindex="-1"/>    <span class="dt">Just</span> <span class="dt">Coroutine</span> {<span class="op">..</span>} <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb1-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-6" tabindex="-1"/><span class="emphasis">      void <span class="op">$</span> takeMVar corReady</span></span>
<span id="cb1-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-7" tabindex="-1"/>      setEnv corEnv</span>
<span id="cb1-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb1-8" tabindex="-1"/>      corCont ()</span></code></pre></div>
<p>The <code>takeMVar</code> function call blocks till the thread that was forked when creating the coroutine wakes up and fills the <code>corReady</code> <code class="sourceCode haskell"><span class="dt">MVar</span></code>. So we donâ€™t have to poll the queue.</p>
<p>Thatâ€™s all we have to do for having delayed coroutines.</p>
<h2 id="implementing-sleep">Implementing Sleep<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#implementing-sleep">#</a></h2>
<p>With the infrastructure in place, the <code>sleep</code> function becomes straightforward:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb6-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-1" tabindex="-1"/><span class="ot">sleep ::</span> <span class="dt">Integer</span> <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> ()</span>
<span id="cb6-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-2" tabindex="-1"/>sleep millis <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb6-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-3" tabindex="-1"/>  now <span class="ot">&lt;-</span> currentSystemTime</span>
<span id="cb6-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-4" tabindex="-1"/>  <span class="kw">let</span> wakeupTime <span class="ot">=</span> now <span class="op">+</span> fromNanoSecs (<span class="fu">fromIntegral</span> millis <span class="op">*</span> <span class="dv">1000000</span>)</span>
<span id="cb6-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-5" tabindex="-1"/>  env <span class="ot">&lt;-</span> State.gets isEnv</span>
<span id="cb6-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-6" tabindex="-1"/>  callCC <span class="op">$</span> \cont <span class="ot">-&gt;</span> <span class="kw">do</span></span>
<span id="cb6-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-7" tabindex="-1"/>    coroutine <span class="ot">&lt;-</span> newDelayedCoroutine millis env cont</span>
<span id="cb6-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-8" tabindex="-1"/>    scheduleDelayedCoroutine wakeupTime coroutine</span>
<span id="cb6-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb6-9" tabindex="-1"/>    runNextCoroutine</span></code></pre></div>
<p>When a coroutine calls <code>sleep</code>, we capture the current environment and use <code>callCC</code> to capture the continuationâ€”the code that should run after the sleep completes. We then create a new delayed coroutine with this continuation, schedule it for the future, and run the next coroutine in the queue. The scheduler machinery takes care of running the delayed coroutine at the right time.</p>
<p>We also modify the <code>awaitTermination</code> function from the previous post to handle delayed coroutines. It now sleeps till the last wakeup time before checking if the queue is empty:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb7-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb7-1" tabindex="-1"/><span class="ot">awaitTermination ::</span> <span class="dt">Interpreter</span> ()</span>
<span id="cb7-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb7-2" tabindex="-1"/>awaitTermination <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb7-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb7-3" tabindex="-1"/>  (coroutines, maxWakeupTime) <span class="ot">&lt;-</span> readIORef <span class="op">=&lt;&lt;</span> State.gets isCoroutines</span>
<span id="cb7-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb7-4" tabindex="-1"/>  dur <span class="ot">&lt;-</span> calcSleepDuration maxWakeupTime</span>
<span id="cb7-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb7-5" tabindex="-1"/>  unless (PQ.null coroutines) <span class="op">$</span> <span class="kw">if</span> dur <span class="op">&gt;</span> <span class="dv">0</span></span>
<span id="cb7-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb7-6" tabindex="-1"/>    <span class="kw">then</span> sleep dur <span class="op">&gt;&gt;</span> awaitTermination</span>
<span id="cb7-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb7-7" tabindex="-1"/>    <span class="kw">else</span> yield <span class="op">&gt;&gt;</span> awaitTermination</span></code></pre></div>
<p>Notice how we use the <code>sleep</code> function we just defined in <code>awaitTermination</code>. The <code>calcSleepDuration</code> function calculates how long to sleep before the last coroutine becomes ready:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode numberSource haskell"><code class="sourceCode haskell"><span id="cb8-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb8-1" tabindex="-1"/><span class="ot">calcSleepDuration ::</span> <span class="dt">TimeSpec</span> <span class="ot">-&gt;</span> <span class="dt">Interpreter</span> <span class="dt">Integer</span></span>
<span id="cb8-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb8-2" tabindex="-1"/>calcSleepDuration maxWakeupTime <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb8-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb8-3" tabindex="-1"/>  now <span class="ot">&lt;-</span> currentSystemTime</span>
<span id="cb8-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb8-4" tabindex="-1"/>  <span class="fu">return</span> <span class="op">$</span> <span class="dv">1</span> <span class="op">+</span> <span class="fu">fromIntegral</span> (maxWakeupTime <span class="op">-</span> now) <span class="ot">`div`</span> <span class="dv">1000000</span></span></code></pre></div>
<p>Thatâ€™s all for sleeping. This may be too much to take in, so letâ€™s go through some examples.</p>
<h2 id="sleep-in-action">Sleep in Action<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#sleep-in-action">#</a></h2>
<p>Sleep can be used for polling/waiting for events, delaying execution, simulating latency, implementing timeouts, and more. Letâ€™s see some simple uses.</p>
<h3 id="sleep-sort">Sleep Sort<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#sleep-sort">#</a></h3>
<p>An interesting example of sleep is the infamous <a href="https://www.geeksforgeeks.org/dsa/sleep-sort-king-laziness-sorting-sleeping/" rel="noopener" target="_blank">sleep sort</a>, which sorts a list of numbers by spawning a coroutine for each number that sleeps for the duration of that number, then prints it:</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb9-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">sleepSort</span>(a<span class="op">,</span> b<span class="op">,</span> c<span class="op">,</span> d<span class="op">,</span> e) {</span>
<span id="cb9-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-2" tabindex="-1"/>  <span class="kw">function</span> <span class="fu">printNum</span>(num) {</span>
<span id="cb9-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-3" tabindex="-1"/>    <span class="fu">sleep</span>(num)<span class="op">;</span></span>
<span id="cb9-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-4" tabindex="-1"/>    <span class="fu">print</span>(num)<span class="op">;</span></span>
<span id="cb9-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-5" tabindex="-1"/>  }</span>
<span id="cb9-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-6" tabindex="-1"/>  spawn <span class="fu">printNum</span>(a)<span class="op">;</span></span>
<span id="cb9-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-7" tabindex="-1"/>  spawn <span class="fu">printNum</span>(b)<span class="op">;</span></span>
<span id="cb9-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-8" tabindex="-1"/>  spawn <span class="fu">printNum</span>(c)<span class="op">;</span></span>
<span id="cb9-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-9" tabindex="-1"/>  spawn <span class="fu">printNum</span>(d)<span class="op">;</span></span>
<span id="cb9-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-10" tabindex="-1"/>  spawn <span class="fu">printNum</span>(e)<span class="op">;</span></span>
<span id="cb9-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-11" tabindex="-1"/>}</span>
<span id="cb9-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-12" tabindex="-1"/></span>
<span id="cb9-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb9-13" tabindex="-1"/><span class="fu">sleepSort</span>(<span class="dv">5</span><span class="op">,</span> <span class="dv">4</span><span class="op">,</span> <span class="dv">3</span><span class="op">,</span> <span class="dv">2</span><span class="op">,</span> <span class="dv">1</span>)<span class="op">;</span></span></code></pre></div>
<p>Running this program prints what we expect:</p>
<pre class="plain"><code>1
2
3
4
5</code></pre>
<p>Donâ€™t use <code>sleepSort</code> for sorting your numbers though. Moving on.</p>
<h3 id="javascript-like-timeouts-and-intervals">JavaScript-like Timeouts and Intervals<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#javascript-like-timeouts-and-intervals">#</a></h3>
<p>With sleep, we can implement JavaScript-like <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/setTimeout" rel="noopener" target="_blank"><code>setTimeout</code></a> and <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/setInterval" rel="noopener" target="_blank"><code>setInterval</code></a> functions:</p>
<div class="sourceCode" id="cb11"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb11-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">setTimeout</span>(callback<span class="op">,</span> millis) {</span>
<span id="cb11-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-2" tabindex="-1"/>  <span class="fu">spawn</span> (<span class="kw">function</span> () { <span class="fu">sleep</span>(millis)<span class="op">;</span> <span class="fu">callback</span>()<span class="op">;</span> })()<span class="op">;</span></span>
<span id="cb11-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-3" tabindex="-1"/>}</span>
<span id="cb11-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-4" tabindex="-1"/></span>
<span id="cb11-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-5" tabindex="-1"/><span class="kw">function</span> <span class="fu">hello</span>() {</span>
<span id="cb11-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-6" tabindex="-1"/>  <span class="fu">print</span>(<span class="st">"hello"</span>)<span class="op">;</span></span>
<span id="cb11-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-7" tabindex="-1"/>}</span>
<span id="cb11-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-8" tabindex="-1"/></span>
<span id="cb11-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb11-9" tabindex="-1"/><span class="pp">setTimeout</span>(hello<span class="op">,</span> <span class="dv">10000</span>)<span class="op">;</span></span></code></pre></div>
<p>The <code>setTimeout</code> function spawns a coroutine that sleeps for the specified duration and then calls the callback function.</p>
<div class="sourceCode" id="cb12"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb12-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">setInterval</span>(callback<span class="op">,</span> seconds) {</span>
<span id="cb12-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-2" tabindex="-1"/>  <span class="kw">function</span> <span class="fu">run</span>() {</span>
<span id="cb12-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-3" tabindex="-1"/>    <span class="fu">callback</span>()<span class="op">;</span></span>
<span id="cb12-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-4" tabindex="-1"/>    <span class="pp">setTimeout</span>(run<span class="op">,</span> seconds)<span class="op">;</span></span>
<span id="cb12-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-5" tabindex="-1"/>  }</span>
<span id="cb12-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-6" tabindex="-1"/>  <span class="pp">setTimeout</span>(run<span class="op">,</span> seconds)<span class="op">;</span></span>
<span id="cb12-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-7" tabindex="-1"/>}</span>
<span id="cb12-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-8" tabindex="-1"/></span>
<span id="cb12-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-9" tabindex="-1"/><span class="kw">var</span> start <span class="op">=</span> <span class="fu">getCurrentMillis</span>()<span class="op">;</span></span>
<span id="cb12-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-10" tabindex="-1"/><span class="kw">function</span> <span class="fu">tick</span>(t) {</span>
<span id="cb12-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-11" tabindex="-1"/>  <span class="kw">var</span> now <span class="op">=</span> <span class="fu">getCurrentMillis</span>()<span class="op">;</span></span>
<span id="cb12-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-12" tabindex="-1"/>  <span class="fu">print</span>(t <span class="op">+</span> <span class="st">" "</span> <span class="op">+</span> (now <span class="op">-</span> start))<span class="op">;</span></span>
<span id="cb12-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-13" tabindex="-1"/>}</span>
<span id="cb12-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-14" tabindex="-1"/></span>
<span id="cb12-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-15" tabindex="-1"/><span class="pp">setInterval</span>(<span class="kw">function</span> () { <span class="fu">tick</span>(<span class="st">"tick"</span>)<span class="op">;</span> }<span class="op">,</span> <span class="dv">2000</span>)<span class="op">;</span></span>
<span id="cb12-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-16" tabindex="-1"/><span class="fu">sleep</span>(<span class="dv">1000</span>)<span class="op">;</span></span>
<span id="cb12-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb12-17" tabindex="-1"/><span class="pp">setInterval</span>(<span class="kw">function</span> () { <span class="fu">tick</span>(<span class="st">"tock"</span>)<span class="op">;</span> }<span class="op">,</span> <span class="dv">2000</span>)<span class="op">;</span></span></code></pre></div>
<p>The <code>setInterval</code> function repeatedly calls a callback at a fixed interval using <code>setTimeout</code> to reschedule itself. Running the above code prints alternating <code>tick</code> and <code>tock</code> every 1 second, forever:</p>
<pre class="plain"><code>tick 2005
tock 3008
tick 4009
tock 5009
tick 6015
tock 7013
tick 8017
tock 9019
tick 10020
tock 11021
tick 12021</code></pre>
<p>Notice that the scheduling is not accurate up to milliseconds, but only approximate.</p>
<h2 id="bonus-round-digital-circuit-simulation">Bonus Round: Digital Circuit Simulation<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#bonus-round-digital-circuit-simulation">#</a></h2>
<p>As a more complex example of using sleep, we implement a simulator for digital logic circuits, from basic <em><a href="https://en.wikipedia.org/wiki/Logic_gates" rel="noopener" target="_blank">Logic gates</a></em> to a <em><a href="https://en.wikipedia.org/wiki/Ripple_carry_adder" rel="noopener" target="_blank">Ripple carry adder</a></em>. The idea is to model circuits as a network of wires and gates, where the wires carry digital signal values (<code>0</code> or <code>1</code>), and the logic gates transform input signals to output signals with a propagation delay.</p>
<p>The digital circuit simulation example is from the <a href="https://mitp-content-server.mit.edu/books/content/sectbyfn/books_pres_0/6515/sicp.zip/full-text/book/book-Z-H-22.html#%_sec_3.3.4" rel="noopener" target="_blank">Wizard Book</a>. Quoting an example:</p>
<blockquote>
<p>An inverter is a primitive function box [logic gate] that inverts its input. If the input signal to an inverter changes to 0, then one inverter-delay later the inverter will change its output signal to 1. If the input signal to an inverter changes to 1, then one inverter-delay later the inverter will change its output signal to 0.</p>
</blockquote>
<p>But first, weâ€™ll need to make some lists.</p>
<h3 id="conjuring-lists">Conjuring Lists<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#conjuring-lists">#</a></h3>
<p>We implement a simple cons list (a singly linked list) using <a href="https://web.archive.org/web/20260116/https://mitp-content-server.mit.edu/books/content/sectbyfn/books_pres_0/6515/sicp.zip/full-text/book/book-Z-H-14.html#%25_sec_2.1.3" rel="noopener" target="_blank">a trick from the book itself</a>:</p>
<div class="sourceCode" id="cb14"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb14-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">Cons</span>(first<span class="op">,</span> rest) {</span>
<span id="cb14-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-2" tabindex="-1"/>  <span class="cf">return</span> <span class="kw">function</span> (command) {</span>
<span id="cb14-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-3" tabindex="-1"/>    <span class="cf">if</span> (command <span class="op">==</span> <span class="st">"first"</span>) { <span class="cf">return</span> first<span class="op">;</span> }</span>
<span id="cb14-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-4" tabindex="-1"/>    <span class="cf">if</span> (command <span class="op">==</span> <span class="st">"rest"</span>) { <span class="cf">return</span> rest<span class="op">;</span> }</span>
<span id="cb14-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-5" tabindex="-1"/>  }<span class="op">;</span></span>
<span id="cb14-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-6" tabindex="-1"/>}</span>
<span id="cb14-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-7" tabindex="-1"/></span>
<span id="cb14-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-8" tabindex="-1"/><span class="kw">function</span> <span class="fu">Empty</span>() { <span class="cf">return</span> <span class="kw">null</span><span class="op">;</span>}</span>
<span id="cb14-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-9" tabindex="-1"/></span>
<span id="cb14-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-10" tabindex="-1"/><span class="kw">function</span> <span class="fu">first</span>(list) { <span class="cf">return</span> <span class="fu">list</span>(<span class="st">"first"</span>)<span class="op">;</span> }</span>
<span id="cb14-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-11" tabindex="-1"/><span class="kw">function</span> <span class="fu">rest</span>(list) { <span class="cf">return</span> <span class="fu">list</span>(<span class="st">"rest"</span>)<span class="op">;</span> }</span>
<span id="cb14-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb14-12" tabindex="-1"/><span class="kw">function</span> <span class="fu">prepend</span>(list<span class="op">,</span> element) { <span class="cf">return</span> <span class="fu">Cons</span>(element<span class="op">,</span> list)<span class="op">;</span> }</span></code></pre></div>
<p><code class="co">Empty</code> creates an empty list, and we grow the list by prepending an element to it by calling the <code>prepend</code> function. <code>first</code> returns the first element of a list, and <code>rest</code> returns the rest of them. Notice that a <code>Cons</code> cell is just a closure that holds references to its first and rest parameters, and returns a selector function to retrieve them.</p>
<p>Next, we define a helper function to call a list of actions, yielding after each one:</p>
<div class="sourceCode" id="cb15"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb15-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb15-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">callEach</span>(actions) {</span>
<span id="cb15-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb15-2" tabindex="-1"/>  <span class="cf">if</span> (actions <span class="op">!=</span> <span class="fu">Empty</span>()) {</span>
<span id="cb15-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb15-3" tabindex="-1"/>    <span class="fu">callEach</span>(<span class="fu">rest</span>(actions))<span class="op">;</span></span>
<span id="cb15-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb15-4" tabindex="-1"/>    <span class="fu">first</span>(actions)()<span class="op">;</span></span>
<span id="cb15-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb15-5" tabindex="-1"/>    <span class="kw">yield</span><span class="op">;</span></span>
<span id="cb15-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb15-6" tabindex="-1"/>  }</span>
<span id="cb15-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb15-7" tabindex="-1"/>}</span></code></pre></div>
<h3 id="wires">Wires<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#wires">#</a></h3>
<p>A wire holds a mutable signal value and a list of actions to call when the signal changes:</p>
<div class="sourceCode" id="cb16"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb16-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">Wire</span>() {</span>
<span id="cb16-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-2" tabindex="-1"/>  <span class="kw">var</span> signalValue <span class="op">=</span> <span class="dv">0</span><span class="op">;</span></span>
<span id="cb16-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-3" tabindex="-1"/>  <span class="kw">var</span> actions <span class="op">=</span> <span class="fu">Empty</span>()<span class="op">;</span></span>
<span id="cb16-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-4" tabindex="-1"/></span>
<span id="cb16-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-5" tabindex="-1"/>  <span class="cf">return</span> <span class="kw">function</span> (command) {</span>
<span id="cb16-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-6" tabindex="-1"/>    <span class="cf">if</span> (command <span class="op">==</span> <span class="st">"get-signal"</span>) {</span>
<span id="cb16-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-7" tabindex="-1"/>      <span class="cf">return</span> signalValue<span class="op">;</span></span>
<span id="cb16-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-8" tabindex="-1"/>    }</span>
<span id="cb16-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-9" tabindex="-1"/>    <span class="cf">if</span> (command <span class="op">==</span> <span class="st">"set-signal"</span>) {</span>
<span id="cb16-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-10" tabindex="-1"/>      <span class="cf">return</span> <span class="kw">function</span> (newValue) {</span>
<span id="cb16-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-11" tabindex="-1"/>        <span class="cf">if</span> (signalValue <span class="op">!=</span> newValue) {</span>
<span id="cb16-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-12" tabindex="-1"/>          signalValue <span class="op">=</span> newValue<span class="op">;</span></span>
<span id="cb16-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-13" tabindex="-1"/>          <span class="fu">callEach</span>(actions)<span class="op">;</span></span>
<span id="cb16-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-14" tabindex="-1"/>        }</span>
<span id="cb16-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-15" tabindex="-1"/>      }<span class="op">;</span></span>
<span id="cb16-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-16" tabindex="-1"/>    }</span>
<span id="cb16-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-17" tabindex="-1"/>    <span class="cf">if</span> (command <span class="op">==</span> <span class="st">"add-action"</span>) {</span>
<span id="cb16-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-18" tabindex="-1"/>      <span class="cf">return</span> <span class="kw">function</span> (action) {</span>
<span id="cb16-19"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-19" tabindex="-1"/>        actions <span class="op">=</span> <span class="fu">prepend</span>(actions<span class="op">,</span> action)<span class="op">;</span></span>
<span id="cb16-20"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-20" tabindex="-1"/>        <span class="fu">action</span>()<span class="op">;</span></span>
<span id="cb16-21"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-21" tabindex="-1"/>      }<span class="op">;</span></span>
<span id="cb16-22"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-22" tabindex="-1"/>    }</span>
<span id="cb16-23"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-23" tabindex="-1"/>    <span class="cf">return</span> <span class="kw">null</span><span class="op">;</span></span>
<span id="cb16-24"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-24" tabindex="-1"/>  }<span class="op">;</span></span>
<span id="cb16-25"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-25" tabindex="-1"/>}</span>
<span id="cb16-26"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-26" tabindex="-1"/></span>
<span id="cb16-27"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-27" tabindex="-1"/><span class="kw">function</span> <span class="fu">getSignal</span>(wire) { <span class="cf">return</span> <span class="fu">wire</span>(<span class="st">"get-signal"</span>)<span class="op">;</span> }</span>
<span id="cb16-28"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-28" tabindex="-1"/><span class="kw">function</span> <span class="fu">setSignal</span>(wire<span class="op">,</span> signal) { <span class="fu">wire</span>(<span class="st">"set-signal"</span>)(signal)<span class="op">;</span> }</span>
<span id="cb16-29"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-29" tabindex="-1"/><span class="kw">function</span> <span class="fu">addAction</span>(wire<span class="op">,</span> action) { <span class="fu">wire</span>(<span class="st">"add-action"</span>)(action)<span class="op">;</span> }</span>
<span id="cb16-30"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-30" tabindex="-1"/></span>
<span id="cb16-31"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-31" tabindex="-1"/><span class="kw">function</span> <span class="fu">connect</span>(a<span class="op">,</span> b) {</span>
<span id="cb16-32"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-32" tabindex="-1"/>  <span class="kw">function</span> <span class="fu">propagate</span>() {</span>
<span id="cb16-33"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-33" tabindex="-1"/>    <span class="fu">setSignal</span>(b<span class="op">,</span> <span class="fu">getSignal</span>(a))<span class="op">;</span></span>
<span id="cb16-34"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-34" tabindex="-1"/>  }</span>
<span id="cb16-35"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-35" tabindex="-1"/>  <span class="fu">addAction</span>(a<span class="op">,</span> propagate)<span class="op">;</span></span>
<span id="cb16-36"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb16-36" tabindex="-1"/>}</span></code></pre></div>
<p>A wire provides three operations:</p>
<ul>
<li><code>get-signal</code>: returns the current signal value.</li>
<li><code>set-signal</code>: sets a new signal value and calls all actions if the value changed.</li>
<li><code>add-action</code>: adds an action to be called when the signal changes, and calls it immediately.</li>
</ul>
<p>The <code>connect</code> function connects two wires, causing the signal from one to propagate to another.</p>
<h3 id="logic-gates">Logic Gates<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#logic-gates">#</a></h3>
<p>First, we define the basic logic operations:</p>
<div class="sourceCode" id="cb17"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb17-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">and</span>(a<span class="op">,</span> b) {</span>
<span id="cb17-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-2" tabindex="-1"/>  <span class="cf">if</span> (a <span class="op">==</span> <span class="dv">1</span>) { <span class="cf">if</span> (b <span class="op">==</span> <span class="dv">1</span>) { <span class="cf">return</span> <span class="dv">1</span><span class="op">;</span> } }</span>
<span id="cb17-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-3" tabindex="-1"/>  <span class="cf">return</span> <span class="dv">0</span><span class="op">;</span></span>
<span id="cb17-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-4" tabindex="-1"/>}</span>
<span id="cb17-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-5" tabindex="-1"/></span>
<span id="cb17-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-6" tabindex="-1"/><span class="kw">function</span> <span class="fu">or</span>(a<span class="op">,</span> b) {</span>
<span id="cb17-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-7" tabindex="-1"/>  <span class="cf">if</span> (a <span class="op">==</span> <span class="dv">1</span>) { <span class="cf">return</span> <span class="dv">1</span><span class="op">;</span> }</span>
<span id="cb17-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-8" tabindex="-1"/>  <span class="cf">if</span> (b <span class="op">==</span> <span class="dv">1</span>) { <span class="cf">return</span> <span class="dv">1</span><span class="op">;</span> }</span>
<span id="cb17-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-9" tabindex="-1"/>  <span class="cf">return</span> <span class="dv">0</span><span class="op">;</span></span>
<span id="cb17-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-10" tabindex="-1"/>}</span>
<span id="cb17-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-11" tabindex="-1"/></span>
<span id="cb17-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-12" tabindex="-1"/><span class="kw">function</span> <span class="fu">not</span>(a) {</span>
<span id="cb17-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-13" tabindex="-1"/>  <span class="cf">if</span> (a <span class="op">==</span> <span class="dv">1</span>) { <span class="cf">return</span> <span class="dv">0</span><span class="op">;</span> }</span>
<span id="cb17-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-14" tabindex="-1"/>  <span class="cf">return</span> <span class="dv">1</span><span class="op">;</span></span>
<span id="cb17-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb17-15" tabindex="-1"/>}</span></code></pre></div>
<p>And a utility function to schedule a function to run after a delay:</p>
<div class="sourceCode" id="cb18"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb18-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb18-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">runAfter</span>(millis<span class="op">,</span> func) {</span>
<span id="cb18-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb18-2" tabindex="-1"/>  <span class="fu">spawn</span> (<span class="kw">function</span> () {</span>
<span id="cb18-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb18-3" tabindex="-1"/>    <span class="fu">sleep</span>(millis)<span class="op">;</span></span>
<span id="cb18-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb18-4" tabindex="-1"/>    <span class="fu">func</span>()<span class="op">;</span></span>
<span id="cb18-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb18-5" tabindex="-1"/>  })()<span class="op">;</span></span>
<span id="cb18-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb18-6" tabindex="-1"/>}</span></code></pre></div>
<p>With these building blocks, we define the logic gates. Each gate computes its output based on its inputs and schedules the output update after a propagation delay specific to the gate:</p>
<div class="sourceCode" id="cb19"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb19-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-1" tabindex="-1"/><span class="kw">var</span> andGateDelay <span class="op">=</span> <span class="dv">300</span><span class="op">;</span></span>
<span id="cb19-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-2" tabindex="-1"/></span>
<span id="cb19-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-3" tabindex="-1"/><span class="kw">function</span> <span class="fu">AndGate</span>(a<span class="op">,</span> b<span class="op">,</span> out) {</span>
<span id="cb19-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-4" tabindex="-1"/>  <span class="kw">function</span> <span class="fu">compute</span>() {</span>
<span id="cb19-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-5" tabindex="-1"/>    <span class="kw">var</span> newSignal <span class="op">=</span> <span class="fu">and</span>(<span class="fu">getSignal</span>(a)<span class="op">,</span> <span class="fu">getSignal</span>(b))<span class="op">;</span></span>
<span id="cb19-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-6" tabindex="-1"/>    <span class="fu">runAfter</span>(andGateDelay<span class="op">,</span> <span class="kw">function</span> () { <span class="fu">setSignal</span>(out<span class="op">,</span> newSignal)<span class="op">;</span> })<span class="op">;</span></span>
<span id="cb19-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-7" tabindex="-1"/>  }</span>
<span id="cb19-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-8" tabindex="-1"/>  <span class="fu">addAction</span>(a<span class="op">,</span> compute)<span class="op">;</span></span>
<span id="cb19-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-9" tabindex="-1"/>  <span class="fu">addAction</span>(b<span class="op">,</span> compute)<span class="op">;</span></span>
<span id="cb19-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-10" tabindex="-1"/>}</span>
<span id="cb19-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-11" tabindex="-1"/></span>
<span id="cb19-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-12" tabindex="-1"/><span class="kw">var</span> orGateDelay <span class="op">=</span> <span class="dv">500</span><span class="op">;</span></span>
<span id="cb19-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-13" tabindex="-1"/></span>
<span id="cb19-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-14" tabindex="-1"/><span class="kw">function</span> <span class="fu">OrGate</span>(a<span class="op">,</span> b<span class="op">,</span> out) {</span>
<span id="cb19-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-15" tabindex="-1"/>  <span class="kw">function</span> <span class="fu">compute</span>() {</span>
<span id="cb19-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-16" tabindex="-1"/>    <span class="kw">var</span> newSignal <span class="op">=</span> <span class="fu">or</span>(<span class="fu">getSignal</span>(a)<span class="op">,</span> <span class="fu">getSignal</span>(b))<span class="op">;</span></span>
<span id="cb19-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-17" tabindex="-1"/>    <span class="fu">runAfter</span>(orGateDelay<span class="op">,</span> <span class="kw">function</span> () { <span class="fu">setSignal</span>(out<span class="op">,</span> newSignal)<span class="op">;</span> })<span class="op">;</span></span>
<span id="cb19-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-18" tabindex="-1"/>  }</span>
<span id="cb19-19"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-19" tabindex="-1"/>  <span class="fu">addAction</span>(a<span class="op">,</span> compute)<span class="op">;</span></span>
<span id="cb19-20"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-20" tabindex="-1"/>  <span class="fu">addAction</span>(b<span class="op">,</span> compute)<span class="op">;</span></span>
<span id="cb19-21"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-21" tabindex="-1"/>}</span>
<span id="cb19-22"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-22" tabindex="-1"/></span>
<span id="cb19-23"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-23" tabindex="-1"/><span class="kw">var</span> notGateDelay <span class="op">=</span> <span class="dv">200</span><span class="op">;</span></span>
<span id="cb19-24"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-24" tabindex="-1"/></span>
<span id="cb19-25"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-25" tabindex="-1"/><span class="kw">function</span> <span class="fu">NotGate</span>(a<span class="op">,</span> out) {</span>
<span id="cb19-26"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-26" tabindex="-1"/>  <span class="kw">function</span> <span class="fu">compute</span>() {</span>
<span id="cb19-27"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-27" tabindex="-1"/>    <span class="kw">var</span> newSignal <span class="op">=</span> <span class="fu">not</span>(<span class="fu">getSignal</span>(a))<span class="op">;</span></span>
<span id="cb19-28"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-28" tabindex="-1"/>    <span class="fu">runAfter</span>(notGateDelay<span class="op">,</span> <span class="kw">function</span> () { <span class="fu">setSignal</span>(out<span class="op">,</span> newSignal)<span class="op">;</span> })<span class="op">;</span></span>
<span id="cb19-29"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-29" tabindex="-1"/>  }</span>
<span id="cb19-30"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-30" tabindex="-1"/>  <span class="fu">addAction</span>(a<span class="op">,</span> compute)<span class="op">;</span></span>
<span id="cb19-31"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb19-31" tabindex="-1"/>}</span></code></pre></div>
<p>We add the <code>compute</code> action to each input wire, which runs when the input signals change, and sets the signal on the output wire after a delay.</p>
<p>Letâ€™s test an And gate:</p>
<div class="sourceCode" id="cb20"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb20-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-1" tabindex="-1"/><span class="kw">var</span> input1 <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb20-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-2" tabindex="-1"/><span class="kw">var</span> input2 <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb20-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-3" tabindex="-1"/><span class="kw">var</span> output <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb20-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-4" tabindex="-1"/></span>
<span id="cb20-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-5" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting up the probes"</span>)<span class="op">;</span></span>
<span id="cb20-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-6" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"input1"</span><span class="op">,</span> input1)<span class="op">;</span></span>
<span id="cb20-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-7" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"input2"</span><span class="op">,</span> input2)<span class="op">;</span></span>
<span id="cb20-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-8" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"output"</span><span class="op">,</span> output)<span class="op">;</span></span>
<span id="cb20-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-9" tabindex="-1"/></span>
<span id="cb20-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-10" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting up the And gate"</span>)<span class="op">;</span></span>
<span id="cb20-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-11" tabindex="-1"/><span class="fu">AndGate</span>(input1<span class="op">,</span> input2<span class="op">,</span> output)<span class="op">;</span></span>
<span id="cb20-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-12" tabindex="-1"/></span>
<span id="cb20-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-13" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting input1 to 1"</span>)<span class="op">;</span></span>
<span id="cb20-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-14" tabindex="-1"/><span class="fu">setSignal</span>(input1<span class="op">,</span> <span class="dv">1</span>)<span class="op">;</span></span>
<span id="cb20-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-15" tabindex="-1"/></span>
<span id="cb20-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-16" tabindex="-1"/><span class="fu">sleep</span>(<span class="dv">1000</span>)<span class="op">;</span></span>
<span id="cb20-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-17" tabindex="-1"/></span>
<span id="cb20-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-18" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting input2 to 1"</span>)<span class="op">;</span></span>
<span id="cb20-19"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb20-19" tabindex="-1"/><span class="fu">setSignal</span>(input2<span class="op">,</span> <span class="dv">1</span>)<span class="op">;</span></span></code></pre></div>
<p>For probing, we define a helper that logs signal changes with milliseconds elapsed since start of the run:</p>
<div class="sourceCode" id="cb21"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb21-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-1" tabindex="-1"/><span class="kw">var</span> start <span class="op">=</span> <span class="fu">getCurrentMillis</span>()<span class="op">;</span></span>
<span id="cb21-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-2" tabindex="-1"/></span>
<span id="cb21-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-3" tabindex="-1"/><span class="kw">function</span> <span class="fu">log</span>(msg) {</span>
<span id="cb21-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-4" tabindex="-1"/>  <span class="fu">print</span>(<span class="st">"["</span> <span class="op">+</span> (<span class="fu">getCurrentMillis</span>() <span class="op">-</span> start) <span class="op">+</span> <span class="st">"] "</span> <span class="op">+</span> msg)<span class="op">;</span></span>
<span id="cb21-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-5" tabindex="-1"/>}</span>
<span id="cb21-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-6" tabindex="-1"/></span>
<span id="cb21-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-7" tabindex="-1"/><span class="kw">function</span> <span class="fu">addProbe</span>(label<span class="op">,</span> wire) {</span>
<span id="cb21-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-8" tabindex="-1"/>  <span class="fu">addAction</span>(wire<span class="op">,</span> <span class="kw">function</span> () {</span>
<span id="cb21-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-9" tabindex="-1"/>    <span class="fu">log</span>(label <span class="op">+</span> <span class="st">": "</span> <span class="op">+</span> <span class="fu">getSignal</span>(wire))<span class="op">;</span></span>
<span id="cb21-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-10" tabindex="-1"/>  })<span class="op">;</span></span>
<span id="cb21-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb21-11" tabindex="-1"/>}</span></code></pre></div>
<p>The output:</p>
<pre class="plain"><code>[0] &gt;&gt; Setting up the probes
[0] input1: 0
[0] input2: 0
[0] output: 0
[0] &gt;&gt; Setting up the And gate
[0] &gt;&gt; Setting input1 to 1
[0] input1: 1
[1005] &gt;&gt; Setting input2 to 1
[1006] input2: 1
[1310] output: 1</code></pre>
<p>It works as expected. You can notice the sleep and the And gate delay in action.</p>
<h3 id="adders">Adders<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#adders">#</a></h3>
<p>Using the basic logic gates, next we build adders. A <a href="https://en.wikipedia.org/wiki/Half_adder" rel="noopener" target="_blank">Half adder</a> is a digital circuit that adds two bits:</p>
<figure>
<img alt="Half Adder" class="lazyload w-100pct nolink mw-60pct"/>
&lt;noscript&gt;<img alt="Half Adder" class="w-100pct nolink mw-60pct" src="https://abhinavsarkar.net/images/implementing-co-5/half-adder.svg"/>&lt;/noscript&gt;
<figcaption>Half Adder</figcaption>
</figure>
<p>It has two input signals/bits <code>Input1</code> and <code>Input2</code>, and two output bits <code>Sum</code> and <code>Carry</code>. We simply connect the And, Or and Not gates with input, output and intermediate wires in our code as shown in the diagram:</p>
<div class="sourceCode" id="cb23"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb23-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">HalfAdder</span>(label<span class="op">,</span> input1<span class="op">,</span> input2<span class="op">,</span> sum<span class="op">,</span> carry) {</span>
<span id="cb23-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-2" tabindex="-1"/>  <span class="kw">var</span> a <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb23-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-3" tabindex="-1"/>  <span class="kw">var</span> b <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb23-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-4" tabindex="-1"/>  <span class="fu">addProbe</span>(label <span class="op">+</span> <span class="st">"-a"</span><span class="op">,</span> a)<span class="op">;</span></span>
<span id="cb23-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-5" tabindex="-1"/>  <span class="fu">addProbe</span>(label <span class="op">+</span> <span class="st">"-b"</span><span class="op">,</span> b)<span class="op">;</span></span>
<span id="cb23-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-6" tabindex="-1"/></span>
<span id="cb23-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-7" tabindex="-1"/>  <span class="fu">OrGate</span>(input1<span class="op">,</span> input2<span class="op">,</span> a)<span class="op">;</span></span>
<span id="cb23-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-8" tabindex="-1"/>  <span class="fu">AndGate</span>(input1<span class="op">,</span> input2<span class="op">,</span> carry)<span class="op">;</span></span>
<span id="cb23-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-9" tabindex="-1"/>  <span class="fu">NotGate</span>(carry<span class="op">,</span> b)<span class="op">;</span></span>
<span id="cb23-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-10" tabindex="-1"/>  <span class="fu">AndGate</span>(a<span class="op">,</span> b<span class="op">,</span> sum)<span class="op">;</span></span>
<span id="cb23-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb23-11" tabindex="-1"/>}</span></code></pre></div>
<p>Nice and simple. Letâ€™s test it:</p>
<div class="sourceCode" id="cb24"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb24-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-1" tabindex="-1"/><span class="kw">var</span> input1 <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb24-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-2" tabindex="-1"/><span class="kw">var</span> input2 <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb24-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-3" tabindex="-1"/><span class="kw">var</span> sum <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb24-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-4" tabindex="-1"/><span class="kw">var</span> carry <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb24-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-5" tabindex="-1"/></span>
<span id="cb24-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-6" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting up the probes"</span>)<span class="op">;</span></span>
<span id="cb24-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-7" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"input1"</span><span class="op">,</span> input1)<span class="op">;</span></span>
<span id="cb24-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-8" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"input2"</span><span class="op">,</span> input2)<span class="op">;</span></span>
<span id="cb24-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-9" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"sum"</span><span class="op">,</span> sum)<span class="op">;</span></span>
<span id="cb24-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-10" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"carry"</span><span class="op">,</span> carry)<span class="op">;</span></span>
<span id="cb24-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-11" tabindex="-1"/></span>
<span id="cb24-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-12" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting up the half-adder"</span>)<span class="op">;</span></span>
<span id="cb24-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-13" tabindex="-1"/><span class="fu">HalfAdder</span>(<span class="st">"half-adder"</span><span class="op">,</span> input1<span class="op">,</span> input2<span class="op">,</span> sum<span class="op">,</span> carry)<span class="op">;</span></span>
<span id="cb24-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-14" tabindex="-1"/></span>
<span id="cb24-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-15" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting input1 to 1"</span>)<span class="op">;</span></span>
<span id="cb24-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-16" tabindex="-1"/><span class="fu">setSignal</span>(input1<span class="op">,</span> <span class="dv">1</span>)<span class="op">;</span></span>
<span id="cb24-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-17" tabindex="-1"/></span>
<span id="cb24-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-18" tabindex="-1"/><span class="fu">sleep</span>(<span class="dv">1000</span>)<span class="op">;</span></span>
<span id="cb24-19"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-19" tabindex="-1"/></span>
<span id="cb24-20"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-20" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting input2 to 1"</span>)<span class="op">;</span></span>
<span id="cb24-21"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb24-21" tabindex="-1"/><span class="fu">setSignal</span>(input2<span class="op">,</span> <span class="dv">1</span>)<span class="op">;</span></span></code></pre></div>
<p>And the output:</p>
<pre class="plain"><code>[0] &gt;&gt; Setting up the probes
[0] input1: 0
[0] input2: 0
[0] sum: 0
[0] carry: 0
[0] &gt;&gt; Setting up the half-adder
[0] half-adder-a: 0
[0] half-adder-b: 0
[1] &gt;&gt; Setting input1 to 1
[1] input1: 1
[205] half-adder-b: 1
[505] half-adder-a: 1
[811] sum: 1
[1003] &gt;&gt; Setting input2 to 1
[1003] input2: 1
[1307] carry: 1
[1508] half-adder-b: 0
[1811] sum: 0</code></pre>
<p>In binary, <code>1 + 1 = 10</code>. Correct! Notice again how the signal propagation through the gates is delayed. Next up is the full adder.</p>
<p>A <a href="https://en.wikipedia.org/wiki/Full_adder" rel="noopener" target="_blank">Full adder</a> adds three bits, two inputs and a carry-in:</p>
<figure>
<img alt="Full Adder" class="lazyload w-100pct nolink mw-70pct"/>
&lt;noscript&gt;<img alt="Full Adder" class="w-100pct nolink mw-70pct" src="https://abhinavsarkar.net/images/implementing-co-5/full-adder.svg"/>&lt;/noscript&gt;
<figcaption>Full Adder</figcaption>
</figure>
<p>Notice that a full adder uses two half adders. Again, we follow the diagram and connect the wires:</p>
<div class="sourceCode" id="cb26"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb26-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">FullAdder</span>(label<span class="op">,</span> input1<span class="op">,</span> input2<span class="op">,</span> carryIn<span class="op">,</span> sum<span class="op">,</span> carryOut) {</span>
<span id="cb26-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-2" tabindex="-1"/>  <span class="kw">var</span> hsum <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb26-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-3" tabindex="-1"/>  <span class="kw">var</span> carry1 <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb26-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-4" tabindex="-1"/>  <span class="kw">var</span> carry2 <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb26-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-5" tabindex="-1"/></span>
<span id="cb26-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-6" tabindex="-1"/>  <span class="fu">addProbe</span>(label <span class="op">+</span> <span class="st">"-hsum"</span><span class="op">,</span> hsum)<span class="op">;</span></span>
<span id="cb26-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-7" tabindex="-1"/>  <span class="fu">addProbe</span>(label <span class="op">+</span> <span class="st">"-carry1"</span><span class="op">,</span> carry1)<span class="op">;</span></span>
<span id="cb26-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-8" tabindex="-1"/>  <span class="fu">addProbe</span>(label <span class="op">+</span> <span class="st">"-carry2"</span><span class="op">,</span> carry2)<span class="op">;</span></span>
<span id="cb26-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-9" tabindex="-1"/></span>
<span id="cb26-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-10" tabindex="-1"/>  <span class="fu">HalfAdder</span>(label <span class="op">+</span> <span class="st">"-HA-"</span> <span class="op">+</span> <span class="dv">1</span><span class="op">,</span> input2<span class="op">,</span> carryIn<span class="op">,</span> hsum<span class="op">,</span> carry1)<span class="op">;</span></span>
<span id="cb26-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-11" tabindex="-1"/>  <span class="fu">HalfAdder</span>(label <span class="op">+</span> <span class="st">"-HA-"</span> <span class="op">+</span> <span class="dv">2</span><span class="op">,</span> input1<span class="op">,</span> hsum<span class="op">,</span> sum<span class="op">,</span> carry2)<span class="op">;</span></span>
<span id="cb26-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-12" tabindex="-1"/>  <span class="fu">OrGate</span>(carry1<span class="op">,</span> carry2<span class="op">,</span> carryOut)<span class="op">;</span></span>
<span id="cb26-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb26-13" tabindex="-1"/>}</span></code></pre></div>
<p>Letâ€™s skip the demo for full adder and jump to something more exciting.</p>
<h3 id="ripple-carry-adder">Ripple-carry Adder<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#ripple-carry-adder">#</a></h3>
<p>A <a href="https://en.wikipedia.org/wiki/Ripple-carry_adder" rel="noopener" target="_blank">Ripple-carry adder</a> chains together multiple full adders to add multi-bit numbers. The diagram below shows a four-bit adder:</p>
<figure>
<img alt="Ripple-carry Adder" class="lazyload w-100pct nolink extra-width"/>
&lt;noscript&gt;<img alt="Ripple-carry Adder" class="w-100pct nolink extra-width" src="https://abhinavsarkar.net/images/implementing-co-5/ripple-adder.svg"/>&lt;/noscript&gt;
<figcaption>Ripple-carry Adder</figcaption>
</figure>
<p>We create a ripple-carry adder that can add any number of bits. First we need some helper functions:</p>
<div class="sourceCode" id="cb27"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb27-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">Bits</span>(label<span class="op">,</span> n) {</span>
<span id="cb27-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-2" tabindex="-1"/>  <span class="kw">var</span> bits <span class="op">=</span> <span class="fu">Empty</span>()<span class="op">;</span></span>
<span id="cb27-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-3" tabindex="-1"/>  <span class="kw">var</span> bit <span class="op">=</span> <span class="kw">null</span><span class="op">;</span></span>
<span id="cb27-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-4" tabindex="-1"/>  <span class="cf">while</span> (n <span class="op">&gt;</span> <span class="dv">0</span>) {</span>
<span id="cb27-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-5" tabindex="-1"/>    bit <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb27-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-6" tabindex="-1"/>    <span class="fu">addProbe</span>(label <span class="op">+</span> (n <span class="op">-</span> <span class="dv">1</span>)<span class="op">,</span> bit)<span class="op">;</span></span>
<span id="cb27-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-7" tabindex="-1"/>    bits <span class="op">=</span> <span class="fu">prepend</span>(bits<span class="op">,</span> bit)<span class="op">;</span></span>
<span id="cb27-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-8" tabindex="-1"/>    n <span class="op">=</span> n <span class="op">-</span> <span class="dv">1</span><span class="op">;</span></span>
<span id="cb27-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-9" tabindex="-1"/>  }</span>
<span id="cb27-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-10" tabindex="-1"/>  <span class="cf">return</span> bits<span class="op">;</span></span>
<span id="cb27-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-11" tabindex="-1"/>}</span>
<span id="cb27-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-12" tabindex="-1"/></span>
<span id="cb27-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-13" tabindex="-1"/><span class="kw">function</span> <span class="fu">setBits</span>(bits<span class="op">,</span> bitValues) {</span>
<span id="cb27-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-14" tabindex="-1"/>  <span class="cf">if</span> (bits <span class="op">!=</span> <span class="fu">Empty</span>()) {</span>
<span id="cb27-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-15" tabindex="-1"/>    <span class="fu">setSignal</span>(<span class="fu">first</span>(bits)<span class="op">,</span> <span class="fu">first</span>(bitValues))<span class="op">;</span></span>
<span id="cb27-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-16" tabindex="-1"/>    <span class="fu">setBits</span>(<span class="fu">rest</span>(bits)<span class="op">,</span> <span class="fu">rest</span>(bitValues))<span class="op">;</span></span>
<span id="cb27-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-17" tabindex="-1"/>  }</span>
<span id="cb27-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb27-18" tabindex="-1"/>}</span></code></pre></div>
<p><code>Bits</code> creates a list of wires to represent an N-bit input/output. <code>setBits</code> sets the bits of a N-bit wire list to a given N-bit value. Now we write a ripple-carry adder:</p>
<div class="sourceCode" id="cb28"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb28-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-1" tabindex="-1"/><span class="kw">function</span> <span class="fu">RippleCarryAdder</span>(label<span class="op">,</span> a<span class="op">,</span> b<span class="op">,</span> carryIn<span class="op">,</span> sum<span class="op">,</span> carryOut) {</span>
<span id="cb28-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-2" tabindex="-1"/>  <span class="kw">var</span> aBits <span class="op">=</span> a<span class="op">;</span></span>
<span id="cb28-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-3" tabindex="-1"/>  <span class="kw">var</span> bBits <span class="op">=</span> b<span class="op">;</span></span>
<span id="cb28-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-4" tabindex="-1"/>  <span class="kw">var</span> sumBits <span class="op">=</span> sum<span class="op">;</span></span>
<span id="cb28-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-5" tabindex="-1"/>  <span class="kw">var</span> cIn <span class="op">=</span> carryIn<span class="op">;</span></span>
<span id="cb28-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-6" tabindex="-1"/>  <span class="kw">var</span> cOut <span class="op">=</span> <span class="kw">null</span><span class="op">;</span></span>
<span id="cb28-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-7" tabindex="-1"/>  <span class="kw">var</span> i <span class="op">=</span> <span class="dv">1</span><span class="op">;</span></span>
<span id="cb28-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-8" tabindex="-1"/>  <span class="cf">while</span> (aBits <span class="op">!=</span> <span class="fu">Empty</span>()) {</span>
<span id="cb28-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-9" tabindex="-1"/>    cOut <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb28-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-10" tabindex="-1"/>    <span class="fu">addProbe</span>(label <span class="op">+</span> <span class="st">"-carryIn-"</span> <span class="op">+</span> i<span class="op">,</span> cIn)<span class="op">;</span></span>
<span id="cb28-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-11" tabindex="-1"/>    <span class="fu">addProbe</span>(label <span class="op">+</span> <span class="st">"-carryOut-"</span> <span class="op">+</span> i<span class="op">,</span> cOut)<span class="op">;</span></span>
<span id="cb28-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-12" tabindex="-1"/>    <span class="fu">FullAdder</span>(label <span class="op">+</span> <span class="st">"-FA-"</span> <span class="op">+</span> i<span class="op">,</span></span>
<span id="cb28-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-13" tabindex="-1"/>      <span class="fu">first</span>(aBits)<span class="op">,</span> <span class="fu">first</span>(bBits)<span class="op">,</span> cIn<span class="op">,</span> <span class="fu">first</span>(sumBits)<span class="op">,</span> cOut)<span class="op">;</span></span>
<span id="cb28-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-14" tabindex="-1"/>    aBits <span class="op">=</span> <span class="fu">rest</span>(aBits)<span class="op">;</span></span>
<span id="cb28-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-15" tabindex="-1"/>    bBits <span class="op">=</span> <span class="fu">rest</span>(bBits)<span class="op">;</span></span>
<span id="cb28-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-16" tabindex="-1"/>    sumBits <span class="op">=</span> <span class="fu">rest</span>(sumBits)<span class="op">;</span></span>
<span id="cb28-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-17" tabindex="-1"/>    cIn <span class="op">=</span> cOut<span class="op">;</span></span>
<span id="cb28-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-18" tabindex="-1"/>    i <span class="op">=</span> i <span class="op">+</span> <span class="dv">1</span><span class="op">;</span></span>
<span id="cb28-19"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-19" tabindex="-1"/>  }</span>
<span id="cb28-20"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-20" tabindex="-1"/>  <span class="fu">connect</span>(cOut<span class="op">,</span> carryOut)<span class="op">;</span></span>
<span id="cb28-21"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb28-21" tabindex="-1"/>}</span></code></pre></div>
<p>The ripple-carry adder uses one full adder per bit, cascading the carry-out bit of each input bit-pairâ€™s sum to the next pair of bits. To demonstrate, letâ€™s add two 4-bit numbers:</p>
<div class="sourceCode" id="cb29"><pre class="sourceCode javascript numberSource"><code class="sourceCode javascript"><span id="cb29-1"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-1" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting up the probes"</span>)<span class="op">;</span></span>
<span id="cb29-2"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-2" tabindex="-1"/><span class="kw">var</span> a <span class="op">=</span> <span class="fu">Bits</span>(<span class="st">"a"</span><span class="op">,</span> <span class="dv">4</span>)<span class="op">;</span></span>
<span id="cb29-3"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-3" tabindex="-1"/><span class="kw">var</span> b <span class="op">=</span> <span class="fu">Bits</span>(<span class="st">"b"</span><span class="op">,</span> <span class="dv">4</span>)<span class="op">;</span></span>
<span id="cb29-4"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-4" tabindex="-1"/><span class="kw">var</span> sum <span class="op">=</span> <span class="fu">Bits</span>(<span class="st">"sum"</span><span class="op">,</span> <span class="dv">4</span>)<span class="op">;</span></span>
<span id="cb29-5"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-5" tabindex="-1"/><span class="kw">var</span> carryIn <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb29-6"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-6" tabindex="-1"/><span class="kw">var</span> carryOut <span class="op">=</span> <span class="fu">Wire</span>()<span class="op">;</span></span>
<span id="cb29-7"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-7" tabindex="-1"/></span>
<span id="cb29-8"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-8" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"carryIn"</span><span class="op">,</span> carryIn)<span class="op">;</span></span>
<span id="cb29-9"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-9" tabindex="-1"/><span class="fu">addProbe</span>(<span class="st">"carryOut"</span><span class="op">,</span> carryOut)<span class="op">;</span></span>
<span id="cb29-10"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-10" tabindex="-1"/></span>
<span id="cb29-11"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-11" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting up the ripple carry adder"</span>)<span class="op">;</span></span>
<span id="cb29-12"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-12" tabindex="-1"/><span class="fu">RippleCarryAdder</span>(<span class="st">"RCA"</span><span class="op">,</span> a<span class="op">,</span> b<span class="op">,</span> carryIn<span class="op">,</span> sum<span class="op">,</span> carryOut)<span class="op">;</span></span>
<span id="cb29-13"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-13" tabindex="-1"/></span>
<span id="cb29-14"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-14" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting carryIn to 0"</span>)<span class="op">;</span></span>
<span id="cb29-15"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-15" tabindex="-1"/><span class="fu">setSignal</span>(carryIn<span class="op">,</span> <span class="dv">0</span>)<span class="op">;</span></span>
<span id="cb29-16"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-16" tabindex="-1"/></span>
<span id="cb29-17"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-17" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting a to 5 = 0101 in binary"</span>)<span class="op">;</span></span>
<span id="cb29-18"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-18" tabindex="-1"/><span class="fu">setBits</span>(a<span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">1</span><span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">0</span><span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">1</span><span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">0</span><span class="op">,</span> <span class="fu">Empty</span>())))))<span class="op">;</span></span>
<span id="cb29-19"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-19" tabindex="-1"/></span>
<span id="cb29-20"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-20" tabindex="-1"/><span class="fu">sleep</span>(<span class="dv">1000</span>)<span class="op">;</span></span>
<span id="cb29-21"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-21" tabindex="-1"/></span>
<span id="cb29-22"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-22" tabindex="-1"/><span class="fu">log</span>(<span class="st">"&gt;&gt; Setting b to 11 = 1011 in binary"</span>)<span class="op">;</span></span>
<span id="cb29-23"><a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#cb29-23" tabindex="-1"/><span class="fu">setBits</span>(b<span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">1</span><span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">1</span><span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">0</span><span class="op">,</span> <span class="fu">Cons</span>(<span class="dv">1</span><span class="op">,</span> <span class="fu">Empty</span>())))))<span class="op">;</span></span></code></pre></div>
<p>This one runs for a while because of the collective delays.</p>
<details>

The output

<pre class="plain"><code>[0] &gt;&gt; Setting up the probes
[1] a3: 0
[1] a2: 0
[1] a1: 0
[1] a0: 0
[1] b3: 0
[1] b2: 0
[1] b1: 0
[1] b0: 0
[1] sum3: 0
[1] sum2: 0
[1] sum1: 0
[1] sum0: 0
[1] carryIn: 0
[1] carryOut: 0
[1] &gt;&gt; Setting up the ripple carry adder
[1] RCA-carryIn-1: 0
[1] RCA-carryOut-1: 0
[1] RCA-FA-1-hsum: 0
[1] RCA-FA-1-carry1: 0
[1] RCA-FA-1-carry2: 0
[1] RCA-FA-1-HA-1-a: 0
[1] RCA-FA-1-HA-1-b: 0
[1] RCA-FA-1-HA-2-a: 0
[1] RCA-FA-1-HA-2-b: 0
[1] RCA-carryIn-2: 0
[1] RCA-carryOut-2: 0
[1] RCA-FA-2-hsum: 0
[2] RCA-FA-2-carry1: 0
[2] RCA-FA-2-carry2: 0
[2] RCA-FA-2-HA-1-a: 0
[2] RCA-FA-2-HA-1-b: 0
[2] RCA-FA-2-HA-2-a: 0
[2] RCA-FA-2-HA-2-b: 0
[2] RCA-carryIn-3: 0
[2] RCA-carryOut-3: 0
[2] RCA-FA-3-hsum: 0
[2] RCA-FA-3-carry1: 0
[2] RCA-FA-3-carry2: 0
[2] RCA-FA-3-HA-1-a: 0
[2] RCA-FA-3-HA-1-b: 0
[2] RCA-FA-3-HA-2-a: 0
[2] RCA-FA-3-HA-2-b: 0
[2] RCA-carryIn-4: 0
[2] RCA-carryOut-4: 0
[2] RCA-FA-4-hsum: 0
[2] RCA-FA-4-carry1: 0
[2] RCA-FA-4-carry2: 0
[2] RCA-FA-4-HA-1-a: 0
[2] RCA-FA-4-HA-1-b: 0
[2] RCA-FA-4-HA-2-a: 0
[2] RCA-FA-4-HA-2-b: 0
[2] &gt;&gt; Setting carryIn to 0
[2] &gt;&gt; Setting a to 5 = 0101 in binary
[2] a0: 1
[3] a2: 1
[205] RCA-FA-1-HA-1-b: 1
[205] RCA-FA-1-HA-2-b: 1
[205] RCA-FA-2-HA-1-b: 1
[207] RCA-FA-2-HA-2-b: 1
[207] RCA-FA-3-HA-1-b: 1
[207] RCA-FA-3-HA-2-b: 1
[207] RCA-FA-4-HA-1-b: 1
[207] RCA-FA-4-HA-2-b: 1
[505] RCA-FA-1-HA-2-a: 1
[505] RCA-FA-3-HA-2-a: 1
[810] sum0: 1
[810] sum2: 1
[1014] &gt;&gt; Setting b to 11 = 1011 in binary
[1015] b0: 1
[1015] b1: 1
[1016] b3: 1
[1521] RCA-FA-1-HA-1-a: 1
[1521] RCA-FA-2-HA-1-a: 1
[1521] RCA-FA-4-HA-1-a: 1
[1826] RCA-FA-1-hsum: 1
[1826] RCA-FA-2-hsum: 1
[1826] RCA-FA-4-hsum: 1
[2129] RCA-FA-1-carry2: 1
[2331] RCA-FA-2-HA-2-a: 1
[2331] RCA-FA-4-HA-2-a: 1
[2331] RCA-FA-1-HA-2-b: 0
[2633] RCA-carryOut-1: 1
[2633] sum1: 1
[2633] sum3: 1
[2633] sum0: 0
[2634] RCA-carryIn-2: 1
[2936] RCA-FA-2-carry1: 1
[3138] RCA-FA-2-HA-1-b: 0
[3440] RCA-carryOut-2: 1
[3440] RCA-FA-2-hsum: 0
[3440] RCA-carryIn-3: 1
[3946] RCA-FA-2-HA-2-a: 0
[3946] RCA-FA-3-HA-1-a: 1
[4251] sum1: 0
[4252] RCA-FA-3-hsum: 1
[4556] RCA-FA-3-carry2: 1
[4756] RCA-FA-3-HA-2-b: 0
[5059] RCA-carryOut-3: 1
[5059] sum2: 0
[5059] RCA-carryIn-4: 1
[5363] RCA-FA-4-carry1: 1
[5564] RCA-FA-4-HA-1-b: 0
[5867] RCA-carryOut-4: 1
[5867] RCA-FA-4-hsum: 0
[5867] carryOut: 1
[6370] RCA-FA-4-HA-2-a: 0
[6675] sum3: 0</code></pre>
</details>
<p>Let me pick out the final output:</p>
<pre class="plain"><code>sum0: 0
sum1: 0
sum2: 0
carryOut: 1
sum3: 0</code></pre>
<p>We add <code>0101</code> and <code>1011</code> in binary, resulting in <code>10000</code>, which is correct again. Everything works perfectly. With sleep, weâ€™ve now implemented all major features of <span class="fancy">Co</span>â€”a complete concurrent language with first-class coroutines, channels, and time-based scheduling.</p>
<h2 id="conclusion">Conclusion<a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#conclusion">#</a></h2>
<p>With the addition of sleep, weâ€™ve completed our implementation of <span class="fancy">Co</span>â€”a small language with coroutines and channels. Over these five posts, we went from parsing source code to building a full interpreter that handles cooperative multitasking using coroutines.</p>
<p>The key insight was realizing that coroutines are just environments plus continuations. By designing our interpreter to use continuation-passing style, we gained the ability to suspend execution at any point and resume it later. Channels built naturally on top of that, providing a way for coroutines to synchronize and pass messages. And sleep extended the scheduler to handle time-based execution, unlocking patterns like timeouts and periodic tasks.</p>
<p>The examples we built along the wayâ€”pubsub system, actor system, and digital circuit simulationâ€”show what becomes possible once these primitives are in place. Starting with basic arithmetic and functions, we ended up with a language capable of expressing real concurrent programs.</p>
<p>What comes next? Maybe a compiler for <span class="fancy">Co</span>? Stay tuned by subscribing to the <a href="https://abhinavsarkar.net/feed.atom?mtm_campaign=feed">feed</a> or the <a href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#subscription">email newsletter</a>.</p>
<p>The full code for the <span class="fancy">Co</span> interpreter is available <a href="https://abhinavsarkar.net/code/co-interpreter.html?mtm_campaign=feed">here</a>.</p>
<p class="like-msg">
If you have any questions or comments, please leave a comment below. If you liked this post, please share it. Thanks for reading!
</p>
<section class="footnotes footnotes-end-of-document" id="footnotes">
<hr/>
<ol>
<li id="fn1"><p>The sleep implementation in <span class="fancy">Co</span> is not interruptible. That is, if a coroutine is sleeping, it cannot be resumed before the specified duration. This is different from sleep implementations in most programming languages, where the sleep operation can be interrupted by sending a signal to the sleeping ToC.<a class="footnote-back" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fnref1">â†©ï¸�</a></p></li>
<li id="fn2"><p>Threads in GHC are <em><a href="https://en.wikipedia.org/wiki/Green_Threads" rel="noopener" target="_blank">Green Threads</a></em> and are very cheap to create and run. It is perfectly okay to fork a new one for each delayed coroutine.<a class="footnote-back" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fnref2">â†©ï¸�</a></p></li>
<li id="fn3"><p>So in a way, we cheat here by using the sleep primitive provided by the GHC runtime to implement our sleep primitive. If we write a compiler for <span class="fancy">Co</span>, weâ€™ll have to write our own runtime where weâ€™ll have to implement our sleep function using the functionalities provided by the operating systems.<a class="footnote-back" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fnref3">â†©ï¸�</a></p></li>
<li id="fn4"><p>To learn more about how <code class="sourceCode haskell"><span class="dt">MVar</span></code>s can be used to communicate between threads, read the <a href="https://web.archive.org/web/20260116/https://book.realworldhaskell.org/read/concurrent-and-multicore-programming.html#id673028" rel="noopener" target="_blank">chapter 24 of Real World Haskell</a>.<a class="footnote-back" href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom#fnref4">â†©ï¸�</a></p></li>
</ol>
</section><section class="series-info">
  <p>This post is a part of the series: <strong>Implementing Co, a Small Language With Coroutines</strong>.</p>
  <ol>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-1/?mtm_campaign=feed">The Parser</a> 
    </li>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-2/?mtm_campaign=feed">The Interpreter</a> 
    </li>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-3/?mtm_campaign=feed">Adding Coroutines</a> 
    </li>
    <li>
      
       <a href="https://abhinavsarkar.net/posts/implementing-co-4/?mtm_campaign=feed">Adding Channels</a> 
    </li>
    <li>
       <strong>Adding Sleep</strong> ğŸ‘ˆ
      
    </li>
  </ol>
</section>
<hr/><p>Thanks for reading this post via feed. Feeds are great, and you're great for using them. â™¥</p><p>This post was originally published on <a href="https://abhinavsarkar.net/posts/implementing-co-5/?mtm_campaign=feed">abhinavsarkar.net</a>.</p><h3>Like, repost, or comment on:</h3><ul><li><a class="u-syndication" href="https://fantastic.earth/@abnv/115905408192306242" rel="syndication" target="_blank">Fediverse</a></li>
<li><a class="u-syndication" href="https://lobste.rs/s/taxbgk" rel="syndication" target="_blank">Lobsters</a></li>
<li><a class="u-syndication" href="https://www.reddit.com/r/haskell/comments/1qehkip/" rel="syndication" target="_blank">Reddit</a></li>
<li><a class="u-syndication" href="https://discourse.haskell.org/t//13549" rel="syndication" target="_blank">Discourse</a></li>
<li><a class="u-syndication" href="https://news.ycombinator.com/item?id=46645854" rel="syndication" target="_blank">Hacker News</a></li><li><a href="https://abhinavsarkar.net/posts/implementing-co-5/?mtm_campaign=feed#comment-container">My website</a></li></ul><p>Read more of my <a href="https://abhinavsarkar.net/posts/">posts</a> and <a href="https://abhinavsarkar.net/notes/">notes</a>.</p><img alt="" src="https://anna.abhinavsarkar.net/matomo.php?idsite=1&amp;rec=1" style="border: 0;"/></div>
    </content>
    <updated>2026-01-16T00:00:00Z</updated>
    <published>2026-01-16T00:00:00Z</published>
    <category label="programming-languages" term="programming-languages"/>
    <category label="interpreters" term="interpreters"/>
    <category label="concurrency" term="concurrency"/>
    <category label="haskell" term="haskell"/>
    <category label="co" term="co"/>
    <author>
      <name>Abhinav Sarkar</name>
      <email>abhinav@abhinavsarkar.net</email>
      <uri>https://abhinavsarkar.net/about/</uri>
    </author>
    <source>
      <id>https://abhinavsarkar.net/posts/tags/haskell/feed.atom</id>
      <icon>https://abhinavsarkar.net/images/favicon.ico</icon>
      <category label="JSON" term="JSON"/>
      <category label="advent-of-code" term="advent-of-code"/>
      <category label="brainfuck" term="brainfuck"/>
      <category label="build-systems" term="build-systems"/>
      <category label="co" term="co"/>
      <category label="compilers" term="compilers"/>
      <category label="concurrency" term="concurrency"/>
      <category label="continuations" term="continuations"/>
      <category label="haskell" term="haskell"/>
      <category label="interpreters" term="interpreters"/>
      <category label="nilenso" term="nilenso"/>
      <category label="nix" term="nix"/>
      <category label="notes" term="notes"/>
      <category label="parsers" term="parsers"/>
      <category label="programming-languages" term="programming-languages"/>
      <category label="puzzles" term="puzzles"/>
      <category label="type-level" term="type-level"/>
      <category label="virtual-machines" term="virtual-machines"/>
      <category label="zippers" term="zippers"/>
      <author>
        <name>Abhinav Sarkar</name>
        <email>abhinav@abhinavsarkar.net</email>
        <uri>https://abhinavsarkar.net/about/</uri>
      </author>
      <link href="https://abhinavsarkar.net/posts/tags/haskell/feed.atom" rel="self" type="application/atom+xml"/>
      <link href="https://abhinavsarkar.net/posts/tags/haskell/" rel="alternate" type="text/html"/>
      <rights>© 2017–2026, Abhinav Sarkar</rights>
      <title>Posts tagged ‘haskell’ on abhinavsarkar.net</title>
      <updated>2026-01-16T00:00:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://tweag.io/blog/2026-01-15-codeql-wrapper/</id>
    <link href="https://tweag.io/blog/2026-01-15-codeql-wrapper/" rel="alternate" type="text/html"/>
    <title>Streamlining CodeQL Analysis with CodeQL Wrapper</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Security in software development has evolved dramatically, yet it’s still one of the easiest things to postpone. Everyone knows it should start early — vulnerabilities are cheaper and simpler to fix when caught before deployment, but deadlines and complexity often push it to the sidelines.</p>
<p>That’s where Static Application Security Testing (SAST) tools like CodeQL come in.</p>
<p>If you’ve used CodeQL, you already know how powerful it is. Instead of relying on predefined rules or pattern matching, it treats your code like data, allowing deep semantic analysis that finds subtle, logic-based vulnerabilities other scanners miss — and it does so with impressive precision, minimizing false positives.</p>
<p>If this is your first contact with CodeQL, it’s worth checking out <a href="https://www.tweag.io/blog/2025-08-07-codeql-intro/">this great introduction</a> before diving in.</p>
<p>However, while CodeQL’s power is undeniable, integrating it smoothly across large organizations, monorepos, or custom CI/CD pipelines can be challenging.</p>
<p>Here are some of the issues teams typically encounter:</p>
<ul>
<li>Multiple programming languages and build systems in a single repository</li>
<li>The need to scan only what actually changed in a pull request</li>
<li>Running CodeQL outside of GitHub Actions</li>
</ul>
<p>We’ve been there ourselves. That’s why we built CodeQL Wrapper, an open-source tool that simplifies running CodeQL in any environment, no matter how complex your repo or CI system might be.</p>
<p>This post explains why we built it, what it does in practice, and how to use it to simplify your CodeQL workflows without a deep technical dive.</p>
<h2 id="what-is-codeql-wrapper"><a class="anchor before" href="https://www.tweag.io/rss.xml#what-is-codeql-wrapper"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>What Is CodeQL Wrapper?</h2>
<p><a href="https://github.com/tweag/codeql-wrapper"><strong>CodeQL Wrapper</strong></a> is a universal Python CLI that abstracts away much of the setup pain and provides a consistent way to run CodeQL across projects.</p>
<p>It allows you to run CodeQL anywhere — locally or in CI systems like GitHub Actions, Jenkins, Azure Pipelines, CircleCI, or Harness while ensuring consistent behavior across environments.</p>
<p>It automatically fetches and installs the correct version of the CodeQL CLI, meaning your local runs and CI analyses always stay in sync. And even if your pipelines don’t run on GitHub, CodeQL Wrapper can still send results back to GitHub Advanced Security in SARIF format for centralized visibility.</p>
<p>Beyond simplifying setup, CodeQL Wrapper helps teams maintain consistent configuration using a <strong>flexible .codeql.json file.</strong> This lets you define build modes, custom queries, and paths once — then apply them consistently across projects.</p>
<p>But where the wrapper really shines is in its ability to handle monorepos, tackling two of the biggest pain points: language detection and performance.</p>
<h3 id="automatic-language-detection"><a class="anchor before" href="https://www.tweag.io/rss.xml#automatic-language-detection"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Automatic Language Detection</h3>
<p>Many modern projects mix languages. A Python API here, a JavaScript frontend there, maybe a bit of Go for background services.</p>
<p>CodeQL’s default GitHub Action setup requires manually specifying which language to analyze. That’s fine for small projects, but a maintenance nightmare for monorepos.</p>
<p>CodeQL Wrapper removes this burden entirely. It automatically detects languages present in your codebase and configures the analysis for you.</p>
<p>This automation ensures nothing slips through the cracks (no forgotten languages, no partial scans) and keeps your configuration simple and future-proof.</p>
<h3 id="smarter-performance-through-parallelization-and-change-detection"><a class="anchor before" href="https://www.tweag.io/rss.xml#smarter-performance-through-parallelization-and-change-detection"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Smarter Performance Through Parallelization and Change Detection</h3>
<p>Running CodeQL can be time-consuming, especially in large repositories. CodeQL Wrapper optimizes this in two complementary ways:
it runs analyses in parallel and skips unchanged code altogether.</p>
<p>The parallelization happens on two levels:</p>
<ul>
<li>Across projects – In the context of CodeQL Wrapper, a project refers to a distinct subdirectory within a repository that can be analyzed independently. Each project gets its own CodeQL database, allowing multiple components to be analyzed simultaneously.</li>
<li>Within each project – If a project contains multiple languages (Python, JavaScript, Go, etc) CodeQL Wrapper can run analysis for each language concurrently instead of processing them sequentially.</li>
</ul>
<p>The tool intelligently adapts to your system’s hardware using all available CPU cores without exhausting memory. Though you can manually tune the settings if needed.</p>
<p>And because avoiding unnecessary work is even better than parallelizing it, CodeQL Wrapper includes a change detection algorithm that analyzes only the code that has actually changed since the last scan.</p>
<p>It uses Git to identify modified files and determines which projects those files belong to. If no relevant files have changed, the tool automatically skips redundant analysis steps, avoiding unnecessary database creation and query execution, cutting analysis time <strong>from hours to minutes</strong> on large monorepos.</p>
<p>At a high level, the wrapper determines a base commit (from CI variables or an explicit flag), fetches any missing refs, computes the diff, and maps changed files to projects. Only those projects proceed to database creation and query execution. The logic is platform‑agnostic, so the same behavior applies whether you run in GitHub Actions, Azure Pipelines, CircleCI, Harness, or locally.</p>
<h3 id="installing-and-running-codeql-wrapper"><a class="anchor before" href="https://www.tweag.io/rss.xml#installing-and-running-codeql-wrapper"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Installing and Running CodeQL Wrapper</h3>
<p>We designed CodeQL Wrapper so that the first run feels frictionless and the hundredth run still feels predictable. Installation is a single step, and from there you can either prepare the environment or jump straight into analysis.</p>
<div class="gatsby-highlight"><pre class="language-bash"><code class="language-bash">pip <span class="token function">install</span> codeql-wrapper</code></pre></div>
<p>This will install the tool in your environment. Once installed, you’ll have access to two commands:</p>
<ul>
<li>install – Installs CodeQL CLI and query packs (you can pin a specific version if needed).</li>
<li>analyze – Runs CodeQL analysis on your project(s), with automatic project detection and parallelized scans.
If you only need the CodeQL CLI (and don’t want to run a scan yet), just run:</li>
</ul>
<div class="gatsby-highlight"><pre class="language-bash"><code class="language-bash">codeql-wrapper <span class="token function">install</span></code></pre></div>
<p>This fetches the CodeQL CLI and query packs, and you can pin versions with <code class="language-text">--version</code> to keep results reproducible. When you’re ready to analyze, <code class="language-text">analyze</code> takes over the heavy lifting: it confirms CodeQL is installed, detects languages and projects, creates databases, and runs queries in parallel to suit your hardware.</p>
<h3 id="basic-usage"><a class="anchor before" href="https://www.tweag.io/rss.xml#basic-usage"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Basic Usage</h3>
<p>Most teams start by pointing the wrapper at the repository root. That’s intentional: you shouldn’t need to hand‑craft language lists or database steps to get useful results.</p>
<div class="gatsby-highlight"><pre class="language-bash"><code class="language-bash">codeql-wrapper analyze ./repo</code></pre></div>
<p>Behind the scenes, the wrapper inspects your codebase, identifies the languages that matter, and builds one database per language. If your repo is a monorepo, it scopes the work to projects it finds, running them concurrently. The default configuration leans on safe, broadly applicable queries and <code class="language-text">build-mode: none</code>, which makes first‑time adoption smooth even without bespoke build instructions.</p>
<p>As codebases grow, consistency matters more than one‑off tweaks. The wrapper’s model — automatic detection plus ergonomic overrides — offers practical advantages: it adapts to monorepos with different needs per project, keeps query policies uniform across CI providers, and makes exceptions explicit without touching pipeline YAML.</p>
<h3 id="why-ci-integration-is-easier"><a class="anchor before" href="https://www.tweag.io/rss.xml#why-ci-integration-is-easier"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Why CI Integration is Easier</h3>
<p>Rather than generic promises, here’s what changes in day‑to‑day pipelines:</p>
<ul>
<li>One command surface: <code class="language-text">install</code> and <code class="language-text">analyze</code>. The wrapper handles language detection, database creation, and query execution, so pipelines don’t need fragile per‑language steps.</li>
<li>Version pinning: <code class="language-text">install --version</code> locks the CodeQL CLI version, avoiding environment drift between developer machines and CI runners.</li>
<li>Monorepo awareness: Independent projects (subdirectories) are processed in parallel, reducing custom CI logic.</li>
<li>Change detection: Git diffs are mapped to affected projects; unchanged ones are skipped, cutting incremental scan time.</li>
<li>Consistent outputs: SARIF is generated uniformly, so reporting stays the same regardless of CI platform.</li>
</ul>
<h2 id="usage-examples"><a class="anchor before" href="https://www.tweag.io/rss.xml#usage-examples"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Usage Examples</h2>
<p>Local runs establish trust, but enterprise adoption hinges on repeatable automation. To make that transition easy, we ship <a href="https://github.com/tweag/codeql-wrapper-pipelines">codeql-wrapper-pipelines</a>, a companion repository with templates for common CI/CD providers. These examples handle the practicalities — auth, artifacts, and reporting — while keeping the interface consistent. You get one way of invoking the wrapper, regardless of whether your pipelines live in GitHub Actions, Azure Pipelines, CircleCI, Harness, or elsewhere.</p>
<h2 id="why-we-built-it"><a class="anchor before" href="https://www.tweag.io/rss.xml#why-we-built-it"><svg height="16" version="1.1" viewBox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"/></svg></a>Why We Built It</h2>
<p>Our work with global enterprises adopting GitHub Advanced Security led us to build both <a href="https://github.com/tweag/codeql-wrapper">codeql-wrapper</a> and <a href="https://github.com/tweag/codeql-wrapper-pipelines">codeql-wrapper-pipelines</a>. The tools distill hard‑won lessons from monorepo rollouts: minimize manual configuration, keep behavior consistent across environments, and make smart defaults easy to override.</p>
<p>The goal isn’t just faster scans; it’s a smoother path to reliable, organization‑wide CodeQL usage. If you’re strengthening DevSecOps without piling on process, we think this approach strikes the right balance. Give the wrapper a try, explore the pipelines, and reach out on the repositories if you want help shaping them to your environment.</p></div>
    </summary>
    <updated>2026-01-15T00:00:00Z</updated>
    <published>2026-01-15T00:00:00Z</published>
    <source>
      <id>https://tweag.io</id>
      <author>
        <name>Tweag I/O</name>
      </author>
      <link href="https://tweag.io" rel="alternate" type="text/html"/>
      <link href="http://www.tweag.io/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Scale your engineering power. We enable deep-tech startups to achieve
their vision, from research to product delivery.</subtitle>
      <title>Tweag - Engineering blog</title>
      <updated>2026-04-02T12:24:07Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-us">
    <id>Buzzsprout-18487464</id>
    <link href="https://www.buzzsprout.com/1817535/episodes/18487464-75-kathrin-stark.mp3" length="36952643" rel="enclosure" type="audio/mpeg"/>
    <title>75: Kathrin Stark</title>
    <summary>We are joined by Kathrin Stark, a professor at Heriot-Watt University in Edinburgh. Kathrin works on program verification with proof assistants, so her focus is not exactly on Haskell, but on topics dear to Haskellers' hearts such as interactive theorem provers, writing correct programs, and the activities needed to produce them. We discuss many aspects of proofs and specifications, and the languages involved in the process, as well as verifying and producing provably correct neural networks.</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>We are joined by Kathrin Stark, a professor at Heriot-Watt University in Edinburgh. Kathrin works on program verification with proof assistants, so her focus is not exactly on Haskell, but on topics dear to Haskellers' hearts such as interactive theorem provers, writing correct programs, and the activities needed to produce them. We discuss many aspects of proofs and specifications, and the languages involved in the process, as well as verifying and producing provably correct neural networks.</p></div>
    </content>
    <updated>2026-01-11T13:00:00Z</updated>
    <published>2026-01-11T13:00:00Z</published>
    <author>
      <name>Haskell Podcast</name>
    </author>
    <source>
      <id>https://haskell.foundation/podcast/</id>
      <logo>https://storage.buzzsprout.com/tnk1ztokn5vmeiufqmr4kkp37mw2?.jpg</logo>
      <category scheme="http://www.itunes.com/" term="Technology"/>
      <author>
        <name>Haskell Podcast</name>
      </author>
      <link href="https://rss.buzzsprout.com/1817535.rss" rel="self" type="application/rss+xml"/>
      <link href="https://pubsubhubbub.appspot.com/" rel="hub" type="text/html"/>
      <link href="https://haskell.foundation/podcast/" rel="alternate" type="text/html"/>
      <rights>Â© 2026 The Haskell Interlude</rights>
      <subtitle>This is the Haskell Interlude, where the five co-hosts (Wouter Swierstra, Andres LÃ¶h, Alejandro Serrano, Niki Vazou, and Joachim Breitner) chat with Haskell guests!</subtitle>
      <title>The Haskell Interlude</title>
      <updated>2026-04-10T16:12:47Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://byorgey.github.io/blog/posts/2026/01/09/disco-live.html</id>
    <link href="http://byorgey.github.io/blog/posts/2026/01/09/disco-live.html" rel="alternate" type="text/html"/>
    <title>Disco Live!</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><h1>Disco Live!</h1>

<div class="info">
  Posted on January  9, 2026
  
  
  <br/>
  Tagged <a href="https://byorgey.github.io/tag/disco.html" rel="tag" title="All pages tagged 'disco'.">disco</a>, <a href="https://byorgey.github.io/tag/web.html" rel="tag" title="All pages tagged 'web'.">web</a>, <a href="https://byorgey.github.io/tag/WASM.html" rel="tag" title="All pages tagged 'WASM'.">WASM</a>, <a href="https://byorgey.github.io/tag/UI.html" rel="tag" title="All pages tagged 'UI'.">UI</a>, <a href="https://byorgey.github.io/tag/teaching.html" rel="tag" title="All pages tagged 'teaching'.">teaching</a>, <a href="https://byorgey.github.io/tag/discrete.html" rel="tag" title="All pages tagged 'discrete'.">discrete</a>, <a href="https://byorgey.github.io/tag/math.html" rel="tag" title="All pages tagged 'math'.">math</a>, <a href="https://byorgey.github.io/tag/Haskell.html" rel="tag" title="All pages tagged 'Haskell'.">Haskell</a>
  
</div>

<a href="https://share.joinmastodon.org/#text=Disco Live! https://byorgey.github.io/blog/posts/2026/01/09/disco-live.html" rel="noopener" target="_blank">Share on <svg fill="currentColor" height="1em" viewBox="0 0 74 79"><path d="M73.7014 17.4323C72.5616 9.05152 65.1774 2.4469 56.424 1.1671C54.9472 0.950843 49.3518 0.163818 36.3901 0.163818H36.2933C23.3281 0.163818 20.5465 0.950843 19.0697 1.1671C10.56 2.41145 2.78877 8.34604 0.903306 16.826C-0.00357854 21.0022 -0.100361 25.6322 0.068112 29.8793C0.308275 35.9699 0.354874 42.0498 0.91406 48.1156C1.30064 52.1448 1.97502 56.1419 2.93215 60.0769C4.72441 67.3445 11.9795 73.3925 19.0876 75.86C26.6979 78.4332 34.8821 78.8603 42.724 77.0937C43.5866 76.8952 44.4398 76.6647 45.2833 76.4024C47.1867 75.8033 49.4199 75.1332 51.0616 73.9562C51.0841 73.9397 51.1026 73.9184 51.1156 73.8938C51.1286 73.8693 51.1359 73.8421 51.1368 73.8144V67.9366C51.1364 67.9107 51.1302 67.8852 51.1186 67.862C51.1069 67.8388 51.0902 67.8184 51.0695 67.8025C51.0489 67.7865 51.0249 67.7753 50.9994 67.7696C50.9738 67.764 50.9473 67.7641 50.9218 67.7699C45.8976 68.9569 40.7491 69.5519 35.5836 69.5425C26.694 69.5425 24.3031 65.3699 23.6184 63.6327C23.0681 62.1314 22.7186 60.5654 22.5789 58.9744C22.5775 58.9477 22.5825 58.921 22.5934 58.8965C22.6043 58.8721 22.621 58.8505 22.6419 58.8336C22.6629 58.8167 22.6876 58.8049 22.714 58.7992C22.7404 58.7934 22.7678 58.794 22.794 58.8007C27.7345 59.9796 32.799 60.5746 37.8813 60.5733C39.1036 60.5733 40.3223 60.5733 41.5447 60.5414C46.6562 60.3996 52.0437 60.1408 57.0728 59.1694C57.1983 59.1446 57.3237 59.1233 57.4313 59.0914C65.3638 57.5847 72.9128 52.8555 73.6799 40.8799C73.7086 40.4084 73.7803 35.9415 73.7803 35.4523C73.7839 33.7896 74.3216 23.6576 73.7014 17.4323ZM61.4925 47.3144H53.1514V27.107C53.1514 22.8528 51.3591 20.6832 47.7136 20.6832C43.7061 20.6832 41.6988 23.2499 41.6988 28.3194V39.3803H33.4078V28.3194C33.4078 23.2499 31.3969 20.6832 27.3894 20.6832C23.7654 20.6832 21.9552 22.8528 21.9516 27.107V47.3144H13.6176V26.4937C13.6176 22.2395 14.7157 18.8598 16.9118 16.3545C19.1772 13.8552 22.1488 12.5719 25.8373 12.5719C30.1064 12.5719 33.3325 14.1955 35.4832 17.4394L37.5587 20.8853L39.6377 17.4394C41.7884 14.1955 45.0145 12.5719 49.2765 12.5719C52.9614 12.5719 55.9329 13.8552 58.2055 16.3545C60.4017 18.8574 61.4997 22.2371 61.4997 26.4937L61.4925 47.3144Z" fill="inherit"/></svg></a>
<section>
<p>A couple months ago, <a href="https://byorgey.github.io/blog/posts/2025/11/10/disco-web-ui.html">I asked for
help</a>
creating a web interface for
<a href="https://github.com/disco-lang/disco">Disco</a>, a
student-oriented programming language for teaching functional
programming and discrete mathematics. I am very happy to report that
<a href="https://apfelmus.nfshost.com/">Heinrich Apfelmus</a> responded to the
call, and this is the result:</p>
<p><a href="https://disco-lang.github.io/disco-live/">Disco Live!</a></p>
<p>It’s fairly bare bones at the moment, but it’s a fantastic place to
start, and far more than I would have been able to accomplish in a few
weeks. Give it a try, and let me know of any bugs you find! You are
also very welcome to contribute fixes, features, etc. on GitHub:</p>
<ul>
<li><a href="https://github.com/disco-lang/disco-live">The disco-live repo</a></li>
<li><a href="https://github.com/disco-lang/disco">The repo for disco itself</a></li>
</ul>
<p>If you want to know more about Disco, you can <a href="http://ozark.hendrix.edu/~yorgey/forest/yorgey-disco-2023/index.xml">read a paper about it
here</a>,
or <a href="https://disco-lang.readthedocs.io/en/latest/">read the language documentation</a>.</p>

</section>


<section id="isso-thread">
  &lt;noscript&gt;Javascript needs to be activated to view comments.&lt;/noscript&gt;
</section></div>
    </summary>
    <updated>2026-01-09T00:00:00Z</updated>
    <published>2026-01-09T00:00:00Z</published>
    <author>
      <name>Brent Yorgey</name>
    </author>
    <source>
      <id>http://byorgey.github.io/blog</id>
      <link href="http://byorgey.github.io/blog" rel="alternate" type="text/html"/>
      <link href="http://byorgey.github.io/blog/rss.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Brent Yorgey's academic blog</subtitle>
      <title>blog :: Brent -&gt; [String]</title>
      <updated>2026-01-09T00:00:00Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>http://www.well-typed.com/blog/2026/01/the-haskell-debugger-for-ghc-9</id>
    <link href="https://well-typed.com/blog/2026/01/the-haskell-debugger-for-ghc-9" rel="alternate" type="text/html"/>
    <title>The Haskell Debugger for GHC 9.14</title>
    <summary>The Haskell Debugger is ready to use with GHC-9.14!

The installation, configuration, and talks can be found in the official website. The tl;dr first step is installing the debugger:

``` bash
$ ghc --version # MUST BE GHC 9.14
The Glorious Glasgow Haskell Compilation System, version 9.14.1

$ cabal [...]</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><strong>The Haskell Debugger is ready to use with GHC-9.14!</strong></p>
<p>The installation, configuration, and talks can be found in <a href="https://well-typed.github.io/haskell-debugger">the official website</a>. The tl;dr first step is installing the debugger:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="https://well-typed.com/blog/rss2.xml#cb1-1" tabindex="-1"/><span class="ex">$</span> ghc <span class="at">--version</span> <span class="co"># MUST BE GHC 9.14</span></span>
<span id="cb1-2"><a href="https://well-typed.com/blog/rss2.xml#cb1-2" tabindex="-1"/><span class="ex">The</span> Glorious Glasgow Haskell Compilation System, version 9.14.1</span>
<span id="cb1-3"><a href="https://well-typed.com/blog/rss2.xml#cb1-3" tabindex="-1"/></span>
<span id="cb1-4"><a href="https://well-typed.com/blog/rss2.xml#cb1-4" tabindex="-1"/><span class="ex">$</span> cabal install haskell-debugger <span class="at">--allow-newer</span><span class="op">=</span>base,time,containers,ghc,ghc-bignum,template-haskell <span class="at">--enable-executable-dynamic</span> <span class="co"># ON WINDOWS, DO NOT PASS --enable-executable-dynamic</span></span>
<span id="cb1-5"><a href="https://well-typed.com/blog/rss2.xml#cb1-5" tabindex="-1"/><span class="ex">...</span></span>
<span id="cb1-6"><a href="https://well-typed.com/blog/rss2.xml#cb1-6" tabindex="-1"/></span>
<span id="cb1-7"><a href="https://well-typed.com/blog/rss2.xml#cb1-7" tabindex="-1"/><span class="ex">$</span> ~/.local/bin/hdb <span class="at">--version</span> <span class="co"># VERIFY IT'S THE LATEST!</span></span>
<span id="cb1-8"><a href="https://well-typed.com/blog/rss2.xml#cb1-8" tabindex="-1"/><span class="ex">Haskell</span> Debugger, version 0.11.0.0</span></code></pre></div>
<p>The second step is configuring your editor to use the debugger via the <strong>D</strong>ebug <strong>A</strong>dapter <strong>P</strong>rotocol (DAP).</p>
<ul>
<li>For VSCode, install <a href="https://marketplace.visualstudio.com/items?itemName=Well-Typed.haskell-debugger-extension">the haskell debugger extension</a>.</li>
<li>For Neovim, install <code>nvim-dap</code> and <a href="https://codeberg.org/mfussenegger/nvim-dap/wiki/Debug-Adapter-installation#haskell-hdb">configure it for haskell-debugger</a></li>
<li>For other editors, consult your DAP documentation and let others know how!</li>
</ul>
<p>Bug reports and discussions are welcome in the <a href="https://github.com/well-typed/haskell-debugger">haskell-debugger issue tracker</a>.</p>
<p>My MuniHac 2025 talk also walks through the installation, usage, and design of the debugger. Do note much has been improved since the talk was given, and much more will still improve.</p>
<div style="width: 560px; height: 315px; margin-bottom: 1em;">

</div>
<h4 id="a-little-bit-more-info">A little bit more info</h4>

<p>The debugger work is sponsored by Mercury. It’s the project in which I’ve spent most of my full working days (for almost a full year now), with the invaluable help from my team at Well-Typed.</p>
<p>The debugger is meant to work both on trivial files and on large and complex codebases.<a class="footnote-ref" href="https://well-typed.com/blog/rss2.xml#fn1" id="fnref1"><sup>1</sup></a> It is a GHC application so <em>all features are supported</em>. Like HLS, it also uses <code>hie-bios</code> to automatically configure the session based on your cabal or stack project.</p>
<p>Robustness is a main goal of the debugger. If anything doesn’t work, or if you have performance issues, or something crashes, please don’t hesitate to submit a bug. We’ve got a small but respectable testsuite, and have tested performance by debugging GHC itself, but there’s much still to be fixed and improved.</p>
<p><strong>Roadmap</strong>: There’s a lot to do. I’m currently working on callstacks and multi-threaded support. Do let me know what features would be most important to you, so I can also factor that into the future planning.</p>
<section class="footnotes footnotes-end-of-document" id="footnotes">
<hr/>
<ol>
<li id="fn1"><p>Although, for large codebases, the usability is still rough around the edges because of possibly long bytecode compilation times, and library code not being interpreted. We’ve made considerable progress to improve this with the <a href="https://discourse.haskell.org/t/rfc-introduce-a-serialisable-bytecode-format-and-corresponding-bytecode-way/12678">bytecode artifacts work</a>.<a class="footnote-back" href="https://well-typed.com/blog/rss2.xml#fnref1">↩︎</a></p></li>
</ol>
</section></div>
    </content>
    <updated>2026-01-08T00:00:00Z</updated>
    <published>2026-01-08T00:00:00Z</published>
    <category term="debugging"/>
    <category term="ghc"/>
    <category term="open-source"/>
    <category term="community"/>
    <author>
      <name>rodrigo</name>
    </author>
    <source>
      <id>https://well-typed.com/blog/</id>
      <logo>https://well-typed.com/img/wtnlogoplain.svg</logo>
      <link href="https://well-typed.com/blog/rss2.xml" rel="self" type="application/rss+xml"/>
      <link href="https://well-typed.com/blog/" rel="alternate" type="text/html"/>
      <subtitle>Because Well-Typed Programs Don't Go Wrong</subtitle>
      <title>Well-Typed Blog</title>
      <updated>2026-01-08T00:00:00Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>http://www.well-typed.com/blog/2026/01/haskell-debugger</id>
    <link href="https://well-typed.com/blog/2026/01/haskell-debugger" rel="alternate" type="text/html"/>
    <title>The Haskell Debugger for GHC 9.14</title>
    <summary>The Haskell Debugger is ready to use with GHC-9.14!

The installation, configuration, and talks can be found in the official website. The tl;dr first step is installing the debugger:

``` bash
$ ghc --version # MUST BE GHC 9.14
The Glorious Glasgow Haskell Compilation System, version 9.14.1

$ cabal [...]</summary>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><strong>The Haskell Debugger is ready to use with GHC-9.14!</strong></p>
<p>The installation, configuration, and talks can be found in <a href="https://well-typed.github.io/haskell-debugger">the official website</a>. The tl;dr first step is installing the debugger:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="https://well-typed.com/blog/rss2.xml#cb1-1" tabindex="-1"/><span class="ex">$</span> ghc <span class="at">--version</span> <span class="co"># MUST BE GHC 9.14</span></span>
<span id="cb1-2"><a href="https://well-typed.com/blog/rss2.xml#cb1-2" tabindex="-1"/><span class="ex">The</span> Glorious Glasgow Haskell Compilation System, version 9.14.1</span>
<span id="cb1-3"><a href="https://well-typed.com/blog/rss2.xml#cb1-3" tabindex="-1"/></span>
<span id="cb1-4"><a href="https://well-typed.com/blog/rss2.xml#cb1-4" tabindex="-1"/><span class="ex">$</span> cabal install haskell-debugger <span class="at">--allow-newer</span><span class="op">=</span>base,time,containers,ghc,ghc-bignum,template-haskell <span class="at">--enable-executable-dynamic</span> <span class="co"># ON WINDOWS, DO NOT PASS --enable-executable-dynamic</span></span>
<span id="cb1-5"><a href="https://well-typed.com/blog/rss2.xml#cb1-5" tabindex="-1"/><span class="ex">...</span></span>
<span id="cb1-6"><a href="https://well-typed.com/blog/rss2.xml#cb1-6" tabindex="-1"/></span>
<span id="cb1-7"><a href="https://well-typed.com/blog/rss2.xml#cb1-7" tabindex="-1"/><span class="ex">$</span> ~/.local/bin/hdb <span class="at">--version</span> <span class="co"># VERIFY IT'S THE LATEST!</span></span>
<span id="cb1-8"><a href="https://well-typed.com/blog/rss2.xml#cb1-8" tabindex="-1"/><span class="ex">Haskell</span> Debugger, version 0.11.0.0</span></code></pre></div>
<p>The second step is configuring your editor to use the debugger via the <strong>D</strong>ebug <strong>A</strong>dapter <strong>P</strong>rotocol (DAP).</p>
<ul>
<li>For VSCode, install <a href="https://marketplace.visualstudio.com/items?itemName=Well-Typed.haskell-debugger-extension">the haskell debugger extension</a>.</li>
<li>For Neovim, install <code>nvim-dap</code> and <a href="https://codeberg.org/mfussenegger/nvim-dap/wiki/Debug-Adapter-installation#haskell-hdb">configure it for haskell-debugger</a></li>
<li>For other editors, consult your DAP documentation and let others know how!</li>
</ul>
<p>Bug reports and discussions are welcome in the <a href="https://github.com/well-typed/haskell-debugger">haskell-debugger issue tracker</a>.</p>
<p>My MuniHac 2025 talk also walks through the installation, usage, and design of the debugger. Do note much has been improved since the talk was given, and much more will still improve.</p>
<div style="width: 560px; height: 315px; margin-bottom: 1em;">

</div>
<h4 id="a-little-bit-more-info">A little bit more info</h4>

<p>The debugger work is sponsored by Mercury. It’s the project in which I’ve spent most of my full working days (for almost a full year now), with the invaluable help from my team at Well-Typed.</p>
<p>The debugger is meant to work both on trivial files and on large and complex codebases.<a class="footnote-ref" href="https://well-typed.com/blog/rss2.xml#fn1" id="fnref1"><sup>1</sup></a> It is a GHC application so <em>all features are supported</em>. Like HLS, it also uses <code>hie-bios</code> to automatically configure the session based on your cabal or stack project.</p>
<p>Robustness is a main goal of the debugger. If anything doesn’t work, or if you have performance issues, or something crashes, please don’t hesitate to submit a bug. We’ve got a small but respectable testsuite, and have tested performance by debugging GHC itself, but there’s much still to be fixed and improved.</p>
<p><strong>Roadmap</strong>: There’s a lot to do. I’m currently working on callstacks and multi-threaded support. Do let me know what features would be most important to you, so I can also factor that into the future planning.</p>
<section class="footnotes footnotes-end-of-document" id="footnotes">
<hr/>
<ol>
<li id="fn1"><p>Although, for large codebases, the usability is still rough around the edges because of possibly long bytecode compilation times, and library code not being interpreted. We’ve made considerable progress to improve this with the <a href="https://discourse.haskell.org/t/rfc-introduce-a-serialisable-bytecode-format-and-corresponding-bytecode-way/12678">bytecode artifacts work</a>.<a class="footnote-back" href="https://well-typed.com/blog/rss2.xml#fnref1">↩︎</a></p></li>
</ol>
</section></div>
    </content>
    <updated>2026-01-08T00:00:00Z</updated>
    <published>2026-01-08T00:00:00Z</published>
    <category term="debugging"/>
    <category term="ghc"/>
    <category term="open-source"/>
    <category term="community"/>
    <author>
      <name>rodrigo</name>
    </author>
    <source>
      <id>https://well-typed.com/blog/</id>
      <logo>https://well-typed.com/img/wtnlogoplain.svg</logo>
      <link href="https://well-typed.com/blog/rss2.xml" rel="self" type="application/rss+xml"/>
      <link href="https://well-typed.com/blog/" rel="alternate" type="text/html"/>
      <subtitle>Because Well-Typed Programs Don't Go Wrong</subtitle>
      <title>Well-Typed Blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://magnus.therning.org/2026-01-04-validation-of-data-in-a-servant-server.html</id>
    <link href="https://magnus.therning.org/2026-01-04-validation-of-data-in-a-servant-server.html" rel="alternate" type="text/html"/>
    <title>Validation of data in a servant server</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>
I've been playing around with adding more validation of data received by an HTTP
endpoint in a <a href="https://hackage.haskell.org/package/servant">servant</a> server. Defining a type with a <code>FromJSON</code> instance is very
easy, just derive a <code>Generic</code> instance and it just works. Here's a simple
example
</p>

<div class="org-src-container">
<pre class="src src-haskell"><code><span class="org-keyword">data</span> Person = Person
    { name :: <span class="org-type">Text</span>
    , age :: <span class="org-type">Int</span>
    , occupation :: <span class="org-type">Occupation</span>
    }
    <span class="org-keyword">deriving</span> (Generic, Show)
    <span class="org-keyword">deriving</span> (FromJSON, ToJSON) via <span class="org-type">(Generically Person)</span>

<span class="org-keyword">data</span> Occupation = UnderAge | Student | Unemployed | SelfEmployed | Retired | Occupation <span class="org-type">Text</span>
    <span class="org-keyword">deriving</span> (Eq, Generic, Ord, Show)
    <span class="org-keyword">deriving</span> (FromJSON, ToJSON) via <span class="org-type">(Generically Occupation)</span>
</code></pre>
</div>

<p>
However, the validation is rather limited, basically it's just checking that
each field is present and of the correct type. For the type above I'd like to
enforce some constraints for the combination of <code>age</code> and <code>occupation</code>.
</p>

<p>
The steps I thought of are
</p>

<ol class="org-ol">
<li>Hide the default constructor and define a <a href="https://wiki.haskell.org/Smart_constructors">smart</a> one. (This is the standard
suggestion for placing extra constraints values.)</li>
<li>Manually define the <code>FromJSON</code> instance using the <code>Generic</code> instance to limit
the amount of code and the smart constructor.</li>
</ol>
<div class="outline-2" id="outline-container-org8b6637c">
<h2 id="org8b6637c">The smart constructor</h2>
<div class="outline-text-2" id="text-org8b6637c">
<p>
I give the constructor the result type <code>Either String Person</code> to make sure it
can both be usable in code and when defining <code>parseJSON</code>.
</p>

<div class="org-src-container">
<pre class="src src-haskell"><code><span class="org-function-name">mkPerson</span> :: <span class="org-type">Text</span> <span class="org-operator">-&gt;</span> <span class="org-type">Int</span> <span class="org-operator">-&gt;</span> <span class="org-type">Occupation</span> <span class="org-operator">-&gt;</span> <span class="org-type">Either String Person</span>
<span class="org-function-name">mkPerson</span> name age occupation = <span class="org-keyword">do</span>
    guardE mustBeUnderAge
    guardE notUnderAge
    guardE tooOldToBeStudent
    guardE mustBeRetired
    pure <span class="org-operator">$</span> Person name age occupation
  <span class="org-keyword">where</span>
    <span class="org-function-name">guardE</span> (pred, err) = when pred <span class="org-operator">$</span> Left err
    <span class="org-function-name">mustBeUnderAge</span> = (age <span class="org-operator">&lt;</span> <span class="org-number">8</span> <span class="org-operator">&amp;&amp;</span> occupation <span class="org-operator">&gt;</span> UnderAge, <span class="org-string">"too young for occupation"</span>)
    <span class="org-function-name">notUnderAge</span> = (age <span class="org-operator">&gt;</span> <span class="org-number">15</span> <span class="org-operator">&amp;&amp;</span> occupation <span class="org-operator">==</span> UnderAge, <span class="org-string">"too old to be under age"</span>)
    <span class="org-function-name">tooOldToBeStudent</span> = (age <span class="org-operator">&gt;</span> <span class="org-number">45</span> <span class="org-operator">&amp;&amp;</span> occupation <span class="org-operator">==</span> Student, <span class="org-string">"too old to be a student"</span>)
    <span class="org-function-name">mustBeRetired</span> = (age <span class="org-operator">&gt;</span> <span class="org-number">65</span> <span class="org-operator">&amp;&amp;</span> occupation <span class="org-operator">/=</span> Retired, <span class="org-string">"too old to not be retired"</span>)
</code></pre>
</div>

<p>
Here I'm making use of <code>Either e</code> being a <code>Monad</code> and use <code>when</code> to apply the
constraints and ensure the reason for failure is given to the caller.
</p>
</div>
</div>
<div class="outline-2" id="outline-container-org323ebdf">
<h2 id="org323ebdf">The <code>FromJSON</code> instance</h2>
<div class="outline-text-2" id="text-org323ebdf">
<p>
When defining the instance I take advantage of the <code>Generic</code> instance to make
the implementation short and simple.
</p>

<div class="org-src-container">
<pre class="src src-haskell"><code><span class="org-keyword">instance</span> FromJSON <span class="org-type">Person</span> <span class="org-keyword">where</span>
    <span class="org-function-name">parseJSON</span> v = <span class="org-keyword">do</span>
        Person{name, age, occupation} &lt;- genericParseJSON defaultOptions v
        either fail pure <span class="org-operator">$</span> mkPerson name age occupation
</code></pre>
</div>

<p>
If there are many more fields in the type I'd consider using <a href="https://downloads.haskell.org/ghc/latest/docs/users_guide/exts/record_wildcards.html"><code>RecordWildCards</code></a>.
</p>
</div>
</div>
<div class="outline-2" id="outline-container-orgb6bf602">
<h2 id="orgb6bf602">Conclusion</h2>
<div class="outline-text-2" id="text-orgb6bf602">
<p>
No, it's nothing ground-breaking but I think it's a fairly nice example of how
things can fit together in Haskell.
</p>
</div>
</div>
<div class="taglist"><a href="https://magnus.therning.org/tags.html">Tags</a>: <a href="https://magnus.therning.org/tag-haskell.html">haskell</a> <a href="https://magnus.therning.org/tag-servant.html">servant</a> </div></div>
    </summary>
    <updated>2026-01-04T09:58:00Z</updated>
    <published>2026-01-04T09:58:00Z</published>
    <category term="haskell"/>
    <category term="servant"/>
    <source>
      <id>https://magnus.therning.org/</id>
      <author>
        <name>Magnus Therning</name>
      </author>
      <link href="https://magnus.therning.org/" rel="alternate" type="text/html"/>
      <link href="https://magnus.therning.org/feed.xml" rel="self" type="application/rss+xml"/>
      <subtitle>Magnus web site</subtitle>
      <title>Magnus web site</title>
      <updated>2026-02-17T23:09:47Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://www.joachim-breitner.de/blog/818-Seemingly_impossible_programs_in_Lean</id>
    <link href="https://www.joachim-breitner.de/blog/818-Seemingly_impossible_programs_in_Lean" rel="alternate" type="text/html"/>
    <title>Seemingly impossible programs in Lean</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>In 2007, Martin Escardo wrote a often-read <a href="https://math.andrej.com/2007/09/28/seemingly-impossible-functional-programs/">blog post about “Seemingly impossible functional programs”</a>. One such seemingly impossible function is <code>find</code>, which takes a predicate on infinite sequences of bits, and returns an infinite sequence for which that predicate hold (unless the predicate is just always false, in which case it returns some arbitrary sequence).</p>
<p>Inspired by conversations with and experiments by Massin Guerdi at the dinner of <a href="https://leaning.in/2025/">LeaningIn 2025</a> in Berlin (yes, this blog post has been in my pipeline for far too long), I wanted to play around these concepts in Lean.</p>
<p>Let’s represent infinite sequences of bits as functions from <code>Nat</code> to <code>Bit</code>, and give them a nice name, and some basic functionality, including a binary operator for consing an element to the front:</p>
<pre class="lean"><code>import Mathlib.Data.Nat.Find

abbrev Bit := Bool

def Cantor : Type := Nat → Bit

def Cantor.head (a : Cantor) : Bit := a 0

def Cantor.tail (a : Cantor) : Cantor := fun i =&gt; a (i + 1)

@[simp, grind] def Cantor.cons (x : Bit) (a : Cantor) : Cantor
  | 0 =&gt; x
  | i+1 =&gt; a i

infix:60 " # " =&gt; Cantor.cons</code></pre>
<p>With this in place, we can write Escardo’s function in Lean. His blog post discusses a few variants; I’ll focus on just one of them:</p>
<pre class="lean"><code>mutual
  partial def forsome (p : Cantor → Bool) : Bool :=
    p (find p)

  partial def find (p : Cantor → Bool) : Cantor :=
    have b := forsome (fun a =&gt; p (true # a))
    (b # find (fun a =&gt; p (b # a)))
end</code></pre>
<p>We define <code>find</code> together with <code>forsome</code>, which checks if the predicate <code>p</code> holds for any sequence. Using that <code>find</code> sets the first element of the result to <code>true</code> if there exists a squence starting with <code>true</code>, else to <code>false</code>, and then tries to find the rest of the sequence.</p>
<p>It is a bit of a brian twiter that this code works, but it does:</p>
<pre class="lean"><code>def fifth_false : Cantor → Bool := fun a =&gt; not (a 5)

/-- info: [true, true, true, true, true, false, true, true, true, true] -/
#guard_msgs in
#eval List.ofFn (fun (i : Fin 10) =&gt; find fifth_false i)</code></pre>
<p>Of course, in Lean we don’t just want to define these functions, but we want to prove that they do what we expect them to do.</p>
<p>Above we defined them as <code>partial</code> functions, even though we hope that they are not actually partial: The <code>partial</code> keyword means that we don’t have to do a termination proof, but also that we cannot prove anything about these functions.</p>
<p>So can we convince Lean that these functions are total after all? We can, but it’s a bit of a puzzle, and we have to adjust the definitions.</p>
<p>First of all, these “seemingly impossible functions” are only possible because we assume that the predicate we pass to it, <code>p</code>, is computable and total. This is where the whole magic comes from, and I recommend to read Escardo’s blog posts and papers for more on this fascinating topic. In particular, you will learn that a predicate on <code>Cantor</code> that is computable and total necessarily only looks at some initial fragment of the sequence. The length of that prefix is called the “modulus”. So if we hope to prove termination of <code>find</code> and <code>forsome</code>, we have to restrict their argument <code>p</code> to only such computable predicates.</p>
<p>To that end I introduce <code>HasModulus</code> and the subtype of predicates on <code>Cantor</code> that have such a modulus:</p>
<pre class="lean"><code>-- Extensional (!) modulus of uniform continuity
def HasModulus (p : Cantor → α) := ∃ n, ∀ a b : Cantor, (∀ i &lt; n, a i = b i) → p a = p b

@[ext] structure CantorPred where
  pred : Cantor → Bool
  hasModulus : HasModulus pred</code></pre>
<p>The modulus of such a predicate is now the least prefix lenght that determines the predicate. In particular, if the modulus is zero, the predicate is constant:</p>
<pre class="lean"><code>namespace CantorPred

variable (p : CantorPred)

noncomputable def modulus : Nat :=
  open Classical in Nat.find p.hasModulus

theorem eq_of_modulus : ∀a b : Cantor, (∀ i &lt; p.modulus, a i = b i) → p a = p b := by
  open Classical in
  unfold modulus
  exact Nat.find_spec p.hasModulus

theorem eq_of_modulus_eq_0 (hm : p.modulus = 0) : ∀ a b, p a = p b := by
  intro a b
  apply p.eq_of_modulus
  simp [hm]</code></pre>
<p>Because we want to work with <code>CantorPred</code> and not <code>Cantor → Bool</code> I have to define some operations on that new type; in particular the “cons element before predicate” operation that we saw above in <code>find</code>:</p>
<pre class="lean"><code>def comp_cons (b : Bit) : CantorPred where
  pred := fun a =&gt; p (b # a)
  hasModulus := by
    obtain ⟨n, h_n⟩ := p.hasModulus
    cases n with
    | zero =&gt; exists 0; grind
    | succ m =&gt;
      exists m
      intro a b heq
      simp
      apply h_n
      intro i hi
      cases i
      · rfl
      · grind

@[simp, grind =] theorem comp_cons_pred (x : Bit) (a : Cantor) :
  (p.comp_cons x) a = p (x # a) := rfl</code></pre>
<p>For this operation we know that the modulus decreases (if it wasn’t already zero):</p>
<pre class="lean"><code>theorem comp_cons_modulus (x : Bit) :
    (p.comp_cons x).modulus ≤ p.modulus - 1 := by
  open Classical in
  apply Nat.find_le
  intro a b hab
  apply p.eq_of_modulus
  cases hh : p.modulus
  · simp
  · intro i hi
    cases i
    · grind
    · grind
grind_pattern comp_cons_modulus =&gt; (p.comp_cons x).modulus</code></pre>
<p>We can rewrite the <code>find</code> function above to use these operations:</p>
<pre class="lean"><code>mutual
  partial def forsome (p : CantorPred) : Bool := p (find p)

  partial def find (p : CantorPred) : Cantor := fun i =&gt;
    have b := forsome (p.comp_cons true)
    (b # find (p.comp_cons b)) i
end</code></pre>
<p>I have also eta-expanded the <code>Cantor</code> function returned by <code>find</code>; there is now a <code>fun i =&gt; … i</code> around the body. We’ll shortly see why that is needed.</p>
<p>Now have everything in place to attempt a termination proof. Before we do that proof, we could step back and try to come up with an informal termination argument.</p>
<ul>
<li><p>The recursive call from <code>forsome</code> to <code>find</code> doesn’t decrease any argument at all. This is ok if all calls from <code>find</code> to <code>forsome</code> are decreasing.</p></li>
<li><p>The recursive call from <code>find</code> to <code>find</code> decreases the index <code>i</code> as the recursive call is behind the <code>Cantor.cons</code> operation that shifts the index. Good.</p></li>
<li><p>The recursive call from <code>find</code> to <code>forsome</code> decreases the modulus of the argument <code>p</code>, if it wasn’t already zero.</p>
<p>But if was zero, it does not decrease it! But if it zero, then the call from <code>forsome</code> to <code>find</code> doesn’t actually need to call <code>find</code>, because then <code>p</code> doesn’t look at its argument.</p></li>
</ul>
<p>We can express all this reasoning as a termination measure in the form of a lexicographic triple. The <code>0</code> and <code>1</code> in the middle component mean that for zero modulus, we can call <code>forsome</code> from <code>find</code> “for free”.</p>
<pre class="lean"><code>mutual
  def forsome (p : CantorPred) : Bool := p (find p)
  termination_by (p.modulus, if p.modulus = 0 then 0 else 1, 0)
  decreasing_by grind

  def find (p : CantorPred) : Cantor := fun i =&gt;
    have b := forsome (p.comp_cons true)
    (b # find (p.comp_cons b)) i
  termination_by i =&gt; (p.modulus, if p.modulus = 0 then 1 else 0, i)
  decreasing_by all_goals grind
end</code></pre>
<p>The termination proof doesn’t go through just yet: Lean is not able to see that <code>(_ # p) i</code> will call <code>p</code> with <code>i - 1</code>, and it does not see that <code>p (find p)</code> only uses <code>find p</code> if the modulus of <code>p</code> is non-zero. We can use the <a href="https://lean-lang.org/doc/reference/latest/find/?domain=Verso.Genre.Manual.section&amp;name=well-founded-preprocessing"><code>wf_preprocess</code> feature</a> to tell it about that:</p>
<p>The following theorem replaces a call to <code>p f</code>, where <code>p</code> is a function parameter, with the slightly more complex but provably equivalent expression on the right, where the call to <code>f</code> is no in the <code>else</code> branch of an <code>if-then-else</code> and thus has <code>¬p.modulus = 0</code> in scope:</p>
<pre class="lean"><code>@[wf_preprocess]
theorem coe_wf (p : CantorPred) :
    (wfParam p) f = p (if _ : p.modulus = 0 then fun _ =&gt; false else f) := by
  split
  next h =&gt; apply p.eq_of_modulus_eq_0 h
  next =&gt; rfl</code></pre>
<p>And similarly we replace <code>(_ # p) i</code> with a variant that extend the context with information on how <code>p</code> is called:</p>
<pre class="lean"><code>def cantor_cons' (x : Bit) (i : Nat) (a : ∀ j, j + 1 = i → Bit) : Bit :=
  match i with
  | 0 =&gt; x
  | j + 1 =&gt; a j (by grind)

@[wf_preprocess] theorem cantor_cons_congr (b : Bit) (a : Cantor) (i : Nat) :
  (b # a) i = cantor_cons' b i (fun j _ =&gt; a j) := by cases i &lt;;&gt; rfl</code></pre>
<p>After these declarations, the above definition of <code>forsome</code> and <code>find</code> goes through!</p>
<p>It remains to now prove that they do what they should, by a simple induction on the modulus of <code>p</code>:</p>
<pre class="lean"><code>@[simp, grind =] theorem tail_cons_eq (a : Cantor) : (x # a).tail = a := by
  funext i; simp [Cantor.tail, Cantor.cons]

@[simp, grind =] theorem head_cons_tail_eq (a : Cantor) : a.head # a.tail = a := by
  funext i; cases i &lt;;&gt; rfl

theorem find_correct (p : CantorPred) (h_exists : ∃ a, p a) : p (find p) := by
  by_cases h0 : p.modulus = 0
  · obtain ⟨a, h_a⟩ := h_exists
    rw [← h_a]
    apply p.eq_of_modulus_eq_0 h0
  · rw [find.eq_unfold, forsome.eq_unfold]
    dsimp -zeta
    extract_lets b
    change p (_ # _)
    by_cases htrue : ∃ a, p (true # a)
    next =&gt;
      have := find_correct (p.comp_cons true) htrue
      grind
    next =&gt;
      have : b = false := by grind
      clear_value b; subst b
      have hfalse : ∃ a, p (false # a) := by
        obtain ⟨a, h_a⟩ := h_exists
        cases h : a.head
        · exists Cantor.tail a
          grind
        · exfalso
          apply htrue
          exists Cantor.tail a
          grind
      clear h_exists
      exact find_correct (p.comp_cons false) hfalse
termination_by p.modulus
decreasing_by all_goals grind

theorem forsome_correct (p : CantorPred) :
    forsome p ↔ (∃ a, p a) where
  mp hfind := by unfold forsome at hfind; exists find p
  mpr hex := by unfold forsome; exact find_correct p hex</code></pre>
<p>This is pretty nice! However there is more to do. For example, Escardo has a “massively faster” variant of <code>find</code> that we can implement as a partial function in Lean:</p>
<pre class="lean"><code>def findBit (p : Bit → Bool) : Bit :=
  if p false then false else true

def branch (x : Bit) (l r : Cantor) : Cantor :=
  fun n =&gt;
    if n = 0      then x
    else if 2 ∣ n then r ((n - 2) / 2)
                  else l ((n - 1) / 2)

mutual
  partial def forsome (p : Cantor -&gt; Bool) : Bool :=
    p (find p)

  partial def find (p : Cantor -&gt; Bool) : Cantor :=
    let x := findBit (fun x =&gt; forsome (fun l =&gt; forsome (fun r =&gt; p (branch x l r))))
    let l := find (fun l =&gt; forsome (fun r =&gt; p (branch x l r)))
    let r := find (fun r =&gt; p (branch x l r))
    branch x l r
end</code></pre>
<p>But can we get this past Lean’s termination checker? In order to prove that the modulus of <code>p</code> is decreasing, we’d have to know that, for example, <code>find (fun r =&gt; p (branch x l r))</code> is behaving nicely. Unforunately, it is rather hard to do termination proof for a function that relies on the behaviour of the function itself.</p>
<p>So I’ll leave this open as a future exercise.</p>
<p>I have dumped the code for this post at <a class="uri" href="https://github.com/nomeata/lean-cantor">https://github.com/nomeata/lean-cantor</a>.</p></div>
    </summary>
    <updated>2026-01-02T14:30:24Z</updated>
    <published>2026-01-02T14:30:24Z</published>
    <author>
      <name>Joachim Breitner</name>
      <email>mail@joachim-breitner.de</email>
    </author>
    <source>
      <id>https://www.joachim-breitner.de/blog</id>
      <logo>https://joachim-breitner.de/avatars/avatar_128.png</logo>
      <link href="https://www.joachim-breitner.de/blog" rel="alternate" type="text/html"/>
      <link href="https://www.joachim-breitner.de/blog_feed.rss" rel="self" type="application/rss+xml"/>
      <subtitle>Joachim Breitners Denkblogade</subtitle>
      <title>nomeataâ€™s mind shares</title>
      <updated>2026-03-25T18:01:51Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://reasonablypolymorphic.com/blog/more-algebraic-music/index.html</id>
    <link href="https://reasonablypolymorphic.com/blog/more-algebraic-music/index.html" rel="alternate" type="text/html"/>
    <title>An Algebraic Theory of Music</title>
    <content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>In my last post, I was <a href="https://reasonablypolymorphic.com/blog/algebraic-music/">struggling towards an algebraic theory of music.</a> This idea has been burning in my mind ever since, and I wanted to give some updates with where I’ve landed.</p>
<h2 id="differentiating-voices-from-music">Differentiating Voices from Music</h2>
<p>We begin by modeling a <a href="https://en.wikipedia.org/wiki/Part_(music)#Polyphony_and_part-writing">musical voice</a>, which is, roughly speaking, the abstract version of a human voice. The voice can be doing one thing at a time, or can choose to not be doing anything.</p>
<p>Voices are modeled by <a href="https://hackage.haskell.org/package/step-function">step functions</a>, which are divisions of the real line into discrete chunks. We interpret each discrete chunk as a note being played by the voice for the duration of the chunk.</p>
<p>This gives rise to a nice applicative structure that I alluded to in my previous post:</p>
<pre><code>liftA2 f
  |---- a ----|-- b --|
  |-- x --|---- y ----|
=
  |- fax -|fay|- fby -|</code></pre>
<p>where we take the union of the note boundaries in order to form the applicative. If either voice is resting, so too is the applicative. There is also an <code>Alternative</code> instance here, which chooses the first non-rest.</p>
<p>There is a similar monoidal structure here, where multiplication is given by “play these two things simultaneously,” relying on an underlying <code>Semigroup</code> instance for the meaning of “play these two things:”</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb2-1"><a href="https://reasonablypolymorphic.com#cb2-1" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Semigroup</span> a <span class="ot">=&gt;</span> <span class="dt">Semigroup</span> (<span class="dt">Voice</span> a)</span>
<span id="cb2-2"><a href="https://reasonablypolymorphic.com#cb2-2" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Semigroup</span> a <span class="ot">=&gt;</span> <span class="dt">Monoid</span> (<span class="dt">Voice</span> a)</span></code></pre></div>
<p>If either voice is resting, we treat its value as <code>mempty</code>, and can happily combine the two parts in parallel.</p>
<p>All of this gives rise to the following rich structure:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb3-1"><a href="https://reasonablypolymorphic.com#cb3-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">Voice</span> a <span class="ot">=</span> <span class="dt">Voice</span> {<span class="ot"> unVoice ::</span> <span class="dt">SF</span> <span class="dt">Time</span> (<span class="dt">Maybe</span> a) }</span>
<span id="cb3-2"><a href="https://reasonablypolymorphic.com#cb3-2" tabindex="-1"/>  <span class="kw">deriving</span> stock</span>
<span id="cb3-3"><a href="https://reasonablypolymorphic.com#cb3-3" tabindex="-1"/>    (<span class="dt">Eq</span>, <span class="dt">Ord</span>, <span class="dt">Show</span>, <span class="dt">Functor</span>, <span class="dt">Foldable</span>, <span class="dt">Traversable</span>, <span class="dt">Generic</span>, <span class="dt">Generic1</span>)</span>
<span id="cb3-4"><a href="https://reasonablypolymorphic.com#cb3-4" tabindex="-1"/>  <span class="kw">deriving</span> <span class="kw">newtype</span></span>
<span id="cb3-5"><a href="https://reasonablypolymorphic.com#cb3-5" tabindex="-1"/>    (<span class="dt">Semigroup</span>, <span class="dt">Monoid</span>)</span>
<span id="cb3-6"><a href="https://reasonablypolymorphic.com#cb3-6" tabindex="-1"/>  <span class="kw">deriving</span></span>
<span id="cb3-7"><a href="https://reasonablypolymorphic.com#cb3-7" tabindex="-1"/>    <span class="dt">Applicative</span></span>
<span id="cb3-8"><a href="https://reasonablypolymorphic.com#cb3-8" tabindex="-1"/>    via <span class="dt">Compose</span> (<span class="dt">SF</span> <span class="dt">Time</span>) <span class="dt">Maybe</span></span>
<span id="cb3-9"><a href="https://reasonablypolymorphic.com#cb3-9" tabindex="-1"/></span>
<span id="cb3-10"><a href="https://reasonablypolymorphic.com#cb3-10" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Filterable</span> <span class="dt">Voice</span></span>
<span id="cb3-11"><a href="https://reasonablypolymorphic.com#cb3-11" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Witherable</span> <span class="dt">Voice</span></span>
<span id="cb3-12"><a href="https://reasonablypolymorphic.com#cb3-12" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Alternative</span> <span class="dt">Voice</span></span>
<span id="cb3-13"><a href="https://reasonablypolymorphic.com#cb3-13" tabindex="-1"/></span>
<span id="cb3-14"><a href="https://reasonablypolymorphic.com#cb3-14" tabindex="-1"/><span class="co">-- | Delay a voice by some amount of time.</span></span>
<span id="cb3-15"><a href="https://reasonablypolymorphic.com#cb3-15" tabindex="-1"/><span class="ot">delayV ::</span> <span class="dt">Time</span> <span class="ot">-&gt;</span> <span class="dt">Voice</span> a <span class="ot">-&gt;</span> <span class="dt">Voice</span> a</span></code></pre></div>
<h2 id="from-voices-to-music">From Voices to Music</h2>
<p>Voices, therefore, give us our primitive notion of monophony. But real music usually has many voices doing many things, independently. This was a point in which I got stuck in my previous post.</p>
<p>The solution here, is surprisingly easy. Assign a <code>Voice</code> to each voice name:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb4-1"><a href="https://reasonablypolymorphic.com#cb4-1" tabindex="-1"/><span class="kw">newtype</span> <span class="dt">Music</span> v a <span class="ot">=</span> <span class="dt">Music</span></span>
<span id="cb4-2"><a href="https://reasonablypolymorphic.com#cb4-2" tabindex="-1"/>  {<span class="ot"> getVoices ::</span> v <span class="ot">-&gt;</span> <span class="dt">Voice</span> a</span>
<span id="cb4-3"><a href="https://reasonablypolymorphic.com#cb4-3" tabindex="-1"/>  }</span>
<span id="cb4-4"><a href="https://reasonablypolymorphic.com#cb4-4" tabindex="-1"/>  <span class="kw">deriving</span> stock</span>
<span id="cb4-5"><a href="https://reasonablypolymorphic.com#cb4-5" tabindex="-1"/>    (<span class="dt">Functor</span>, <span class="dt">Generic</span>, <span class="dt">Functor</span>)</span>
<span id="cb4-6"><a href="https://reasonablypolymorphic.com#cb4-6" tabindex="-1"/>  <span class="kw">deriving</span> <span class="kw">newtype</span></span>
<span id="cb4-7"><a href="https://reasonablypolymorphic.com#cb4-7" tabindex="-1"/>    (<span class="dt">Semigroup</span>, <span class="dt">Monoid</span>)</span>
<span id="cb4-8"><a href="https://reasonablypolymorphic.com#cb4-8" tabindex="-1"/>  <span class="kw">deriving</span> (<span class="dt">Applicative</span>, <span class="dt">Alternative</span>)</span>
<span id="cb4-9"><a href="https://reasonablypolymorphic.com#cb4-9" tabindex="-1"/>    via <span class="dt">Compose</span> ((<span class="ot">-&gt;</span>) v) <span class="dt">Voice</span></span>
<span id="cb4-10"><a href="https://reasonablypolymorphic.com#cb4-10" tabindex="-1"/></span>
<span id="cb4-11"><a href="https://reasonablypolymorphic.com#cb4-11" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Profunctor</span> <span class="dt">Music</span></span>
<span id="cb4-12"><a href="https://reasonablypolymorphic.com#cb4-12" tabindex="-1"/></span>
<span id="cb4-13"><a href="https://reasonablypolymorphic.com#cb4-13" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Finite</span> v <span class="ot">=&gt;</span> <span class="dt">Foldable</span> (<span class="dt">Music</span> v)</span>
<span id="cb4-14"><a href="https://reasonablypolymorphic.com#cb4-14" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Finite</span> v <span class="ot">=&gt;</span> <span class="dt">Traversable</span> (<span class="dt">Music</span> v)</span>
<span id="cb4-15"><a href="https://reasonablypolymorphic.com#cb4-15" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Filterable</span> (<span class="dt">Music</span> v)</span>
<span id="cb4-16"><a href="https://reasonablypolymorphic.com#cb4-16" tabindex="-1"/><span class="kw">instance</span> <span class="dt">Finite</span> v <span class="ot">=&gt;</span> <span class="dt">Witherable</span> (<span class="dt">Music</span> v)</span></code></pre></div>
<p>We get an extremely rich structure here completely for free. Our monoid combines all voices in parallel; our applicative combines voices pointwise; etc. However, we also have a new <code>Profunctor Music</code> instance, whose characteristic <code>lmap :: (b -&gt; c) -&gt; Music c a -&gt; Music b a</code> method allows us to trade lines between voices.</p>
<p>In addition to the <em>in-parallel</em> monoid instance, we can also define a <a href="https://dl.acm.org/doi/10.1145/2633638.2633649">tile product</a> operator over <code>Music v a</code>, which composes things sequentially<a class="footnote-ref" href="https://reasonablypolymorphic.com#fn1" id="fnref1"><sup>1</sup></a>:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb5-1"><a href="https://reasonablypolymorphic.com#cb5-1" tabindex="-1"/><span class="ot">duration ::</span> <span class="dt">Music</span> v a <span class="ot">-&gt;</span> <span class="dt">Time</span></span>
<span id="cb5-2"><a href="https://reasonablypolymorphic.com#cb5-2" tabindex="-1"/></span>
<span id="cb5-3"><a href="https://reasonablypolymorphic.com#cb5-3" tabindex="-1"/><span class="ot">(##) ::</span> <span class="dt">Semigroup</span> a <span class="ot">=&gt;</span> <span class="dt">Music</span> v a <span class="ot">-&gt;</span> <span class="dt">Music</span> v a <span class="ot">-&gt;</span> <span class="dt">Music</span> v a</span>
<span id="cb5-4"><a href="https://reasonablypolymorphic.com#cb5-4" tabindex="-1"/>m<span class="op">@</span>(<span class="dt">Music</span> m1) <span class="op">##</span> <span class="dt">Music</span> m2 <span class="ot">=</span></span>
<span id="cb5-5"><a href="https://reasonablypolymorphic.com#cb5-5" tabindex="-1"/>    <span class="dt">Music</span> <span class="op">$</span> liftA2 (<span class="op">&lt;&gt;</span>) m1 <span class="op">$</span> <span class="fu">fmap</span> (delayV <span class="op">$</span> duration m) m2</span></code></pre></div>
<p>The <code>Semigroup a</code> constraint on <code>(##)</code> arises from the fact that the pieces of music might extend off to infinity in either direction (which <code>pure @Voice</code> must do), and we need to deal with that.</p>
<p>There are a few other combinators we care about. First, we can lift anonymous voices (colloquially “tunes”) into multi-part <code>Music</code>:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb6-1"><a href="https://reasonablypolymorphic.com#cb6-1" tabindex="-1"/><span class="ot">voice ::</span> <span class="dt">Eq</span> v <span class="ot">=&gt;</span> v <span class="ot">-&gt;</span> <span class="dt">Music</span> () a <span class="ot">-&gt;</span> <span class="dt">Music</span> v a</span>
<span id="cb6-2"><a href="https://reasonablypolymorphic.com#cb6-2" tabindex="-1"/>voice v (<span class="dt">Music</span> sf) <span class="ot">=</span> <span class="dt">Music</span> <span class="op">$</span></span>
<span id="cb6-3"><a href="https://reasonablypolymorphic.com#cb6-3" tabindex="-1"/>  \<span class="kw">case</span></span>
<span id="cb6-4"><a href="https://reasonablypolymorphic.com#cb6-4" tabindex="-1"/>    ((<span class="op">==</span> v) <span class="ot">-&gt;</span> <span class="dt">True</span>) <span class="ot">-&gt;</span> sf ()</span>
<span id="cb6-5"><a href="https://reasonablypolymorphic.com#cb6-5" tabindex="-1"/>    _ <span class="ot">-&gt;</span> <span class="fu">mempty</span></span></code></pre></div>
<p>and we can assign the same line to everyone:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb7-1"><a href="https://reasonablypolymorphic.com#cb7-1" tabindex="-1"/><span class="ot">everyone ::</span> <span class="dt">Music</span> () a <span class="ot">-&gt;</span> <span class="dt">Music</span> v a</span>
<span id="cb7-2"><a href="https://reasonablypolymorphic.com#cb7-2" tabindex="-1"/>everyone (<span class="dt">Music</span> m) <span class="ot">=</span> <span class="dt">Music</span> <span class="op">$</span> <span class="fu">const</span> <span class="op">$</span> m ()</span></code></pre></div>
<h2 id="writing-lines">Writing Lines</h2>
<p>The primitives for building little tunes are</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb8-1"><a href="https://reasonablypolymorphic.com#cb8-1" tabindex="-1"/><span class="ot">note ::</span> <span class="dt">Time</span> <span class="ot">-&gt;</span> a <span class="ot">-&gt;</span> <span class="dt">Music</span> () a</span>
<span id="cb8-2"><a href="https://reasonablypolymorphic.com#cb8-2" tabindex="-1"/><span class="ot">rest ::</span> <span class="dt">Time</span> <span class="ot">-&gt;</span> <span class="dt">Music</span> () a</span></code></pre></div>
<p>which you can then compose sequentially via <code>(##)</code>, and assign to voices via <code>voice</code>.</p>
<h2 id="harmonic-constraints">Harmonic Constraints</h2>
<p>One of the better responses to my last blog post was a link to <a href="https://dmitri.mycpanel.princeton.edu/index.html">Dmitri Tymoczko</a>’s <a href="https://www.youtube.com/watch?v=yp5Eys2L_04">FARM 2024 talk</a>.</p>
<p>There’s much more in this video than I can possibly due justice to here, but my big takeaway was that this guy is thinking about the same sorts of things that I am. So I dove into his work, and that lead to his <a href="https://www.madmusicalscience.com/quadruple.mp4">quadruple hierarchy</a>:</p>
<blockquote>
<p>Voices move within chords, which move within scales, which move within macro-harmonies.</p>
</blockquote>
<p>Tymoczko presents a <code>T</code> algebra which is a geometric space for reasoning about voice leadings. He’s got a lot of fun websites for exploring the ideas, but I couldn’t find an actual implementation of the idea anywhere, so I <a href="https://github.com/isovector/denomusic/blob/6a313a546cc376bb22f37cec55d76518bff40acd/src/DenoMusic/Harmony.hs">cooked one up</a> myself.</p>
<p>The idea here is that we have some <code>T :: [Nat] -&gt; Type</code> which describes a hierarchy of abstract scales moving with respect to one another. For example, the Western traditional of having triads move within the diatonic scale, which moves within the chromatic scale, would be represented as <code>T '[3, 7, 12]</code>. <code>T</code> forms a monoid, and has some simple generators that give rise to smooth voice leadings (chord changes.)</p>
<p>Having a model for smooth harmonic transformations means we can use it constructively. I am still working out the exact details here, but the rough shape of the idea is to build an underlying field of key changes (represented as smooth voice leadings in <code>T '[7, 12]</code>):</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb9-1"><a href="https://reasonablypolymorphic.com#cb9-1" tabindex="-1"/><span class="ot">keyChanges ::</span> <span class="dt">Music</span> () (<span class="dt">T</span> '[<span class="dv">7</span>, <span class="dv">12</span>])</span></code></pre></div>
<p>We can then make an underlying field of functional harmonic changes (chord changes), modeled as smooth voice leadings in <code>T '[3, 7]</code>:</p>
<div class="sourceCode" id="cb10"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb10-1"><a href="https://reasonablypolymorphic.com#cb10-1" tabindex="-1"/><span class="ot">chordChanges ::</span> <span class="dt">Music</span> () (<span class="dt">T</span> '[<span class="dv">3</span>, <span class="dv">7</span>])</span></code></pre></div>
<p>Our voices responsible for harmony can now be written as values of type</p>
<div class="sourceCode" id="cb11"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb11-1"><a href="https://reasonablypolymorphic.com#cb11-1" tabindex="-1"/><span class="ot">harmonicVoices ::</span> <span class="dt">Music</span> h (<span class="dt">Set</span> (<span class="dt">T</span> '[<span class="dv">3</span>]))</span></code></pre></div>
<p>and we can use the applicative musical structure to combine the elements together:</p>
<div class="sourceCode" id="cb12"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb12-1"><a href="https://reasonablypolymorphic.com#cb12-1" tabindex="-1"/><span class="ot">{-# LANGUAGE ApplicativeDo #-}</span></span>
<span id="cb12-2"><a href="https://reasonablypolymorphic.com#cb12-2" tabindex="-1"/></span>
<span id="cb12-3"><a href="https://reasonablypolymorphic.com#cb12-3" tabindex="-1"/><span class="ot">extend ::</span> <span class="dt">T</span> ns <span class="ot">-&gt;</span> <span class="dt">T</span> (ns <span class="op">++</span> ms)</span>
<span id="cb12-4"><a href="https://reasonablypolymorphic.com#cb12-4" tabindex="-1"/><span class="ot">sink   ::</span> <span class="dt">T</span> ns <span class="ot">-&gt;</span> <span class="dt">T</span> (n '<span class="op">:</span> ns)</span>
<span id="cb12-5"><a href="https://reasonablypolymorphic.com#cb12-5" tabindex="-1"/></span>
<span id="cb12-6"><a href="https://reasonablypolymorphic.com#cb12-6" tabindex="-1"/><span class="ot">harmony ::</span> <span class="dt">Music</span> h (<span class="dt">Set</span> (<span class="dt">T</span> '[<span class="dv">3</span>, <span class="dv">7</span>, <span class="dv">12</span>]))</span>
<span id="cb12-7"><a href="https://reasonablypolymorphic.com#cb12-7" tabindex="-1"/>harmony <span class="ot">=</span> <span class="kw">do</span></span>
<span id="cb12-8"><a href="https://reasonablypolymorphic.com#cb12-8" tabindex="-1"/>  k <span class="ot">&lt;-</span> everyone keyChanges</span>
<span id="cb12-9"><a href="https://reasonablypolymorphic.com#cb12-9" tabindex="-1"/>  c <span class="ot">&lt;-</span> everyone chordChanges</span>
<span id="cb12-10"><a href="https://reasonablypolymorphic.com#cb12-10" tabindex="-1"/>  m <span class="ot">&lt;-</span> harmonicVoices</span>
<span id="cb12-11"><a href="https://reasonablypolymorphic.com#cb12-11" tabindex="-1"/>  <span class="fu">pure</span> <span class="op">$</span> extend m <span class="op">&lt;&gt;</span> extend c <span class="op">&lt;&gt;</span> sink k</span></code></pre></div>
<p>which we can later <code>fmap</code> out into concrete pitches. The result is that we can completely isolate the following pieces:</p>
<ul>
<li>key changes</li>
<li>chord changes</li>
<li>how voices express the current harmony</li>
<li>the rhythms of all of the above</li>
</ul>
<p>and the result is guaranteed to compose in a way that the ear can interpret as music. Not necessarily <em>good music,</em> but undeniably as <em>music.</em></p>
<p>The type indices on <code>T</code> are purely for my book-keeping, and nothing requires them to be there. Which means we could also use the applicative structure to modulate over different sorts of harmony (eg, move from triads to seventh chords.)</p>
<h2 id="melody-still-an-open-question">Melody: Still an Open Question</h2>
<p>I haven’t quite gotten a feel for melody yet; I think it’s probably in <code>T '[7, 12]</code>, but it would be nice to be able to target chord tones as well. Please let me know in the comments if you have any insight here.</p>
<p>However, I have been thinking about contouring, which is the overall “shape” of a musical line. Does it go up, and peak in the middle, and then come down again? Or maybe it smoothly descends down.</p>
<p>We can use the discrete intervals intrinsic inside of <code>Voice</code>s to find “reasonable” times to sample them. In essence this assigns a <code>Time</code> to each segment:</p>
<div class="sourceCode" id="cb13"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb13-1"><a href="https://reasonablypolymorphic.com#cb13-1" tabindex="-1"/><span class="ot">timed ::</span> <span class="dt">Music</span> v a <span class="ot">-&gt;</span> <span class="dt">Music</span> v (<span class="dt">Time</span>, a)</span></code></pre></div>
<p>and we can then use these times to then sample a function <code>Time -&gt; b</code>. This then allows us to apply contours (given as regular <code>Real -&gt; Real</code> functions) to arbitrary rhythms. I currently have this typed as</p>
<div class="sourceCode" id="cb14"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb14-1"><a href="https://reasonablypolymorphic.com#cb14-1" tabindex="-1"/><span class="ot">contour ::</span> <span class="dt">Group</span> a <span class="ot">=&gt;</span> a <span class="ot">-&gt;</span> (<span class="dt">Real</span> <span class="ot">-&gt;</span> <span class="dt">Real</span>) <span class="ot">-&gt;</span> <span class="dt">Music</span> v () <span class="ot">-&gt;</span> <span class="dt">Music</span> v a</span></code></pre></div>
<p>where <code>a ~ T something</code>, and the outputted <code>Real</code>s get rounded to their nearest integer values. I’m not deeply in love with this type, but the rough idea is great—turn arbitrary real-valued functions into musical lines. This generalizes contouring, as well as scale runs.</p>
<h2 id="next-steps">Next Steps?</h2>
<p>I’m writing all of this up because I go back to work on Monday and life is going to get very busy soon. I’m afraid I won’t be able to finish all of this!</p>
<p>The types above I’m pretty certain are relatively close to perfect. They seem to capture everything I could possibly want, and nothing I don’t want. Assuming I’m right about that, they must make up the basis of musical composition.</p>
<p>The next step therefore is to build musical combinators on top. One particular combinator I’ve got my eye on is some sort of general <code>~&gt;</code> “get from here to there” operator:</p>
<div class="sourceCode" id="cb15"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb15-1"><a href="https://reasonablypolymorphic.com#cb15-1" tabindex="-1"/><span class="ot">(~&gt;) ::</span> (<span class="dt">Semigroup</span> a, <span class="op">???</span>) <span class="ot">=&gt;</span> <span class="dt">Music</span> v a <span class="ot">-&gt;</span> <span class="dt">Music</span> v a <span class="ot">-&gt;</span> <span class="dt">Music</span> v a</span></code></pre></div>
<p>which I imagine would bridge a gap between the end of one piece of music with beginning of another. I think this would be roughly as easy as moving each voice linearly in <code>T</code> space from where it was to where it needs to be. This might need to be a ternary operation in order to also associate a rhythmic pattern to use for the bridge.</p>
<p>But I imagine <code>(~&gt;)</code> would be great for lots of dumb little musical things. Like when applied over the chord dimension, it would generate arpeggios. Over the scale dimension, it would generate runs. And it would make chromatic moves in the chroma dimension.</p>
<p>Choosing exactly what moves to make for <code>T</code>s consisting of components in multiple axes might just be some bespoke order, or could do something more intelligent. I think the right approach would be to steal <code>diagrams</code>’ idea of an <code>Envelope</code>, and attach some relevant metadata to each <code>Music</code>. We could then write <code>(~&gt;)</code> as a function of those envelopes, but I must admit I don’t quite know what this would look like.</p>
<p>As usual, I’d love any insight you have! Please leave it in the comments. Although I must admit I appreciate comments of the form “have you tried $X” much more than of the form “music is sublime and you’re an idiot for even trying this.”</p>
<p>Happy new year!</p>
<section class="footnotes">
<hr/>
<ol>
<li id="fn1"><p>Strictly speaking, the tile product can also do parallel composition, as well as sychronizing composition, but that’s not super important right now.<a class="footnote-back" href="https://reasonablypolymorphic.com#fnref1">↩︎</a></p></li>
</ol>
</section></div>
    </content>
    <updated>2026-01-02T14:27:00Z</updated>
    <published>2026-01-02T14:27:00Z</published>
    <source>
      <id>http://reasonablypolymorphic.com</id>
      <author>
        <name>Sandy Maguire</name>
      </author>
      <link href="https://reasonablypolymorphic.com" rel="self" type="application/atom+xml"/>
      <title>Reasonably Polymorphic</title>
      <updated>2026-01-02T14:27:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://conway.rutgers.edu/~ccshan/wiki/blog/posts/lambda4ada/</id>
    <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/posts/lambda4ada/" rel="alternate" type="text/html"/>
    <title>A challenge for a better community</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><a href="https://supportada.org/?campaign=lambda"><img alt="Donation button" height="54" src="https://adainitiative.org/wp-content/uploads/2014/08/donate_red_small.png" width="185"/><br/>
<img alt="Donate to the Ada Initiative" src="https://adainitiative.org/counters/2014counter-lambda.svg" width="400"/></a></p>
<p>Did you know that all ACM-sponsored conferences have an <a href="http://www.acm.org/sigs/volunteer_resources/officers_manual/anti-harassment-policy">
anti-harassment policy</a>? I didn’t, until I chaired the Haskell
Symposium last year. The concise policy says, among other things,
that people shouldn’t use my family constitution to interfere with
my professional participation. And the policy has teeth. That’s
great.</p>
<p>My not knowing the policy and not seeing it publicized didn’t
make me go out of my way to harass anyone. But it did make me less
sure and less comfortable that I belonged at ICFP. Briefly, it’s
because I didn’t know if it would be common ground at the
conference that my actual self was fully human. That’s not
something I can take for granted in general society. Also, it’s
because I didn’t know whether my fellow conference attendees were
aware of the policy. We could all use a reminder, and a public
signal that we mean it.</p>
<p>For these reasons, I’m very happy that ICFP will start to
publicize ACM’s existing anti-harassment policy and make sure
everyone registered knows it. All ACM conferences should do it.
That’s why <a href="http://tim.dreamwidth.org/1856739.html">Tim
Chevalier</a>, <a href="http://blog.clement.delafargue.name/posts/2014-09-13-of-conferences-and-cocs.html">
Clement Delafargue</a>, <a href="http://tim.dreamwidth.org/1856828.html">Adam Foltzer</a>, Eric
Merritt, and I are doing two things. We ask you to join us:</p>
<ol>
<li>Donate to the Ada Initiative. Our goal is for the functional
programming community to raise $4096 by the end of Friday (Sept 19)
UTC. To count toward this goal, please use this link: <a href="http://supportada.org/?campaign=lambda">http://supportada.org/?campaign=lambda</a></li>
<li>Call on the ACM and tell your friends. For example, I tweeted
this:<br/>
I donate to <a href="http://twitter.com/AdaInitiative">@AdaInitiative</a> because I
want <a href="http://twitter.com/TheOfficialACM">@TheOfficialACM</a> events to
announce their anti-harassment policy <a href="http://supportada.org/?campaign=lambda">http://supportada.org/?campaign=lambda</a>
<a href="http://twitter.com/hashtag/lambda4ada?src=hash">#lambda4ada</a></li>
</ol>
<p>Thanks for improving our professional homes!</p></div>
    </summary>
    <updated>2026-01-01T07:11:17Z</updated>
    <published>2014-09-16T16:09:11Z</published>
    <category term="english"/>
    <source>
      <id>http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/</id>
      <author>
        <name>Chung-chieh Shan</name>
      </author>
      <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/" rel="alternate" type="text/html"/>
      <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/index.rss" rel="self" type="application/rss+xml"/>
      <subtitle>Chung-chieh Shan's blog</subtitle>
      <title>Proper Treatment</title>
      <updated>2026-01-01T08:01:49Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://conway.rutgers.edu/~ccshan/wiki/blog/posts/Borderline/</id>
    <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/posts/Borderline/" rel="alternate" type="text/html"/>
    <title>Borderline, just semantics</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p><a href="http://hyperbolic-crochet.blogspot.com/2012/08/bill-thurston-1946-2012.html">
<img align="right" src="http://1.bp.blogspot.com/-kRjyJlu7hdo/UDT7DCy9Q3I/AAAAAAAACeE/j485hiluBF8/s1600/fig8.jpg" width="40%"/></a></p>
<p>I want to discuss two moral aspects of how to talk about
borderline people (e.g., whether to say “MTF” or “female”, “smart”
or “good at math”).</p>
<p>The first aspect is what category of people we should be talking
about, never mind how we talk about it. Usually, for a given
category C, there are many ways to be borderline C: people can
be</p>
<ul>
<li>borderline female due to a lack of chromosomes or a lack of
genitals;</li>
<li>borderline smart because they keep exchanging quantifiers or
because they keep estranging family;</li>
<li>borderline safe because they drive over the speed limit or roll
their bike past a stop sign;</li>
<li>borderline documented because they burned their passport,
overstayed their visa, or came from an unrecognized “country”.</li>
</ul>
<p>Of course, it is easy enough to construct a category C’ that
firmly includes or excludes particular kinds of borderline Cs
(e.g., you are female’ iff you have XX chromosomes), if only to
leave unresolved lower-dimensional borderlines (e.g., what if only
half of your cells have XX chromosomes). So let’s ask not what
categories exist (because they all do) but what categories we
should talk about.</p>
<p>It might well be that we should talk about different versions of
C for different purposes (e.g., medicine, sports, identification,
love). It might also be that delineating categories at their edges
is not usually worth our collective cognitive trouble. Regardless,
sometimes in conversation or other collaboration I find that
multiple versions are actually relevant. For example, some people
wonder whether to call a transgender person “male” or “female”, and
I think a transgender person can <em>firmly</em> belong to a gender
category that we should talk about. Unfortunately, when I tried to
work with others to distinguish and choose among versions of
categories, I have often been frustrated.</p>
<p>When people in power to classify others don’t take the effort to
apply the right category version, the misclassified people can get
slighted. I empathize with the misclassified people because I have
been misclassified. For example:</p>
<ul>
<li>I don’t need a passport to travel internationally, though I do
need a passport number.</li>
<li>I am a computer scientist, though I can’t fix your
computer.</li>
<li>I am from Taiwan, though I don’t speak Taiwanese.</li>
<li>I am from the United States, though I’m not a United States
citizen.</li>
</ul>
<p>Also because I have been misclassified, this discussion of
borderlines matters for me practically and personally. It’s not
just abstract theorizing.</p>
<p><img alt="not-passport.jpg" src="http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/../../posts/Borderline/not-passport.jpg"/></p>
<p>The second aspect is how to name a given category (version).
Even in a context where we agree intellectually that there is
exactly one gender category we should talk about (say “female”) and
a transgender person belongs to it firmly, the term “female” still
takes many people today more thought to produce and comprehend when
applied to a person known to be transgender than when applied to a
person known to be cisgender. So, it is tempting for a well-meaning
speaker to avoid gender categories altogether by terming a
transgender person “female-presenting” instead. The
“female-presenting” characterization</p>
<ul>
<li>is certainly true,</li>
<li>but may be a cop-out because it
<ul>
<li>perpetuates the implicature that the person doesn’t quite
belong</li>
<li>and increases their work to counter misclassification,</li>
</ul>
</li>
<li>yet may be justifiable (perhaps in some strain of
utilitarianism), though a justification remains to be spelled out
and applied uniformly to all instances.</li>
</ul>
<p>Your thoughts?</p></div>
    </summary>
    <updated>2026-01-01T07:11:17Z</updated>
    <published>2012-11-29T02:48:28Z</published>
    <category term="english"/>
    <source>
      <id>http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/</id>
      <author>
        <name>Chung-chieh Shan</name>
      </author>
      <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/" rel="alternate" type="text/html"/>
      <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/index.rss" rel="self" type="application/rss+xml"/>
      <subtitle>Chung-chieh Shan's blog</subtitle>
      <title>Proper Treatment</title>
      <updated>2026-01-01T08:01:50Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://conway.rutgers.edu/~ccshan/wiki/blog/posts/Babel-17/</id>
    <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/posts/Babel-17/" rel="alternate" type="text/html"/>
    <title>Babel-17</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>“Show me.”</p>
<p>He bowed his head in mocking, semi-formal acquiescence. “Modern
warfare can be fought on so many delightfully different levels,” he
continued, walking back to her side as if no interruption of the
tour had been suggested. “One wins a battle by making sure one’s
troops have enough blunderbusses and battle axes like the ones you
saw in the first room; or by the well-placed six-inch length of
vanadium wire in a Type 27-QX communications unit. With the proper
orders delayed, the encounter never takes place. Hand-to-hand
weapons, survival kit, plus training, room, and board: three
thousand credits per enlisted stellarman over a period of two years
active duty. For a garrison of fifteen hundred men that’s an outlay
of four million, five hundred credits. That same garrison will live
in and fight from three hyperstasis battleships which, fully
equipped, run about a million and a half credits apiece—a total
outlay of nine million credits. We have spent, on occasion, perhaps
as much as a million on the preparation of a single spy or
saboteur. That is rather higher than usual. And I can’t believe a
six-inch length of vanadium wire costs a third of a cent. War is
costly. And although it has taken some time, Administrative
Alliance Headquarters is beginning to realize subtlety pays. This
way, Miss—Captain Wong.”</p>
<p>Again they were in a room with only a single display case, but
it was seven feet high.</p>
<p>A statue, Rydra thought. No, real flesh, with detail of muscle
and joint; no, it must be a statue because a human body, dead or in
suspended animation, doesn’t look that… alive. Only art could
produce that vibrancy.</p>
<p>“So you see, the proper spy is very important.” Though the door
had opened automatically, the Baron held it with his hand in
vestigial politeness. “This is one of our more expensive models.
Still well under a million credits, but one of my favorites—though
in practice he has his faults. With a few minor alterations I would
like to make him a permanent part of our arsenal.”</p>
<p>“A model of a spy?” Rydra asked. “Some sort of robot or
android?”</p>
<p>“Not at all.” They approached the display case. “We made half a
dozen TW-55’s. It took the most exacting genetic search. Medical
science has progressed so that all sorts of hopeless human refuse
lives and reproduces at a frightening rate—inferior creatures that
would have been too weak to survive a handful of centuries ago. We
chose our parents carefully, and then with artificial insemination
we got our half dozen zygotes, three male, three female. We raised
them in, oh, such a carefully controlled nutrient environment,
speeding the growth rate by hormones and other things. But the
beauty of it was the experiential imprinting. Gorgeously healthy
creatures; you have no idea how much care they received.”</p>
<p>“I once spent a summer on a cattle farm,” Rydra said
shortly.</p>
<p>The Baron’s nod was brisk. “We’d used the experiential imprints
before, so we knew what we were doing. But never to synthesize
completely the life situation of, say, a sixteen-year-old human.
Sixteen was the physiological age we brought them to in six months.
Look for yourself what a splendid specimen it is. The reflexes are
fifty percent above those of a human aged normally. Human
musculature is beautifully engineered: a three-day-starved,
six-month-atrophied myasthenia gravis case, can, with the proper
stimulant drugs, overturn a ton-and-a-half automobile. It will kill
him—but that’s still remarkable efficiency. Think what the
biologically perfect body, operating at all times at point
nine-nine efficiency, could accomplish in physical strength
alone.”</p>
<p>“I thought hormone growth incentive had been outlawed. Doesn’t
it reduce the life span some drastic amount?”</p>
<p>“To the extent we used it, the life span reduction is
seventy-five percent and over.” He might have smiled the same way
watching some odd animal at its incomprehensible antics. “But,
Madam, we are making weapons. If TW-55 can function twenty years at
peak efficiency, then it will have outlasted the average battle
cruiser by five years. But the experiential imprinting! To find
among ordinary men someone who can function as a spy, is willing to
function as a spy, you must search the fringes of neurosis, often
psychosis. Though such deviations might mean strength in a
particular area, it always means an overall weakness in the
personality. Functioning in any but that particular area, a spy may
be dangerously inefficient. And the Invaders have psyche-indices
too, which will keep the average spy out of anyplace we might want
to put him. Captured, a good spy is a dozen times as dangerous as a
bad one. Post-hypnotic suicide suggestions and the like are easily
gotten around with drugs; and are wasteful. TW-55 here will
register perfectly normal on a psyche integration. He has about six
hours of social conversation, plot synopses of the latest novels,
political situations, music, and art criticism—l believe in the
course of an evening he is programmed to drop your name twice, an
honor you share only with Ronald Quar. He has one subject on which
he can expound with scholarly acumen for an hour and a half—this
one, I believe, is ‘haptoglobin groupings among the marsupials.’
Put him in formal wear and he will be perfectly at home at an
ambassadorial ball or a coffee break at a high-level government
conference. He is a crack assassin, expert with all the weapons you
have seen up till now, and more. TW-55 has twelve hours’ worth of
episodes in fourteen different dialects, accents, or jargons
concerning sexual conquests, gambling experiences, fisticuff
encounters, and humorous anecdotes of semi-illegal enterprises, all
of which failed miserably. Tear his shirt, smear grease on his face
and slip a pair of overalls on him, and he could be a service
mechanic on any one of a hundred spaceyards or stellarcenters on
the other side of the Snap. He can disable any space drive system,
communications components, radar works, or alarm system used by the
Invaders in the past twenty years with little more than—”</p>
<p>“Six inches of vanadium wire?”</p>
<p>The Baron smiled. “His fingerprints and retina pattern, he can
alter at will. A little neural surgery has made all the muscles of
his face voluntary, which means he can alter his facial structure
drastically. Chemical dyes and hormone banks beneath the scalp
enable him to color his hair in seconds, or, if necessary, shed it
completely and grow a new batch in half an hour. He’s a past master
in the psychology and physiology of coercion.”</p>
<p>“Torture?”</p>
<p>“If you will. He is totally obedient to the people whom he has
been conditioned to regard as his superiors; totally destructive
toward what he has been ordered to destroy. There is nothing in
that beautiful head even akin to a superego.”</p>
<p>“He is…” and she wondered at herself speaking, “beautiful.” The
dark-lashed eyes with lids about to quiver open, the broad hands
hung at the naked thighs, fingers half-curled, about to straighten
or become a fist. The display light was misty on the tanned, yet
near-translucent skin. “You say this isn’t a model, but really
alive?”</p>
<p>“Oh, more or less. But it’s rather firmly fixed in something
like a yoga trance, or a lizard’s hibernation. I could activate it
for you—but it’s ten to seven. We don’t want to keep the others
waiting at the table now, do we?”</p>
<p>She looked away from the figure in glass to the dull, taut skin
of the Baron’s face. His jaw, beneath his faintly concave cheek,
was involuntarily working on its hinge.</p>
<p>(from <em>Babel-17</em> by Samuel R. Delany)</p></div>
    </summary>
    <updated>2026-01-01T07:11:17Z</updated>
    <published>2026-01-01T07:11:17Z</published>
    <category term="english"/>
    <source>
      <id>http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/</id>
      <author>
        <name>Chung-chieh Shan</name>
      </author>
      <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/" rel="alternate" type="text/html"/>
      <link href="http://conway.rutgers.edu/~ccshan/wiki/blog/tags/english/index.rss" rel="self" type="application/rss+xml"/>
      <subtitle>Chung-chieh Shan's blog</subtitle>
      <title>Proper Treatment</title>
      <updated>2026-01-01T08:01:50Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-US">
    <id>584219d403596e3099e0ee9b:58462c0e15d5db6feba171c0:68e2d55b4d325815f97c2786</id>
    <link href="https://mmhaskell.com/blog/2025/12/29/parsing-with-an-mcap-index" rel="alternate" type="text/html"/>
    <title>Parsing with an MCAP Index</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Welcome to the sixth and final part of our series on MCAP parsing. In the first four parts, we did all the work to parse certain topic messages from a bag file, going through the file sequentially. In the <a href="https://mmhaskell.com/blog/2025/12/22/mcap-indexing">fifth part</a>, we set up a lot of infrastructure to allow ourselves to instead parse the file using <strong>indexing</strong> so we can get our messages faster. In this part, we’ll finish the process and see how much faster our code is!</p>
<p>This series has combined a lot of ideas from several of our online courses. You can learn more about complex monad structures in <a href="https://academy.mondaymorninghaskell.com/p/making-sense-of-monads">Making Sense of Monads</a> or <a href="https://academy.mondaymorninghaskell.com/p/effectful-haskell">Effectful Haskell</a>. You can learn more about parsing in <a href="https://academy.mondaymorninghaskell.com/p/solve-hs">Solve.hs</a>. You can get lifetime access to all our courses buy purchasing <a href="https://academy.mondaymorninghaskell.com/p/mmh-complete">MMH Complete</a>!</p>
<h2>The Plan</h2>
<p>Let’s recap the most important items from the <a href="https://mmhaskell.com/blog/2025/12/22/mcap-indexing">last part</a> and go over our plan again. We defined a series of types to help us represent various record types, and parsers for them:</p>
<pre><code class="language-haskell">data FooterRec = ...
parseFooter’ :: parseFooter' :: (MonadParsec Void ByteString m) =&gt; m (Word64, FooterRec)

data SummaryOffsetRec = ...
parseSummaryOffset :: (MonadFail m, MonadParsec Void ByteString m) =&gt; m (Word64, SummaryOffsetRec)

data ChunkIndexRec = ...
parseChunkIndex :: (MonadFail m, MonadParsec Void ByteString m) =&gt; m (Word64, ChunkIndexRec)

data MessageIndexRec = ...
parseMessageIndex' :: (MonadFail m, MonadParsec Void ByteString m) =&gt; m (Word64, MessageIndexRec)</code></pre>
<p>We also defined a couple monad structures to capture our parser’s state:</p>
<pre><code class="language-haskell">data TopicState = TopicState
  { tsChannels :: HM.HashMap Word16 MsgChannel
  , tsSchemas :: HM.HashMap Word16 MsgSchema
  , tsMessages :: HM.HashMap (ByteString, ByteString) [MessageData]
  , tsDesiredTopics :: HS.HashSet ByteString
  , tsDesiredChannels :: HS.HashSet Word16
  } deriving (Show)

type MCAPReader a = StateT TopicState (ReaderT Handle (ExceptT ValidationError IO)) a
type MCAPParser a = ParsecT Void ByteString (StateT TopicState (ReaderT Handle (ExceptT ValidationError IO)))  a</code></pre>
<p>Finally, we defined a series of utility functions on these monads:</p>
<pre><code class="language-haskell">seek :: Integer -&gt; MCAPReader ()
curPos :: MCAPReader Integer
mcapGuard :: String -&gt; Bool -&gt; MCAPReader ()
runMCAPParser :: MCAPParser (Word64, a) -&gt; ByteString -&gt; MCAPReader (Word64, a)
parseNextRecord :: RecordType -&gt; MCAPParser (Word64, a) -&gt; MCAPReader (Word64, a)
parseNextRecord' :: RecordType -&gt; (Word64 -&gt; MCAPParser (Word64, a)) -&gt; MCAPReader (Word64, a)
runMCAPReader :: Handle -&gt; TopicState -&gt; MCAPReader a -&gt; IO (Either ValidationError TopicState)</code></pre>
<p>Our overall plan is:</p>
<ol>
<li>Parse the footer and use its data to jump to the start of the summary offset section</li>
<li>Parse summary offsets for schemas, channels, and chunk indexes</li>
<li>Parse summary records for schemas and channels to load them into <code>TopicState</code></li>
<li>Process chunk indexes, which direct us to message index records for our desired channels</li>
<li>Process message indexes, which point us to individual messages for our desired topics</li>
<li>Parse the messages into our Topic State</li>
</ol>
<p>Let’s get started!</p>
<h2>Step 1 - Parsing the Footer</h2>
<p>Our entrypoint for the <code>MCAPReader</code> monad will be a function called <code>parseUsingIndex</code> (we’ll see how to enter this function later).</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()</code></pre>
<p>The first order of business is to seek to the Footer’s location so we can parse the Footer record. The Footer has a fixed size and it is always the last record before the 8 “magic bytes” that conclude the file. This size is 29 bytes: 1 byte for the op code (record type), 8 bytes to define the content size, and 20 bytes for the 3 fields (<code>Word64</code>, <code>Word64</code>, <code>Word32</code>). Combined with the magic bytes, this means we want to find the file size and seek to 37 bytes before it:</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart &lt;- seekToFooterStart
  ...
  where
    seekToFooterStart = do
      hndl &lt;- ask
      sz &lt;- liftIO $ hFileSize hndl
      seek (sz - 37)
      curPos</code></pre>
<p>Now we use <code>parseNextRecord</code> and <code>parseFooter’</code> to get the Footer’s data, which includes the start of the summary offset section. We unpack this value and seek to its location:</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart &lt;- seekToFooterStart
  (_, FooterRec summStart summOffStart _) &lt;- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  ...
  where
    seekToFooterStart = do
      hndl &lt;- ask
      sz &lt;- liftIO $ hFileSize hndl
      seek (sz - 37)
      curPos</code></pre>
<p>Now we’re ready for step 2!</p>
<h2>Step 2 - Parsing Summary Offsets</h2>
<p>Recall that records in the summary section are grouped in blocks, and each summary offset record leads us to the start of the block for a particular op code. We’re interested in the summary records for schemas, channels and chunk indexes. So we want to keep parsing summary offsets until we have all three of these.</p>
<p>So we’ll build a helper function that will construct a map from op codes to summary offset records, but also track booleans for which of our desired types we’ve seen. This function also needs to track the number of bytes remaining in the summary offset section:</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart &lt;- seekToFooterStart
  (_, FooterRec summStart summOffStart _) &lt;- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  ...
  where
    parseSummaryOffsets :: Word64 -&gt; Bool -&gt; Bool -&gt; Bool -&gt; M.Map RecordType SummaryOffsetRec -&gt; MCAPReader (M.Map RecordType SummaryOffsetRec)
    ...</code></pre>
<p>There are two base cases. If we are out of bytes, we return our accumulated map. If all three boolean flags are true, then we can also return the map, as we have what we need.</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  ...
  where
    parseSummaryOffsets :: Word64 -&gt; Bool -&gt; Bool -&gt; Bool -&gt; M.Map RecordType SummaryOffsetRec -&gt; MCAPReader (M.Map RecordType SummaryOffsetRec)
    parseSummaryOffsets 0 _ _ _ acc = return acc
    parseSummaryOffsets _ True True True acc = return acc
    parseSummaryOffsets remBytes hasSchema hasChannel hasChunkIndex acc =
    ...</code></pre>
<p>For the general case, we use <code>parseSummaryOffset</code> to parse the next record. We update our boolean flags depending on the op code, and we insert the new record into our map. Then we recurse:</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  ...
  where
    parseSummaryOffsets :: Word64 -&gt; Bool -&gt; Bool -&gt; Bool -&gt; M.Map RecordType SummaryOffsetRec -&gt; MCAPReader (M.Map RecordType SummaryOffsetRec)
    parseSummaryOffsets 0 _ _ _ acc = return acc
    parseSummaryOffsets _ True True True acc = return acc
    parseSummaryOffsets remBytes hasSchema hasChannel hasChunkIndex acc = do
      mcapGuard ("Not enough bytes for summary offset: " &lt;&gt; show remBytes) (remBytes &gt;= 26)
      (_, summ@(SummaryOffsetRec op _ _)) &lt;- parseNextRecord SummaryOffset parseSummaryOffset
      let hasSchema' = hasSchema || op == Schema
      let hasChannel' = hasChannel || op == Channel
      let hasChunkIndex' = hasChunkIndex || op == ChunkIndex
      let mp = M.insert op summ acc
      parseSummaryOffsets (remBytes - 26) hasSchema' hasChannel' hasChunkIndex' mp</code></pre>
<p>Now we’ll define a second helper. This will use the <code>Maybe</code> monad to extract the 3 records we care about. </p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  ...
  where
    ...
    find3Offsets :: M.Map RecordType SummaryOffsetRec -&gt; Maybe (SummaryOffsetRec, SummaryOffsetRec, SummaryOffsetRec)
    find3Offsets mp = do
      s &lt;- M.lookup Schema mp
      c &lt;- M.lookup Channel mp
      ci &lt;- M.lookup ChunkIndex mp
      return (s, c, ci)</code></pre>
<p>Now back in the main line of the function, we’ll get the size of this section (subtract the summary offset start from the footer start position) and call our two helpers. If we don’t have all three maps, we’ll throw a validation error showing what op codes we did find. Otherwise, we have the 3 offsets.</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart &lt;- seekToFooterStart
  (_, FooterRec summStart summOffStart _) &lt;- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  -- Read Summary Offsets until we have Schema, Channel and ChunkIndex
  let sz = fromIntegral footerStart - summOffStart
  summOffMap &lt;- parseSummaryOffsets sz False False False M.empty
  let offsets = find3Offsets summOffMap
  case offsets of
    Nothing -&gt; throwError (ValidationError $ "Did not find the right summary offsets: " &lt;&gt; show (M.keys summOffMap))
    Just (schemaOffset, channelOffset, chunkIndexOffset) -&gt; ...
  where
    parseSummaryOffsets :: Word64 -&gt; Bool -&gt; Bool -&gt; Bool -&gt; M.Map RecordType SummaryOffsetRec -&gt; MCAPReader (M.Map RecordType SummaryOffsetRec)

    find3Offsets :: M.Map RecordType SummaryOffsetRec -&gt; Maybe (SummaryOffsetRec, SummaryOffsetRec, SummaryOffsetRec)</code></pre>
<p>Now we’ve got these summary offset records! It’s time to use them to start populating our topic state.</p>
<h2>Step 3 - Summary Schemas and Channels</h2>
<p>We’ll begin with the schemas and channels. These are fairly easy, since we can rely on previous code for the most part. Recall that we have the functions <code>parseSchema’</code> and <code>parseChannel’</code> for parsing individual schema and channel records. We need to update these type signatures to be more generic like so:</p>
<pre><code class="language-haskell">parseSchema' :: (MonadParsec Void ByteString m, MS.MonadState TopicState m) =&gt; m (Word64, Record)

parseChannel' :: (MonadFail m, MonadParsec Void ByteString m, MS.MonadState TopicState m) =&gt; m (Word64, Record)</code></pre>
<p>These will work with the <code>MCAPParser</code> monad. They require <code>MonadState TopicState</code> because they’ll save the schemas and channels within our <code>TopicState</code>, rather than requiring us to do anything with the return values.</p>
<p>So we just need to write functions that will look through <strong>all</strong> the records in the summary block and parse them. Since the summary offset contains the start location and the byte length of all the records, we’ll seek to that location, and then use our tried and true method of parsing until we run out of bytes. Here’s a function to load all the schemas in a group:</p>
<pre><code class="language-haskell">loadSchemasForIndex :: SummaryOffsetRec -&gt; MCAPReader ()
loadSchemasForIndex (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, _) &lt;- parseNextRecord Schema parseSchema'
      mcapGuard ("Summary schema is too long! " &lt;&gt; show len &lt;&gt; " " &lt;&gt; show rem) (len &lt;= rem)
      f (rem - len)</code></pre>
<p>And here’s the function for channels:</p>
<pre><code class="language-haskell">loadChannelsForIndex :: SummaryOffsetRec -&gt; MCAPReader ()
loadChannelsForIndex (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, _) &lt;- parseNextRecord Channel parseChannel'
      mcapGuard ("Summary channel is too long! " &lt;&gt; show len &lt;&gt; " " &lt;&gt; show rem) (len &lt;= rem)
      f (rem - len)</code></pre>
<p>We simply call these from within <code>parseUsingIndex</code> like so:</p>
<pre><code class="language-haskell">parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart &lt;- seekToFooterStart
  (_, FooterRec summStart summOffStart _) &lt;- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  -- Read Summary Offsets until we have Schema, Channel and ChunkIndex
  let sz = fromIntegral footerStart - summOffStart
  summOffMap &lt;- parseSummaryOffsets sz False False False M.empty
  let offsets = find3Offsets summOffMap
  case offsets of
    Nothing -&gt; throwError (ValidationError $ "Did not find the right summary offsets: " &lt;&gt; show (M.keys summOffMap))
    Just (schemaOffset, channelOffset, chunkIndexOffset) -&gt; do
      loadSchemasForIndex schemaOffset
      loadChannelsForIndex channelOffset
      ...</code></pre>
<p>Now our internal <code>TopicState</code> is updated with the schemas and channels. This means we’re ready to parse messages, but we need to find where those messages are first!</p>
<h2>Step 4 - Chunk Index Processing</h2>
<p>Now we need to process each Chunk Index record. We’ll write a function for this, and it will be the last thing we need to call from <code>parseUsingIndex</code>:</p>
<pre><code class="language-haskell">processChunkIndexes :: SummaryOffsetRec -&gt; MCAPReader ()

parseUsingIndex :: MCAPReader ()
parseUsingIndex = do
  footerStart &lt;- seekToFooterStart
  (_, FooterRec summStart summOffStart _) &lt;- parseNextRecord FileFooter parseFooter'
  seek (fromIntegral summOffStart)
  -- Read Summary Offsets until we have Schema, Channel and ChunkIndex
  let sz = fromIntegral footerStart - summOffStart
  summOffMap &lt;- parseSummaryOffsets sz False False False M.empty
  let offsets = find3Offsets summOffMap
  case offsets of
    Nothing -&gt; throwError (ValidationError $ "Did not find the right summary offsets: " &lt;&gt; show (M.keys summOffMap))
    Just (schemaOffset, channelOffset, chunkIndexOffset) -&gt; do
      loadSchemasForIndex schemaOffset
      loadChannelsForIndex channelOffset
      processChunkIndexes chunkIndexOffset
  where
    ...</code></pre>
<p>To fill out this function, we follow the same general pattern to loop through these records until we’ve exhausted the bytes:</p>
<pre><code class="language-haskell">processChunkIndexes :: SummaryOffsetRec -&gt; MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) &lt;- parseNextRecord ChunkIndex parseChunkIndex
      ...</code></pre>
<p>The trick is that processing each individual chunk index is more complicated after we’ve parsed it. Each chunk index refers to a single <code>Chunk</code> record and contains its initial position (we saved this in the <code>cirChunkStartOffset</code> field). However, the message index offsets are not absolute positions. They are <strong>relative</strong> to the start of the data section of the chunk. So we want to calculate where this location is (and it took me a bit of debugging to get it right!).</p>
<p>Prior to the data section in the chunk is the record header, which is 9 bytes. Then there are 32 fixed bytes, followed by the compression string. Then there are still 8 more bytes after that because the data section in the chunk is prefixed by the byte count. You can consult the <a href="https://mcap.dev/spec#chunk-op0x06">Chunk record specification</a> to see where these numbers are coming from. So putting it all together, we take the chunk start location, add 49 bytes, and then add the length of the compression string.</p>
<pre><code class="language-haskell">processChunkIndexes :: SummaryOffsetRec -&gt; MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) &lt;- parseNextRecord ChunkIndex parseChunkIndex
      let chunkDataStart = cirChunkStartOffset ci + 49 + (fromIntegral $ BS.length (cirCompression ci))
      savedPosition &lt;- curPos
      ... -- Process message index records
      seek savedPosition</code></pre>
<p>You’ll also see that we <strong>save the current handle position</strong>. We’ll process the message index records and jump around to the messages. But we want to return to the current location (the location after parsing <em>this</em> chunk index) so that we can process the <em>next</em> chunk index afterward.</p>
<p>Now we have to look through all of our desired channels and see which of them are referred to in the <code>cirMessageIndexOffsets</code> of this chunk index:</p>
<pre><code class="language-haskell">processChunkIndexes :: SummaryOffsetRec -&gt; MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) &lt;- parseNextRecord ChunkIndex parseChunkIndex
      let chunkDataStart = cirChunkStartOffset ci + 49 + (fromIntegral $ BS.length (cirCompression ci))
      savedPosition &lt;- curPos
      chans &lt;- MS.gets tsDesiredChannels
      let allMIO = cirMessageIndexOffsets ci
      forM_ chans $ \chanId -&gt; when (HM.member chanId allMIO) $ do
        ...
      seek savedPosition</code></pre>
<p>Now we get the offset for this channel in the map. This will give us the (absolute) location of the Message Index record for this channel. We’ll prepare a new function to process this message index:</p>
<pre><code class="language-haskell">processChunkIndexes :: SummaryOffsetRec -&gt; MCAPReader ()
processChunkIndexes (SummaryOffsetRec _ groupStart groupLen) = do
  seek (fromIntegral groupStart)
  f groupLen
  where
    f 0 = return ()
    f rem = do
      (len, ci) &lt;- parseNextRecord ChunkIndex parseChunkIndex
      let chunkDataStart = cirChunkStartOffset ci + 49 + (fromIntegral $ BS.length (cirCompression ci))
      savedPosition &lt;- curPos
      chans &lt;- MS.gets tsDesiredChannels
      let allMIO = cirMessageIndexOffsets ci
      forM_ chans $ \chanId -&gt; when (HM.member chanId allMIO) $ do
        let chanOffset = allMIO HM.! chanId
        loadMessagesFromMessageIndex chanId chanOffset chunkDataStart
      seek savedPosition

loadMessagesFromMessageIndex :: Word16 -&gt; Word64 -&gt; Word64 -&gt; MCAPReader ()
loadMessagesFromMessageIndex chanId offset chunkDataStart</code></pre>
<p>This is all we need for <code>processChunkIndexes</code>. We’re getting closer to our goal! Now we need to process the message index record.</p>
<h2>Steps 5 &amp; 6 - Processing Message Indexes and Messages</h2>
<p>You’ll note from above that we passed the channel ID, the offset of the message index, and the corresponding “chunk data start”. We begin this function by seeking the to message index location, parsing the message index, and ensuring that we are getting an index for the correct channel:</p>
<pre><code class="language-haskell">loadMessagesFromMessageIndex :: Word16 -&gt; Word64 -&gt; Word64 -&gt; MCAPReader ()
loadMessagesFromMessageIndex chanId offset chunkDataStart = do
  seek (fromIntegral offset)
  (_, MessageIndexRec mirChanId recs) &lt;- parseNextRecord MessageIndex parseMessageIndex'
  mcapGuard ("Mismatched channels for message index record: " &lt;&gt; show chanId &lt;&gt; " " &lt;&gt; show mirChanId) (mirChanId == chanId)
  ...</code></pre>
<p>Now we loop through the record tuples in the message index record. Each of these contains a relative offset. We just need to seek to that offset, adding in the chunk data start location. Then we can use our message parser function!</p>
<pre><code class="language-haskell">loadMessagesFromMessageIndex :: Word16 -&gt; Word64 -&gt; Word64 -&gt; MCAPReader ()
loadMessagesFromMessageIndex chanId offset chunkDataStart = do
  seek (fromIntegral offset)
  (_, MessageIndexRec mirChanId recs) &lt;- parseNextRecord MessageIndex parseMessageIndex'
  mcapGuard ("Mismatched channels for message index record: " &lt;&gt; show chanId &lt;&gt; " " &lt;&gt; show mirChanId) (mirChanId == chanId)
  forM_ recs $ \(_, offset) -&gt; do
    seek (fromIntegral $ chunkDataStart + offset)
    parseNextRecord' Message parseMessage'

parseMessage' :: (MonadParsec Void ByteString m, MS.MonadState TopicState m, MonadFail m) =&gt; Word64 -&gt; m (Word64, Record)</code></pre>
<p>That’s all we need! Like our schema and channel parsers, <code>parseMessage’</code> does the work of saving the messages in our <code>TopicState</code>, so we don’t need to return or process anything. The path we started from <code>parseUsingIndex</code> is now complete!</p>
<h2>Compression Disclaimer</h2>
<p>As an important note, this code takes a shortcut. The approach given will only work with <strong>uncompressed chunks</strong>! The relative offset we used for the message index is actually the offset from the start of the uncompressed data. If the chunk’s data is compressed, we would have to grab that whole uncompressed string, decompress as much of it as we need, and then jump to particular locations within that bytestring. The code required to do this efficiently is much more intricate, as it requires us to manage counts on the streams of compressed and uncompressed data.</p>
<p>This highlights an important tradeoff. By leaving the data uncompressed, it takes up more storage space and would take longer to send over a network if you need to download these bag files. But if the data is compressed, we might have to decompress a significant portion of it to find the few messages we care about, meaning we don’t see the same gains in bag read time. The particular needs of your application &amp; systems will determine what choice makes sense.</p>
<h2>Pulling it Together</h2>
<p>Now we need an entrypoint to call <code>parseUsingIndex</code>. Our function will take the filepath, open a handle to it, and create the initial topic state, looking for the <code>/turtle1/cmd_vel</code> topic. We’ll invoke the <code>parseUsingIndex</code> function with <code>runMCAPParser</code>:</p>
<pre><code class="language-haskell">parseRecordsFromFileWithIndex :: FilePath -&gt; IO ()
parseRecordsFromFileWithIndex fp = do
  parseBareRecordsFromFile
  startTime &lt;- getCurrentTime
  handle &lt;- openFile fp ReadMode
  let initialTs = TopicState HM.empty HM.empty HM.empty (HS.fromList ["/turtle1/cmd_vel"]) HS.empty
  result &lt;- runMCAPReader handle initialTs parseUsingIndex
  ...</code></pre>
<p>Then, in a reprisal of previous code, we’ll loop through all the messages for our topic (they use the <code>geometry_msgs/msg/Twist</code> type from ROS2), and decode them all. (Recall that in the <a href="https://mmhaskell.com/blog/2025/12/22/mcap-indexing">previous part</a> we wrote <code>parseVelMsg</code> for this type).</p>
<pre><code class="language-haskell">parseRecordsFromFileWithIndex :: FilePath -&gt; IO ()
parseRecordsFromFileWithIndex fp = do
  startTime &lt;- getCurrentTime
  handle &lt;- openFile fp ReadMode
  let initialTs = TopicState HM.empty HM.empty HM.empty (HS.fromList ["/turtle1/cmd_vel"]) HS.empty
  result &lt;- runMCAPReader handle initialTs parseUsingIndex
  case result of
    Left e -&gt; print e
    Right finalTs -&gt; do
      let velMessages = fromMaybe [] $ HM.lookup ("/turtle1/cmd_vel", "geometry_msgs/msg/Twist") (tsMessages finalTs)
      forM_ velMessages $ \(MessageData t1 t2 msg) -&gt; do
        parseVelResult &lt;- evalStateT (runParserT parseVelMsg "Velocity" msg) 0
        case parseVelResult of
          Left e -&gt; print e
          Right s -&gt; putStrLn $ "Parsed Velocity Message: " &lt;&gt; show t1 &lt;&gt; " " &lt;&gt; show t2 &lt;&gt; " " &lt;&gt; show s
      endTime &lt;- getCurrentTime
      print (diffUTCTime endTime startTime)</code></pre>
<p>And now we’re done! We added some timing features, so we can compare this to our previous run. It is <strong>much</strong> faster! The original takes around 0.075s. The new version takes about 0.001s. So it’s about <strong>75x faster</strong>, a great speedup!</p>
<h2>Conclusion</h2>
<p>This concludes our series on ROS2 bag parsing! Over the course of this long series, we were able to take an MCAP bag file, parse topic messages out of it, and then use the indexing features to do this even faster! We learn a ton about the MCAP format as well as CDR encoding. We saw how to compose parsers, even when they had stateful side effects like file seeking or saving message data.</p>
<p>With so much ground to cover, there wasn’t a lot of time in this series to go over the basics. If you want to solidify your Haskell skills so that you can write advanced code like this, the <strong>best</strong> way to do that is to take a look at some of our <a href="https://academy.mondaymorninghaskell.com/courses">courses</a>.</p>
<p>The most immediately useful for this series would be <a href="https://academy.mondaymorninghaskell.com/p/solve-hs">Solve.hs</a>. Module 1 teaches you valuable loop structures that form the building block for all of our complicated functions in Haskell. Then Module 4 will teach you about parsing and Megaparsec! In between, you’ll also learn about Data Structures and Algorithms in Haskell!</p>
<p><a href="https://academy.mondaymorninghaskell.com/p/making-sense-of-monads">Making Sense of Monads</a> and <a href="https://academy.mondaymorninghaskell.com/p/effectful-haskell">Effectful Haskell</a> are also great choices if the monad structures in the second half of the series are a bit confusing to you. You can grab both of these courses for a discount by purchasing the <a href="https://academy.mondaymorninghaskell.com/p/effects-bundle">Effects Bundle</a>!</p>
<p>Finally, if you want access to all our course content, past, present and future, you can get the <a href="https://academy.mondaymorninghaskell.com/p/mmh-complete">MMH Complete bundle</a>! This will include all the previous courses, as well as our other full length courses <a href="https://academy.mondaymorninghaskell.com/p/haskell-from-scratch">Haskell From Scratch</a>, <a href="https://academy.mondaymorninghaskell.com/p/practical-haskell">Practical Haskell</a>, plus our mini-course on machine learning, <a href="https://academy.mondaymorninghaskell.com/p/haskell-brain">Haskell Brain</a>.</p></div>
    </summary>
    <updated>2025-12-29T12:30:00Z</updated>
    <published>2025-12-29T12:30:00Z</published>
    <author>
      <name>James Bowen</name>
    </author>
    <source>
      <id>https://mmhaskell.com/blog/</id>
      <link href="https://mmhaskell.com/blog/" rel="alternate" type="text/html"/>
      <link href="https://mmhaskell.com/blog?format=rss" rel="self" type="application/rss+xml"/>
      <title>Blog - Monday Morning Haskell</title>
      <updated>2025-10-05T20:31:39Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://haskell.org/ghc/blog/20251227-ghc-9.12.3-released.html</id>
    <link href="http://haskell.org/ghc/blog/20251227-ghc-9.12.3-released.html" rel="alternate" type="text/html"/>
    <title>GHC 9.12.3 is now available</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><h1>GHC 9.12.3 is now available</h1>
<h4 class="text-muted">zubin - 2025-12-27</h4>

<p>The GHC developers are very pleased to announce the release of GHC 9.12.3.
Binary distributions, source distributions, and documentation are available at
<a href="https://downloads.haskell.org/ghc/9.12.3">downloads.haskell.org</a>.</p>
<p>GHC 9.12.3 is a bug-fix release fixing many issues of a variety of
severities and scopes, including:</p>
<ul>
<li>Fix a number of crashes and miscompilations in the compiler frontend (<a href="https://gitlab.haskell.org/ghc/ghc/issues/25004">#25004</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/25960">#25960</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26256">#26256</a>)</li>
<li>Improvements to efficiency of the runtime linker (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26009">#26009</a>)</li>
<li>Fixes for several bugs in bytecode generation and the bytecode interpreter (<a href="https://gitlab.haskell.org/ghc/ghc/issues/23210">#23210</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/25975">#25975</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/25750">#25750</a>)</li>
<li>Fixes for bugs in the handling of WHITEHOLEs in the RTS (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26204">#26204</a>, <a href="https://gitlab.haskell.org/ghc/ghc/issues/26205">#26205</a>)</li>
<li>Fix incorrect code generation for SSE vector operations (<a href="https://gitlab.haskell.org/ghc/ghc/issues/25859">#25859</a>)</li>
<li>Fix a use-after-free in the Windows runtime linker (<a href="https://gitlab.haskell.org/ghc/ghc/issues/26613">#26613</a>)</li>
<li>Support for synchronous JSFFI exports for the wasm backend</li>
<li>And many more!</li>
</ul>
<p>A full accounting of these fixes can be found in the
<a href="https://downloads.haskell.org/~ghc/9.12.3/docs/users_guide/9.12.3-notes.html">release notes</a>. As always, GHCâ€™s release status, including planned future
releases, can be found on the GHC Wiki <a href="https://gitlab.haskell.org/ghc/ghc/-/wikis/GHC-status">status</a>.</p>
<p>GHC development is sponsored by:</p>
<ul>
<li><a href="https://juspay.com/">Juspay</a></li>
<li><a href="https://qbaylogic.com/">QBayLogic</a></li>
<li><a href="https://www.channable.com/">Channable</a></li>
<li><a href="https://haskell.foundation/">Haskell Foundation</a></li>
<li><a href="https://serokell.io/">Serokell</a></li>
<li><a href="https://well-typed.com/">Well-Typed</a></li>
<li><a href="https://www.tweag.io/">Tweag</a></li>
<li><a href="https://www.dotcom-monitor.com/">Dotcom-Monitor</a></li>
<li><a href="https://www.loadview-testing.com/">LoadView</a></li>
<li><a href="https://webhostingbuddy.com/">Web Hosting Buddy</a></li>
<li><a href="https://www.findmyelectric.com/">Find My Electric</a></li>
<li><a href="https://www.sc.com">Standard Chartered</a></li>
<li><a href="https://upcloud.com">UpCloud</a></li>
<li><a href="https://mercury.com">Mercury</a></li>
</ul>
<p>We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.</p>
<p>As always, do give this release a try and open a <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/new">ticket</a> if you see
anything amiss.</p></div>
    </summary>
    <updated>2025-12-27T00:00:00Z</updated>
    <published>2025-12-27T00:00:00Z</published>
    <author>
      <name>ghc-devs</name>
    </author>
    <source>
      <id>http://haskell.org/ghc</id>
      <link href="http://haskell.org/ghc" rel="alternate" type="text/html"/>
      <link href="http://haskell.org/ghc/rss.xml" rel="self" type="application/rss+xml"/>
      <title>GHC Developer blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry xml:lang="en">
    <id>https://blog.jle.im/entry/advent-of-code-2025.html</id>
    <link href="https://blog.jle.im/entry/advent-of-code-2025.html" rel="alternate" type="text/html"/>
    <title>Advent of Code 2025: Haskell Solution Reflections for all 12 Days</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Merry Christmas all! This is my annual <a href="http://adventofcode.com/">Advent of Code</a> post! Advent of Code is a
series of (this year) 12 daily Christmas-themed programming puzzles that are
meant to be fun diversions from your daily life, help you find a bit of whimsy
in your world, give you a chance to explore new ideas and program together with
your friends. I always enjoy discussing creative ways to solve these puzzles
every day, and it’s become a bit of an annual highlight for me and a lot of
others. My favorite part about these puzzles is that they are open ended enough
that there are usually many different interesting ways to solve them — it’s not
like a stressful interview question where you have to recite the obscure
incantation to pass the test. In the past I’ve leveraged <a href="https://blog.jle.im/entry/alchemical-groups.html">group theory</a>, <a href="https://blog.jle.im/entry/shifting-the-stars.html">galilean
transformations and linear algebra</a>, and <a href="https://blog.jle.im/entry/shuffling-things-up.html">more group
theory</a>.</p>
<p>Haskell is especially fun for these because if you set up your abstractions
in just the right way, the puzzles seem to solve themselves. It’s a good
opportunity every year to get exposed to different parts of the Haskell
ecosystem! Last year, I moved almost all of my Haskell code to <a href="https://github.com/mstksg/advent-of-code">an Advent of Code Megarepo</a>,
and I also write up my favorite ways to solve each one in the <a href="https://github.com/mstksg/advent-of-code/wiki">megarepo wiki</a>.</p>
<p>All of this year’s 12 puzzles <a href="https://github.com/mstksg/advent-of-code/wiki/Reflections-2025">are
here</a>, but I’ve also included links to each individual one in this post. I’m
also considering expanding some of these into full on blog posts, so be on the
look out, or let me know if there are any that you might want fully expanded!
And if you haven’t, why not try these out yourself? Be sure to drop by the
libera-chat <code>##advent-of-code</code> channel to discuss any fun ways you
solve them, or any questions! Thanks again to Eric for a great new fresh take on
the event this year!</p>
<ul>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day01.md">Day
1 - Secret Entrance</a> — The classic Day 1 <code>scanl</code></li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day02.md">Day
2 - Gift Shop</a> — The super efficient <code>IntSet</code></li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day03.md">Day
3 - Lobby</a> — <code>StateT</code> + <code>[]</code> = backtracking search
monad</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day04.md">Day
4 - Printing Department</a> — 2D cellular automata</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day05.md">Day
5 - Cafeteria</a> — The power of the <code>data-interval</code> library</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day06.md">Day
6 - Trash Compactor</a> — <code>Data.List</code> manipulations</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day07.md">Day
7 - Laboratories</a> — Tying the knot</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day08.md">Day
8 - Playground</a> — Iterative Clustering</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day09.md">Day
9 - Movie Theater</a> — <code>IntervalSet</code> and
<code>IntervalMap</code></li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day10.md">Day
10 - Factory</a> — Guassian Elimation, Type-safe Mutable Vectors</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day11.md">Day
11 - Reactor</a> — More Knot Tying and DP!</li>
<li><a href="https://github.com/mstksg/advent-of-code/blob/main/reflections/2025/day12.md">Day
12 - Christmas Tree Farm</a> — Counting</li>
</ul></div>
    </summary>
    <updated>2025-12-24T20:44:05Z</updated>
    <published>2025-12-24T20:44:05Z</published>
    <category term="Haskell"/>
    <author>
      <name>Justin Le</name>
    </author>
    <source>
      <id>https://blog.jle.im/</id>
      <logo>https://blog.jle.im/img/site_logo.jpg</logo>
      <author>
        <name>Justin Le</name>
        <email>justin@jle.im</email>
      </author>
      <link href="https://blog.jle.im/" rel="alternate" type="text/html"/>
      <link href="http://feeds.feedburner.com/incodeblog" rel="self" type="application/rss+xml"/>
      <rights>Copyright 2020 Justin Le</rights>
      <subtitle>Weblog of Justin Le, covering various adventures in programming and explorations in the worlds of computation physics, and knowledge.</subtitle>
      <title>in Code â€” Entries</title>
      <updated>2026-02-02T15:06:46Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-US">
    <id>584219d403596e3099e0ee9b:58462c0e15d5db6feba171c0:68e2d5049ba78f084d62b38a</id>
    <link href="https://mmhaskell.com/blog/2025/12/22/mcap-indexing" rel="alternate" type="text/html"/>
    <title>MCAP Indexing</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>In the first 4 parts of this series (<a href="https://mmhaskell.com/blog/2025/11/24/robotics-amp-parsing-mcap">Part 1</a>, <a href="https://mmhaskell.com/blog/2025/12/01/the-structure-of-an-mcap-file">Part 2</a>, <a href="https://mmhaskell.com/blog/2025/12/08/schemas-channels-amp-messages-in-mcap">Part 3</a>, <a href="https://mmhaskell.com/blog/2025/12/15/parsing-cdr-messages">Part 4</a>), we’ve written a program that can parse specific topic information out of MCAP-based ROS2 bags. The <a href="https://hackage.haskell.org/package/megaparsec-9.7.0/docs/Text-Megaparsec.html">Megaparsec library</a> helped us a lot, but our current approach has a weakness; our program scans through the entire file sequentially. Bag files can be very large! So if we’re only interested in a few topics, it can be wasteful to parse through everything.</p>
<p>This is where indexes come in. In the final two parts of this series, we’ll see how we can use MCAP’s indexing features to quickly locate the messages we’re interested in. In this part, we’ll focus on the monadic structure of our code, because this will need to change quite a bit. We’ll also talk about the high-level details of the solution approach. In the next part, we’ll write the code to reach our goal!</p>
<p>In this article, we’ll employ some advanced monad techniques. If you still struggle with understanding monads, you should take our course, <a href="https://academy.mondaymorninghaskell.com/p/making-sense-of-monads">Making Sense of Monads</a>! It will help you understand these concepts from the ground up. If you’re more advanced and looking for a challenge, <a href="https://academy.mondaymorninghaskell.com/p/effectful-haskell">Effectful Haskell</a> is also a great course to try!</p>
<h2>What is Indexing?</h2>
<p>Indexing is a technique in many programming applications that allows us to query for certain data more quickly. The term originally comes from books, where any research or historical book has an “Index” section at the back where you can look through a sorted list of people, places and concepts and it will tell you the page(s) where they are mentioned.</p>
<p>In the programming context, you’ll hear about them most often with databases. If you have a “Users” table in your database, it will, by default, be very quick to look up a user with their “primary key”. But you’ll probably also add an “index” on their username or email address field, so that when they login, you can quickly find their information without searching the entire users table.</p>
<p>With MCAP bags, we can use indexing to quickly locate data based on the topic or the timestamp, as these are the most common filtering mechanisms for parsing robotics data. Indexing makes use of a number of the record types we skipped over in previous parts. In these final articles, we’ll use the following record types </p>
<h2>The Problem Context</h2>
<p>To measure our success with indexing, I created a new bag to parse using the <a href="https://docs.ros.org/en/kilted/Tutorials/Beginner-CLI-Tools/Introducing-Turtlesim/Introducing-Turtlesim.html">Turtlesim</a> node that comes with ROS2 by default. I used this to create a bag file with two topics: <code>/turtle1/pose</code> and <code>/turtle1/cmd_vel</code>.</p>
<p>You don’t need to know much about what these mean, except that our bag has around 12000 messages on the <code>/turtle1/pose</code> topic and only 22 messages on the <code>/turtle1/cmd_vel</code> topic. So if we want to search for the latter topic, we don’t want to process all the pose data.</p>
<p>The <a href="https://docs.ros2.org/foxy/api/geometry_msgs/msg/Twist.html">velocity topic message</a> type is easy to parse, consisting only of 6 floating point values:</p>
<pre><code class="language-haskell">data VelocityMsg = VelocityMsg
  { linearX :: Double
  , linearY :: Double
  , linearZ :: Double
  , angularX :: Double
  , angularY :: Double
  , angularZ :: Double
  } deriving (Show, Eq)

parseVelMsg :: CDRParser VelocityMsg
parseVelMsg = do
  parseCdrHeader
  d1 &lt;- parseCdrDoubleLE
  d2 &lt;- parseCdrDoubleLE
  d3 &lt;- parseCdrDoubleLE
  d4 &lt;- parseCdrDoubleLE
  d5 &lt;- parseCdrDoubleLE
  d6 &lt;- parseCdrDoubleLE
  return $ VelocityMsg d1 d2 d3 d4 d5 d6</code></pre>
<p>We can substitute this type into our code from previous parts, and use a timer to see how long it takes:</p>
<pre><code class="language-haskell">parseBareRecordsFromFile :: FilePath -&gt; IO ()
parseBareRecordsFromFile fp = do
  t1 &lt;- getCurrentTime
  input &lt;- BS.readFile fp
  (result, st) &lt;- runStateT (runParserT parseMcapFile' fp input) (TopicState HM.empty HM.empty HM.empty (HS.fromList ["/turtle1/cmd_vel"]) HS.empty)
  case result of
    Left e -&gt; print e
    Right recs -&gt; do
      let velMessages = fromMaybe [] $ HM.lookup ("/turtle1/cmd_vel", "geometry_msgs/msg/Twist") (tsMessages st)
      forM_ velMessages $ \(MessageData t1 t2 msg) -&gt; do
        parseVelResult &lt;- evalStateT (runParserT parseVelMsg "Velocity" msg) 0
        case parseVelResult of
          Left e -&gt; print e
          Right s -&gt; putStrLn $ "Parsed Velocity Message: " &lt;&gt; show t1 &lt;&gt; " " &lt;&gt; show t2 &lt;&gt; " " &lt;&gt; show s
      -- forM_ recs $ \(_, rec) -&gt; printRec rec
      -- print st
      t2 &lt;- getCurrentTime
      print (diffUTCTime t2 t1)</code></pre>
<p>This will print out all 22 messages, and it takes around 0.08 seconds. That’s not a long time, but it’s also not trivial for a bag spanning a few minutes with only 2 topics. Real ROS bags can contain a LOT more data, and this would slow down our current approach! Our goal in these next two articles is to find these velocity messages much, much faster, in a way that isn’t so dependent on the total bag size.</p>
<h2>New Record Types</h2>
<p>In the prior part, the only record types we used were Schema, Channel, Chunk and Message. We could basically ignore everything else, simply consuming it based on the content length counts.</p>
<p>To use indexing, we’ll have to acquaint ourselves with 4 new index types:</p>
<ol>
<li>Footer</li>
<li>Summary Offset</li>
<li>Chunk Index</li>
<li>Message Index</li>
</ol>
<p>These records form a kind of data chain that will lead us to the messages we want. From the <a href="https://mcap.dev/spec">MCAP specification</a> we can build types to represent the data in each of these records:</p>
<pre><code class="language-haskell">data FooterRec = FooterRec
  { summaryStart :: Word64
  , summaryOffsetStart :: Word64
  , summaryCrc :: Word32
  }
  deriving (Show, Eq)

data SummaryOffsetRec = SummaryOffsetRec
  { groupOpCode :: RecordType
  , groupStart :: Word64
  , groupLength :: Word64
  } deriving (Show, Eq)

data ChunkIndexRec = ChunkIndexRec
  { cirStartTime :: Word64
  , cirEndTime :: Word64
  , cirChunkStartOffset :: Word64
  , cirChunkLength :: Word64
  , cirMessageIndexOffsets :: HM.HashMap Word16 Word64
  , cirMessageIndexLength :: Word64
  , cirCompression :: ByteString
  , cirCompressedSize :: Word64
  , cirUncompressedSize :: Word64
  } deriving (Show, Eq)


data MessageIndexRec = MessageIndexRec
  { mirChannelId :: Word16
  , mirRecords :: [(Word64, Word64)]
  } deriving (Show, Eq)</code></pre>
<p>Of course, we need code to parse these records, similar to what we wrote for previous record types. These parsers simply make use of primitives we’ve already written, so we won’t dwell on the implementation details. The parsing functions are included at the end of this article in the appendix for completeness.</p>
<p>What we want to understand next is where these record types fit in the context of the whole file.</p>
<h2>File Structure Review</h2>
<p>We introduced the MCAP file structure in the first part of this series, but it’s worth revisiting now that we’re looking at indexing. Here’s the overall structure:</p>
<pre><code>&lt;Magic&gt;
&lt;Header&gt;
&lt;Data section&gt;
[&lt;Summary section&gt;]
[&lt;Summary Offset section&gt;]
&lt;Footer&gt;
&lt;Magic&gt;</code></pre>
<p>The magic bytes, header and footer are mandatory. Per the specification, it is possible for an “empty” file to omit all the other sections, but having a single message will require having the “Data Section”. But the summary sections are optional, only included to facilitate indexing. The code we’ll write here will assume they exist.</p>
<p>The “Summary section” can contain schema and channel records. These are identical copies of the schemas and channels we could find in the data section. But the summary section also contains chunk index records, as well as other index record types we won’t discuss. The chunk index records then point to particular chunks in the data section. What is distinct and important about the summary section is that all records are <strong>grouped by type</strong>. All schema records will be in a single consecutive block, all channel records will be in a single consecutive block, and so on. This sets us up to understand summary offsets.</p>
<p>Each summary offset record gives the location of one of these record type blocks. So we should expect one summary offset for schema records in the summary section, one for channels, and one for chunk indexes.</p>
<p>Finally, we should consider where “Message Index” records live. These are in the data section, and they follow after “Chunk” records. Each message index points us to records for a particular channel in the proceeding chunk.</p>
<p>I mentioned the “data chain” earlier, and now we can start to see how it works looking at the file structure and the fields of these particular records. </p>
<ol>
<li>The Footer points us to the summary offset section</li>
<li>The summary offsets point us to the summary section</li>
<li>We can read schemas and channels in the summary section</li>
<li>Then chunk index records will point to message index records</li>
<li>Message index records point us to messages for our channel</li>
</ol>
<p>That’s the high-level idea, but let’s go into a little more detail, highlighting the exact fields we’ll use to accomplish this.</p>
<h2>A Detailed Strategy</h2>
<p>Before we go any further, it’s important to understand <strong>file seeking</strong>. Given a numeric offset within a file and a file handle, it is very quick and efficient for us to tell the handle to “seek”, effectively moving it so that it will then parse the next location when we use an operation like <code>getChar</code>. Lots of the fields within our index records are file locations that we’ll seek to.</p>
<p>Knowing this, let’s lay out the details of exactly which fields we’ll use.</p>
<p>First, we use the Footer’s <code>summaryOffsetStart</code> to jump to the beginning of the summary offset section.</p>
<p>Second, we parse records in the summary offset section until we have a <code>SummaryOffsetRec</code> corresponding to Schemas, Channels, and Chunk Indexes.</p>
<p>Third, we use the <code>groupStart</code> field in each <code>SummaryOffsetRec</code> to jump to each of these summary sections in turn. We’ll parse the schemas, and then parse the channels, loading them into our <code>TopicState</code> like we’ve already been doing.</p>
<p>Step 4: We process each Chunk Index record. The <code>cirMessageIndexOffsets</code> field then provides us with a map from channel IDs to the locations of message index records. So we’ll loop through our desired channel IDs (populated in the third step) and, if this chunk has an index for them, we’ll jump to that message index.</p>
<p>Step 5: Process the message index record for our desired channel. The <code>mirRecords</code> field contains an array with one tuple per message in the corresponding chunk. The second value of this tuple is an offset, relative to where the data starts in the chunk. This is the only relative offset in our program…we need to bring forward the <code>cirChunkStartOffset</code> to make sense of this value.</p>
<p>Step 6: Using the chunk start offset and the offsets from the message index, jump to each <code>Message</code> record and parse it, storing its data like we have previously.</p>
<p>These steps will make more sense when we start writing code. We’ll write most of that code in the next part. First though, we have to come to grips with the fact that this whole jumping around business with “seek” is <strong>not</strong> consistent with how we’ve been parsing with Megaparsec so far!</p>
<h2>Megaparsec vs. Seeking</h2>
<p>We’ve written a lot of parsing functions so far with Megaparsec. However, this library assumes we are just walking straight through all the data. It does <strong>not</strong> supporting seeking at all, especially in the backwards direction.</p>
<p>Our indexing approach requires a lot of seeking, which makes sense. We hop around the file so that we don’t have to parse every individual byte.</p>
<p>The normal way to seek around a file in Haskell is to use a <code>Handle</code> object. Then we can use the <code>hSeek</code> function. It will occasionally also be important to use <code>hTell</code> to get our current position in the file. Then we can use <code>hGet</code> to read as many bytes as we specify into a ByteString.</p>
<pre><code class="language-haskell">import System.IO (hSeek, hTell)
import Data.ByteString.Lazy (hGet)

hSeek :: Handle -&gt; Integer -&gt; IO ()

hTell :: Handle -&gt; IO Integer

hGet :: Handle -&gt; Int -&gt; IO ByteString</code></pre>
<p>So instead of reading the file as a lazy bytestring upfront and passing that to Megaparsec, we want to open up a file handle and use that to move around the file and read bytes on demand. As long as <code>IO</code> is on our monad stack, we can do this.</p>
<p>So can we still use the <code>Parsec</code> functions we’ve written so far in this series? Yes, we can! The solution to this lies in the parameterization we employed <a href="https://mmhaskell.com/blog/2025/12/22/parsing-cdr-messages">last time</a> to use the same parsers with CDR. We just want to replace most of the <code>Parser</code> type signatures with something that captures the requirements of the function, such as <code>MonadParsec</code> and <code>MonadFail</code>. For example, here’s how we re-write the signature for <code>parseArray</code>:</p>
<pre><code class="language-haskell">parseArray :: (MonadFail m, MonadParsec Void ByteString m) =&gt; m (Word64, a) -&gt; m (Word64, [a])</code></pre>
<p>A special case occurs with our parsers for Schemas, Channels and Messages. These impact the <code>TopicState</code> value we have wrapped in a <code>StateT</code> layer of our <code>Parser</code> monad. We can also capture this with a monad class! However, we do need a different import for this:</p>
<pre><code class="language-haskell">import qualified Control.Monad.State as MS

parseSchema' :: (MonadParsec Void ByteString m, MS.MonadState TopicState m) =&gt; m (Word64, Record)</code></pre>
<p>Now we can access the state using <code>MS.get</code> and <code>MS.put</code> here.</p>
<p>So our general approach will be to use a different IO-based monad to parse out bytestrings containing record data. We can get all the data up front as a lazy bytestring since record headers contain the length of data we need. Then we pass that bytestring into a new invocation of a parsec-based monad like we did with CDR parsing.</p>
<p>Let’s see how we construct the new monad.</p>
<h2>Defining a New Monad</h2>
<p>Our previous monad looked like this:</p>
<pre><code class="language-haskell">type Parser a = ParsecT Void ByteString (StateT TopicState IO) a</code></pre>
<p>This parser has 3 features: parsing a bytestring, making stateful updates to the <code>TopicState</code>, and allowing <code>IO</code> for debugging.</p>
<p>Our desired monad doesn’t have parsec as a feature, but it still requires using <code>TopicState</code> and <code>IO</code>. However we have two new features.</p>
<p>First, we want to track a <code>Handle</code>. We will use this to seek around the file and read from it. This sounds stateful, but because <code>IO</code> handles the “state” part of the handle, we actually only need the handle wrapped in a <code>Reader</code> layer. This could give us a monad like this:</p>
<pre><code class="language-haskell">type MCAPReader a = StateT TopicState (ReaderT Handle IO) a</code></pre>
<p>We’ll add one more layer to this though. The <code>ParsecT</code> gave us a graceful layer for <code>MonadFail</code>, causing it to return a <code>Left</code> value instead of <code>Right</code> at the end. The <code>IO</code> monad also has this, but it’s not as graceful, causing a program crash. So we’ll add an <code>ExceptT</code> layer. This will allow us to use <code>throwError</code> on a custom error type so that we can have more graceful error handling.</p>
<pre><code class="language-haskell">data ValidationError = ValidationError String
  deriving (Show, Eq, Exception)

type MCAPReader a = StateT TopicState (ReaderT Handle (ExceptT ValidationError IO)) a</code></pre>
<p>We can further build another monad <code>MCAPParser</code>, which will put a <code>ParsecT</code> layer on top of <code>MCAPReader</code>. This will serve as the bridge between our new code and the old code.</p>
<pre><code class="language-haskell">type MCAPParser a = ParsecT Void ByteString (StateT TopicState (ReaderT Handle (ExceptT ValidationError IO)))  a</code></pre>
<p>Since it has <code>ParsecT</code> on top, we can call our existing functions with any function that has an <code>MCAPParser</code> signature! To see this in action, let’s start writing some utility functions for this monad.</p>
<h2>Monad Utilities</h2>
<p>So let’s wrap up this article by writing some functions for our new monad that we can use across the board. To start, we’ll write wrappers for <code>hSeek</code> and <code>hTell</code>, so that we don’t have to constantly use <code>ask</code> and <code>liftIO</code> in our code:</p>
<pre><code class="language-haskell">seek :: Integer -&gt; MCAPReader ()
seek pos = do
  hndl &lt;- ask
  liftIO (hSeek hndl AbsoluteSeek pos)

curPos :: MCAPReader Integer
curPos = do
  hndl &lt;- ask
  liftIO (hTell hndl)</code></pre>
<p>Next, let’s add some functions to handle failure cases. These resemble what we already have for <code>guard’</code>. We’ll use <code>throwError</code> on a <code>ValidationError</code> and include the current handle location as part of the error.</p>
<pre><code class="language-haskell">failParse :: String -&gt; MCAPReader ()
failParse str = throwError (ValidationError str)

mcapGuard :: String -&gt; Bool -&gt; MCAPReader ()
mcapGuard str cond = do
  p &lt;- curPos
  unless cond $ failParse (str &lt;&gt; " " &lt;&gt; show p)</code></pre>
<p>Now let’s write a function to run a “Parser” from our “Reader” monad, given a bytestring. We use <code>runParserT</code>, and if the result is <code>Left</code>, we throw a <code>ValidationError</code>. Otherwise we return the result. We’ll be able to pass any of our existing parsers to this function!</p>
<pre><code class="language-haskell">runMCAPParser :: MCAPParser (Word64, a) -&gt; ByteString -&gt; MCAPReader (Word64, a)
runMCAPParser parser dataBytes = do
  parseResult &lt;- runParserT parser "MCAP" dataBytes
  case parseResult of
    Left e -&gt; throwError (ValidationError (show e))
    Right result -&gt; return result</code></pre>
<p>Now let’s get into the meat of how we use this. We want to generalize the process of parsing a record in our <strong>reader</strong> monad. The issue with <code>runMCAPParser</code> is that it requires the Bytestring for the body for the record. So we need to actually load those characters. So we’ll wrap it with another function <code>parseNextRecord</code>. This function will take a general parser function as <code>(Word64 -&gt; MCAPParser (Word64, a))</code>.</p>
<pre><code class="language-haskell">parseNextRecord' :: RecordType -&gt; (Word64 -&gt; MCAPParser (Word64, a)) -&gt; MCAPReader (Word64, a)
parseNextRecord’ = undefined</code></pre>
<p>We need this extra <code>Word64</code> parameter strictly for <code>parseMessage</code>, which needs the content length to parse the body. However, our other record parsers can rely on this version, which ignores the parameter:</p>
<pre><code>parseNextRecord :: RecordType -&gt; MCAPParser (Word64, a) -&gt; MCAPReader (Word64, a)
parseNextRecord typ parser = parseNextRecord' typ (const parser)</code></pre>
<p>So how do we actually parse the record? Remember we’ll use <code>hGet</code> to actually parse a certain number of characters. We start with 1 character, for the record’s type. We’ll compare this to the input record type to make sure we’re parsing the expected record:</p>
<pre><code class="language-haskell">parseNextRecord' :: RecordType -&gt; (Word64 -&gt; MCAPParser (Word64, a)) -&gt; MCAPReader (Word64, a)
parseNextRecord' typ parser = do
  hndl &lt;- ask
  recordByte &lt;- liftIO $ BS.hGet hndl 1
  (n1, recordType) &lt;- runMCAPParser parseRecordType' recordByte
  mcapGuard ("Unexpeted record type: " &lt;&gt; show recordType &lt;&gt; " " &lt;&gt; show typ) (recordType == typ)
  ...</code></pre>
<p>Now we’ll do the same thing for the 8-byte record content length.</p>
<pre><code class="language-haskell">parseNextRecord' :: RecordType -&gt; (Word64 -&gt; MCAPParser (Word64, a)) -&gt; MCAPReader (Word64, a)
parseNextRecord' typ parser = do
  hndl &lt;- ask
  recordByte &lt;- liftIO $ BS.hGet hndl 1
  (n1, recordType) &lt;- runMCAPParser parseRecordType' recordByte
  mcapGuard ("Unexpeted record type: " &lt;&gt; show recordType &lt;&gt; " " &lt;&gt; show typ) (recordType == typ)
  lenBytes &lt;- liftIO $ BS.hGet hndl 8
  (n2, contentLength) &lt;- runMCAPParser parseUint64LE lenBytes
  ...</code></pre>
<p>Finally, we load the next bytes based on <code>contentLength</code> and pass them to our parser using <code>runMCAPParser</code> above!</p>
<pre><code class="language-haskell">parseNextRecord' :: RecordType -&gt; (Word64 -&gt; MCAPParser (Word64, a)) -&gt; MCAPReader (Word64, a)
parseNextRecord' typ parser = do
  hndl &lt;- ask
  recordByte &lt;- liftIO $ BS.hGet hndl 1
  (n1, recordType) &lt;- runMCAPParser parseRecordType' recordByte
  mcapGuard ("Unexpeted record type: " &lt;&gt; show recordType &lt;&gt; " " &lt;&gt; show typ) (recordType == typ)
  lenBytes &lt;- liftIO $ BS.hGet hndl 8
  (n2, contentLength) &lt;- runMCAPParser parseUint64LE lenBytes
  contentBytes &lt;- liftIO $ BS.hGet hndl (fromIntegral contentLength)
  (n3, record) &lt;- runMCAPParser (parser contentLength) contentBytes
  return (n1 + n2 + n3, record)</code></pre>
<p>An important note on <code>parseNextRecord</code> is that it depends on us already being in the right file location. So we’ll have to <code>seek</code> <strong>before</strong> calling this. But we’ll handle those details next time. </p>
<p>For now, let’s finish this article by writing a <code>run</code> function for our monad. This will dispatch to the <code>run</code> functions for our sub-monads like <code>execStateT</code> and <code>runExceptT</code>.</p>
<pre><code class="language-haskell">runMCAPReader :: Handle -&gt; TopicState -&gt; MCAPReader a -&gt; IO (Either ValidationError TopicState)
runMCAPReader handle ts parser = runExceptT (runReaderT (execStateT parser ts) handle)</code></pre>
<p>Now we have all the tools we’ll need for next time!</p>
<h2>Conclusion</h2>
<p>In this article, we learned more about the structure of an MCAP bag, particularly how various record types facilitate indexing so we can access desired topic data more quickly. We also built a new monad stack to use to help us jump around the file more quickly. In the next and final part of this ROS2 series, we’ll implement the remaining details of using these index records to find our topic data!</p>
<p>Up til now, this series has focused on parsing mechanics. But this article centered more on monad composition, which can be a tricky topic. To learn more about monads, including how to build useful monad stacks, you can take a look at 2 of our courses. <a href="https://academy.mondaymorninghaskell.com/p/making-sense-of-monads">Making Sense of Monads</a> is geared more towards beginners, and <a href="https://academy.mondaymorninghaskell.com/p/effectful-haskell">Effectful Haskell</a> is a more advanced course for those trying to take their understanding of monads to the next level. You can get both of these by purchasing our <a href="https://academy.mondaymorninghaskell.com/p/effects-bundle">Effects Bundle</a>!</p>
<h2>Appendix: Parsing New Record Types</h2>
<p>Note how these use the generalized monad structure we discussed in this article (<code>MonadParsec</code>, <code>MonadFail</code>). </p>
<pre><code class="language-haskell">parseFooter' :: (MonadParsec Void ByteString m) =&gt; m (Word64, FooterRec)
parseFooter' = do
  (n1, summaryStart) &lt;- parseUint64LE
  (n2, summaryOffsetStart) &lt;- parseUint64LE
  (n3, summaryCrc) &lt;- parseUint32LE
  return (n1 + n2 + n3, FooterRec summaryStart summaryOffsetStart summaryCrc)

parseSummaryOffset :: (MonadFail m, MonadParsec Void ByteString m) =&gt; m (Word64, SummaryOffsetRec)
parseSummaryOffset = do
  (n1, op) &lt;- parseRecordType'
  (n2, gs) &lt;- parseUint64LE
  (n3, gl) &lt;- parseUint64LE
  return (n1 + n2 + n3, SummaryOffsetRec op gs gl)

parseChunkIndex :: (MonadFail m, MonadParsec Void ByteString m) =&gt; m (Word64, ChunkIndexRec)
parseChunkIndex = do
  (n1, startT) &lt;- parseTimestamp
  (n2, endT) &lt;- parseTimestamp
  (n3, chunkStart) &lt;- parseUint64LE
  (n4, chunkLen) &lt;- parseUint64LE
  (n5, offsets) &lt;- parseMap parseUint16LE parseUint64LE
  (n6, indexLength) &lt;- parseUint64LE
  (n7, compression) &lt;- parseString
  (n8, compressedSize) &lt;- parseUint64LE
  (n9, uncompressedSize) &lt;- parseUint64LE
  let totalLen = n1 + n2 + n3 + n4 + n5 + n6 + n7 + n8 + n9
  let rec = ChunkIndexRec startT endT chunkStart chunkLen offsets indexLength compression compressedSize uncompressedSize
  return (totalLen, rec)

parseMessageIndex' :: (MonadFail m, MonadParsec Void ByteString m) =&gt; m (Word64, MessageIndexRec)
parseMessageIndex' = do
  (n1, cid) &lt;- parseUint16LE
  (n2, recs) &lt;- parseArray (parseTuple parseUint64LE parseUint64LE)
  return (n1 + n2, MessageIndexRec cid recs)</code></pre></div>
    </summary>
    <updated>2025-12-22T12:30:00Z</updated>
    <published>2025-12-22T12:30:00Z</published>
    <author>
      <name>James Bowen</name>
    </author>
    <source>
      <id>https://mmhaskell.com/blog/</id>
      <link href="https://mmhaskell.com/blog/" rel="alternate" type="text/html"/>
      <link href="https://mmhaskell.com/blog?format=rss" rel="self" type="application/rss+xml"/>
      <title>Blog - Monday Morning Haskell</title>
      <updated>2025-10-05T20:31:39Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-US">
    <id>http://blog.ezyang.com/?p=10605</id>
    <link href="http://blog.ezyang.com/2025/12/code-review-as-human-alignment-in-the-era-of-llms/" rel="alternate" type="text/html"/>
    <link href="http://blog.ezyang.com/2025/12/code-review-as-human-alignment-in-the-era-of-llms/#comments" rel="replies" type="text/html"/>
    <link href="http://blog.ezyang.com/2025/12/code-review-as-human-alignment-in-the-era-of-llms/feed/atom/" rel="replies" type="application/atom+xml"/>
    <title xml:lang="en-US">Code review as human alignment, in the era of LLMs</title>
    <summary xml:lang="en-US">I've recently been doing a lot of both submitting and reviewing pull requests to PyTorch that were authored with substantial LLM assistance. This is a big difference from earlier this year, where it was clear LLMs worked well for greenfield projects but the code was too hopelessly sloppy for a production codebase. Here are my [â€¦]</summary>
    <content type="xhtml" xml:lang="en-US"><div xmlns="http://www.w3.org/1999/xhtml"><div class="document">



<p>I've recently been doing a lot of both submitting and reviewing pull requests to PyTorch that were authored with substantial LLM assistance.  This is a big difference from earlier this year, where it was clear LLMs worked well for greenfield projects but the code was too hopelessly sloppy for a production codebase.  Here are <a class="reference external" href="https://github.com/pytorch/pytorch/pulls?q=is%3Apr+author%3Aezyang+claude+code+is%3Aclosed+label%3AMerged">my merged PRs</a> that mention claude code in their description; Jason Ansel has also had a similar experience (<a class="reference external" href="https://fb.workplace.com/groups/257735836456307/posts/997873152442568">Meta only link</a>, here is the <a class="reference external" href="https://gist.github.com/ezyang/ece0a1976366a959372740b3173da816">list of issues</a> he referenced in his writeup).  There already has been increasing discourse (<a class="reference external" href="https://simonwillison.net/2025/Dec/18/code-proven-to-work/">Simon Willison</a>, <a class="reference external" href="https://discourse.llvm.org/t/our-ai-policy-vs-code-of-conduct-and-vs-reality/88300?utm_source=chatgpt.com">LLVM</a>) on how code review should adapt to this new era of LLMs.  My contribution to this discourse is this: within teams, code review should change to being primarily be a <strong>human alignment</strong> mechanism.</p>
<p>Here is a simple example: it is well known that LLMs are prone to generating overly defensive code: e.g., they will be constantly sprinkling <tt class="docutils literal"><span class="pre">try...catch</span></tt> everywhere or testing if a variable is some type when system invariants imply that it should always be that type.  If someone sends me a PR with these problems, I am not commenting on these problems solely because I want them to be fixed. If that's all I cared about, I could have just fed my comments directly to claude code. The real problem is that the human who was operating the LLM didn't agree with me that this defensive code was bad, and the point of the review is to <strong>align</strong> them with me on what is overly defensive versus not.  In the most trivial cases, maybe the engineer didn't read the LLM output, in which case the remedy is to make them actually read the code.  But sometimes real human work has to happen; for example, maybe there is a global system invariant that one has to understand to know if the defensiveness is necessary or not.  If we agree about the global system invariants, there's no reason the code review has to go through me: the original code author can just instruct the LLM to fix problems and keep me out of the loop until they have aligned the LLM output to themselves--at which point we should do the more expensive human to human alignment.  The ideal is that I don't need to ever write review comments about mechanical problems, because they have already been fixed by the original author ahead of time.</p>
<p>Conversely, when I am putting up an LLM generated PR for human review, I am trying to transmit higher level information. How does the new code work? What do I need to know about the existing system to understand this code?  This doesn't even have to be in the PR description: if the LLM proposes a fix that I myself don't understand, or seems difficult to understand, I will simply instruct it to try it a different way, until the resulting diff is obviously correct.  Tokens are cheap: we should expect <em>more</em> out of the author of code, because the cost of generating these PRs has gone way down. Similarly, I am willing to throw out the code and start again; you don't have to feel bad about wasting my time (I didn't type it! I spent my time understanding the problem, and none of that is regretted.)</p>
<p>There is a lot of scaremongering about how engineers who don't pick up AI tools will be left behind.  My take on this is that there a number of different skills that make up what it means to be a good software engineer, and it is clear that LLM coding, even today, is clearly reweighting the relative importance of these skills. I care a lot more about your ability to read code, reason about the big picture, communicate clearly and to have good taste, than I care about your ability to <em>mechanically</em> write code. There is an archetype of junior engineer who is not that good at coding but very good at the softer, higher level skills, and I think they will be very valuable in this new world order.  Conversely, I think going forward I will have substantially less patience if I have to keep telling you the same things over and over, because I just don't value raw "ability to code" as much anymore.  My ideal state is like that with long time senior teammates: I can trust that they have made good low level decisions, and I can focus on understanding the bigger picture and updating my mental model of how the system works.</p>
<p>Today's LLMs have no memory: they have to rediscover everything in the system from first principles every time they are run.  The purpose of the humans, of the team, is to collectively maintain a shared vision of what, platonically, the system should do.  I want code review to reconfigure itself around this purpose.</p>
</div></div>
    </content>
    <updated>2025-12-20T22:48:45Z</updated>
    <published>2025-12-20T22:48:45Z</published>
    <category scheme="http://blog.ezyang.com" term="AI Coding"/>
    <author>
      <name>Edward Z. Yang</name>
      <uri>http://ezyang.com</uri>
    </author>
    <source>
      <id>http://blog.ezyang.com/feed/atom/</id>
      <link href="http://blog.ezyang.com" rel="alternate" type="text/html"/>
      <link href="http://blog.ezyang.com/feed/atom/" rel="self" type="application/atom+xml"/>
      <subtitle xml:lang="en-US">the arc of software bends towards understanding</subtitle>
      <title xml:lang="en-US">ezyangâ€™s blog</title>
      <updated>2025-12-20T22:55:05Z</updated>
    </source>
  </entry>

  <entry>
    <id>http://haskell.org/ghc/blog/20251219-ghc-9.14.1-released.html</id>
    <link href="http://haskell.org/ghc/blog/20251219-ghc-9.14.1-released.html" rel="alternate" type="text/html"/>
    <title>GHC 9.14.1 is now available</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><h1>GHC 9.14.1 is now available</h1>
<h4 class="text-muted">zubin - 2025-12-19</h4>

<p>The GHC developers are very pleased to announce the release of GHC 9.14.1.
Binary distributions, source distributions, and documentation are available at
<a href="https://downloads.haskell.org/ghc/9.14.1">downloads.haskell.org</a>.</p>
<p>GHC 9.14 brings a number of new features and improvements, including:</p>
<ul>
<li><p>Significant improvements in specialisation:</p>
<ul>
<li>The <code>SPECIALISE</code> pragma now allows use of type application syntax</li>
<li>The <code>SPECIALISE</code> pragma can be used to specialise for expression arguments
as well as type arguments.</li>
<li>Specialisation is now considerably more reliable in the presence of
<code>newtype</code>s</li>
</ul></li>
<li><p>Significant GHCi improvements including:</p>
<ul>
<li>Correctness and performance improvements in the bytecode interpreter</li>
<li>Features in the GHCi debugger</li>
<li>Support for multiple home units in GHCi</li>
</ul></li>
<li><p>Implementation of the <a href="https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0682-explicit-level-imports.rst">Explicit Level Imports proposal</a></p></li>
<li><p><code>RequiredTypeArguments</code> can now be used in more contexts</p></li>
<li><p>Greatly improved <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/25030">SSE/AVX2 support</a> in the x86 native code generator backend</p></li>
<li><p>Initial native code generator support for LoongArch</p></li>
<li><p>The WebAssembly backend now supports <a href="https://www.tweag.io/blog/2025-04-17-wasm-ghci-browser/">evaluation via the interpreter</a>, allowing
GHCi and TemplateHaskell evaluation, including <code>foreign import javascript</code>
usage from within the browser</p></li>
<li><p>A new primop <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26218"><code>annotateStack#</code></a> for pushing arbitrary data onto the call stack
for later extraction when decoding stack traces</p></li>
<li><p><a href="https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0516-incomplete-record-selectors.rst"><code>-Wincomplete-record-selectors</code></a> is now part of <code>-Wall</code>. Libraries compiled
with <code>-Werror</code> may need adjustment.</p></li>
<li><p>A major update of the Windows toolchain</p></li>
<li><p>Improved compatibility with <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26166">macOS Tahoe</a></p></li>
<li><p>â€¦ and many more</p></li>
</ul>
<p>A full accounting of changes can be found in the <a href="https://downloads.haskell.org/ghc/9.14.1/docs/users_guide/9.14.1-notes.html">release notes</a>.
See the <a href="https://gitlab.haskell.org/ghc/ghc/-/wikis/migration/9.14">migration guide</a> for guidance on migrating programs to this release.</p>
<p>Note that while this release makes many improvements in the specialisation
optimisation, polymorphic specialisation remains disabled by default in the
release due to concern over regressions of the sort identified in
<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/26329"><span>#26329</span></a>. Users needing more aggressive specialisation can explicitly
enable this feature with the <code>-fpolymorphic-specialisation</code> flag. Depending
upon our experience with 9.14.1, we may enable this feature by default in a
later minor release.</p>
<p>GHC development is sponsored by:</p>
<ul>
<li><a href="https://juspay.com/">Juspay</a></li>
<li><a href="https://qbaylogic.com/">QBayLogic</a></li>
<li><a href="https://www.channable.com/">Channable</a></li>
<li><a href="https://haskell.foundation/">Haskell Foundation</a></li>
<li><a href="https://serokell.io/">Serokell</a></li>
<li><a href="https://well-typed.com/">Well-Typed</a></li>
<li><a href="https://www.tweag.io/">Tweag</a></li>
<li><a href="https://www.dotcom-monitor.com/">Dotcom-Monitor</a></li>
<li><a href="https://www.loadview-testing.com/">LoadView</a></li>
<li><a href="https://webhostingbuddy.com/">Web Hosting Buddy</a></li>
<li><a href="https://www.findmyelectric.com/">Find My Electric</a></li>
<li><a href="https://www.sc.com">Standard Chartered</a></li>
<li><a href="https://upcloud.com">UpCloud</a></li>
</ul>
<p>We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprises this release.</p>
<p>As always, do give this release a try and open a <a href="https://gitlab.haskell.org/ghc/homepage/-/issues/new">ticket</a> if you see
anything amiss.</p></div>
    </summary>
    <updated>2025-12-19T00:00:00Z</updated>
    <published>2025-12-19T00:00:00Z</published>
    <author>
      <name>ghc-devs</name>
    </author>
    <source>
      <id>http://haskell.org/ghc</id>
      <link href="http://haskell.org/ghc" rel="alternate" type="text/html"/>
      <link href="http://haskell.org/ghc/rss.xml" rel="self" type="application/rss+xml"/>
      <title>GHC Developer blog</title>
      <updated>2026-03-27T00:00:00Z</updated>
    </source>
  </entry>

  <entry>
    <id>https://www.parsonsmatt.org/2025/12/17/the_subtle_footgun_of_tvar_(map____).html</id>
    <link href="https://www.parsonsmatt.org/2025/12/17/the_subtle_footgun_of_tvar_(map____).html" rel="alternate" type="text/html"/>
    <title>The Subtle Footgun of TVar (Map _ _)</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><blockquote>
  <p>How coarse-grained STM containers can livelock under load</p>
</blockquote>

<ul>
  <li>Edit history:
    <ul>
      <li>12-19: <a href="https://discourse.haskell.org/t/the-subtle-footgun-of-tvar-map/13429/3">@teofilC on the Haskell Discourse</a> remarked a place where <code class="language-plaintext highlighter-rouge">TVar (Map _ _)</code> may be appropriate - I’ve modified the post to incorporate this.</li>
    </ul>
  </li>
</ul>

<p>Software Transactional Memory (STM) is one of Haskell’s crown jewels.
The promise is easy, lock-free concurrency with guaranteed transactional semantics and great performance.
Used correctly, you get all of these benefits.
However, concurrency is fundamentally difficult, and STM has a failure mode: livelock.</p>

<p>Livelock is when the program is repeatedly retrying transactions - working furiously and making no progress.
Livelock is subtle and extremely difficult to diagnose.
The problem of livelock is often simply “performance is really bad and seems to get worse with more concurrency.”
Avoiding livelock proactively will pay massive dividends in the performance of your concurrent systems, as well as the time spent diagnosing and fixing them later on.</p>

<h1 id="avoiding-livelock-on-shared-containers">Avoiding Livelock on Shared Containers</h1>

<p>A common causes of livelock is fortunately discoverable with simple text search:</p>

<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">uhoh</span> <span class="o">::</span> <span class="kt">TVar</span> <span class="p">(</span><span class="kt">Map</span> <span class="n">k</span> <span class="p">(</span><span class="kt">TVar</span> <span class="n">v</span><span class="p">))</span> <span class="o">-&gt;</span> <span class="kt">STM</span> <span class="nb">()</span>

<span class="n">oops</span> <span class="o">::</span> <span class="kt">TVar</span> <span class="p">(</span><span class="kt">HashMap</span> <span class="n">k</span> <span class="n">v</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kt">STM</span> <span class="nb">()</span>

<span class="n">ohno</span> <span class="o">::</span> <span class="kt">TVar</span> <span class="p">(</span><span class="kt">Set</span> <span class="n">v</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kt">STM</span> <span class="nb">()</span>
</code></pre></div></div>

<p>A <code class="language-plaintext highlighter-rouge">TVar</code> to a container data structure, like <code class="language-plaintext highlighter-rouge">Map k v</code> or <code class="language-plaintext highlighter-rouge">Set a</code> or <code class="language-plaintext highlighter-rouge">[a]</code> or <code class="language-plaintext highlighter-rouge">Seq a</code>, is a common source of livelock.
Any change to the <code class="language-plaintext highlighter-rouge">Map</code> invalidates any transaction that is reading from that <code class="language-plaintext highlighter-rouge">Map</code>, even if it is reading at a totally different key or value.
If the <code class="language-plaintext highlighter-rouge">Map</code> is being concurrently updated, you are nearly guaranteed to run into performance problems.</p>

<p>Fortunately, you can easily avoid this pattern, thanks to the <code class="language-plaintext highlighter-rouge">stm-containers</code> library.</p>

<h1 id="stm-containers"><code class="language-plaintext highlighter-rouge">stm-containers</code></h1>

<p>The <a href="https://nikita-volkov.github.io/stm-containers/"><code class="language-plaintext highlighter-rouge">stm-containers</code></a> package was released over ten years ago.
<code class="language-plaintext highlighter-rouge">stm-containers</code> allows you to have a structure <em>similar to</em> a <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> or <code class="language-plaintext highlighter-rouge">TVar (Map k (TVar v))</code> with the added benefit that it actually scales up with increased concurrency, avoiding the dreaded livelock.
Modifications to the map impact a significantly smaller portion of the map structure, so you’re way less likely to run into livelock from concurrent updates.</p>

<p>Switching is really easy.
The API is mostly the same as the <code class="language-plaintext highlighter-rouge">containers</code> library, but modifications are effectful in <code class="language-plaintext highlighter-rouge">STM ()</code> instead of returning the new data structure.</p>

<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">StmMap</span><span class="o">.</span><span class="n">lookup</span>
    <span class="o">::</span> <span class="p">(</span><span class="kt">Hashable</span> <span class="n">k</span><span class="p">)</span>
    <span class="o">=&gt;</span> <span class="n">k</span>
    <span class="o">-&gt;</span> <span class="kt">StmMap</span><span class="o">.</span><span class="kt">Map</span> <span class="n">k</span> <span class="n">v</span>
    <span class="o">-&gt;</span> <span class="kt">STM</span> <span class="p">(</span><span class="kt">Maybe</span> <span class="n">v</span><span class="p">)</span>

<span class="kt">HashMap</span><span class="o">.</span><span class="n">lookup</span>
    <span class="o">::</span> <span class="p">(</span><span class="kt">Hashable</span> <span class="n">k</span><span class="p">)</span>
    <span class="o">=&gt;</span> <span class="n">k</span>
    <span class="o">-&gt;</span> <span class="kt">HashMap</span> <span class="n">k</span> <span class="n">v</span>
    <span class="o">-&gt;</span> <span class="kt">Maybe</span> <span class="n">v</span>

<span class="kt">StmMap</span><span class="o">.</span><span class="n">insert</span>
    <span class="o">::</span> <span class="p">(</span><span class="kt">Hashable</span> <span class="n">k</span><span class="p">)</span>
    <span class="o">=&gt;</span> <span class="n">v</span> <span class="c1">-- note that the value is first, not the key</span>
    <span class="o">-&gt;</span> <span class="n">k</span>
    <span class="o">-&gt;</span> <span class="kt">StmMap</span><span class="o">.</span><span class="kt">Map</span> <span class="n">k</span> <span class="n">v</span>
    <span class="o">-&gt;</span> <span class="kt">STM</span> <span class="nb">()</span>

<span class="kt">HashMap</span><span class="o">.</span><span class="n">insert</span>
    <span class="o">::</span> <span class="p">(</span><span class="kt">Hashable</span> <span class="n">k</span><span class="p">)</span>
    <span class="o">=&gt;</span> <span class="n">k</span>
    <span class="o">-&gt;</span> <span class="n">v</span>
    <span class="o">-&gt;</span> <span class="kt">HashMap</span> <span class="n">k</span> <span class="n">v</span>
    <span class="o">-&gt;</span> <span class="kt">HashMap</span> <span class="n">k</span> <span class="n">v</span>
</code></pre></div></div>

<p>Here’s an example of using <code class="language-plaintext highlighter-rouge">stm-containers</code> in practice, compared to a <code class="language-plaintext highlighter-rouge">TVar Map</code>.</p>

<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.Map</span> <span class="k">as</span> <span class="n">Map</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">StmContainers.Map</span> <span class="k">as</span> <span class="n">StmMap</span>

<span class="c1">-- old:</span>
<span class="n">doStuff</span> <span class="o">::</span> <span class="kt">TVar</span> <span class="p">(</span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">String</span> <span class="kt">Int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kt">String</span> <span class="o">-&gt;</span> <span class="kt">STM</span> <span class="kt">Int</span>
<span class="n">doStuff</span> <span class="n">tmap</span> <span class="n">str</span> <span class="o">=</span> <span class="kr">do</span>
    <span class="n">map</span> <span class="o">&lt;-</span> <span class="n">readTVar</span> <span class="n">tmap</span>
    <span class="kr">let</span> <span class="n">mval</span> <span class="o">=</span> <span class="kt">Map</span><span class="o">.</span><span class="n">lookup</span> <span class="n">str</span> <span class="n">map</span>
    <span class="kr">let</span> <span class="n">newVal</span> <span class="o">=</span> <span class="n">maybe</span> <span class="mi">0</span> <span class="p">(</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="n">mval</span>
    <span class="n">writeTVar</span> <span class="p">(</span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span> <span class="n">str</span> <span class="n">newVal</span> <span class="n">map</span><span class="p">)</span>
    <span class="n">pure</span> <span class="n">newVal</span>

<span class="c1">-- new:</span>
<span class="n">doStuff</span> <span class="o">::</span> <span class="kt">StmMap</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">String</span> <span class="kt">Int</span> <span class="o">-&gt;</span> <span class="kt">String</span> <span class="o">-&gt;</span> <span class="kt">STM</span> <span class="kt">Int</span>
<span class="n">doStuff</span> <span class="n">tmap</span> <span class="n">str</span> <span class="o">=</span> <span class="kr">do</span>
    <span class="n">mval</span> <span class="o">&lt;-</span> <span class="kt">StmMap</span><span class="o">.</span><span class="n">lookup</span> <span class="n">str</span> <span class="n">tmap</span>
    <span class="kr">let</span> <span class="n">newVal</span> <span class="o">=</span> <span class="n">maybe</span> <span class="mi">0</span> <span class="p">(</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="n">mval</span>
    <span class="kt">StmMap</span><span class="o">.</span><span class="n">insert</span> <span class="n">newVal</span> <span class="n">str</span> <span class="n">tmap</span>
    <span class="n">pure</span> <span class="n">newVal</span>
</code></pre></div></div>

<p>The library also offers an interesting <code class="language-plaintext highlighter-rouge">focus</code> function, which allows you to combine operations at a single key into a single operation.</p>

<p>If you’re sharing a <code class="language-plaintext highlighter-rouge">Map</code> or <code class="language-plaintext highlighter-rouge">Set</code>-like container across many threads, then you almost certainly want to just use <code class="language-plaintext highlighter-rouge">stm-containers</code>.
If you stop reading here, and merely use <code class="language-plaintext highlighter-rouge">stm-containers</code> by default, then you’ll avoid a lot of pain without having to invest much time or energy.
<code class="language-plaintext highlighter-rouge">stm-containers</code> is faster and safer than a <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code>.</p>

<p><a href="https://github.com/MercuryTechnologies/hs-temporal-sdk/pull/279">In this PR to the <code class="language-plaintext highlighter-rouge">hs-temporal-sdk</code> library</a>, I mechanically translated a few <code class="language-plaintext highlighter-rouge">TVar (HashMap _ _)</code> to <code class="language-plaintext highlighter-rouge">stm-containers</code> datatypes.
We were observing degraded performance with increased concurrency, some of which were causing an ordinary 10 minute test suite run to time out after 50 minutes.
By switching from <code class="language-plaintext highlighter-rouge">TVar (HashMap _ _)</code> to <code class="language-plaintext highlighter-rouge">stm-containers</code>, the problem was completely fixed - performance remained consistent with increased concurrency.</p>

<h1 id="when-should-i-not-use-stm-containers">When should I not use <code class="language-plaintext highlighter-rouge">stm-containers</code>?</h1>

<p>By default, you should use <code class="language-plaintext highlighter-rouge">stm-containers</code>.
It is very fast, simple to use, and has no downsides (aside from the downsides inherent to <code class="language-plaintext highlighter-rouge">STM</code> vs <code class="language-plaintext highlighter-rouge">MVar</code> or <code class="language-plaintext highlighter-rouge">IORef</code> based shared references).</p>

<p>A <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> isn’t <em>always</em> going to be dangerous.
And that’s part of the problem - there are some places where <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> won’t be slow or break.
These rely on <em>non-local assumptions</em> about your code structure - you <em>cannot</em> enforce this easily without writing complicated wrappers that enforce encapsulation around how the shared reference is used.
Writing code that works great with a <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> in the small is quite easy, but guaranteeing that code won’t break as it scales is challenging.</p>

<p>There are many situations where going from <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> to <code class="language-plaintext highlighter-rouge">StmMap.Map k v</code> will make a huge improvement.</p>

<p>There is one known situation where an <code class="language-plaintext highlighter-rouge">StmMap.Map k v</code> may run into a significant performance problem - if you need to get a snapshot of the <code class="language-plaintext highlighter-rouge">Map</code> atomically, <code class="language-plaintext highlighter-rouge">StmMap.Map k v</code> requires reading <code class="language-plaintext highlighter-rouge">O(n log n)</code> <code class="language-plaintext highlighter-rouge">TVar</code>s, and <a href="https://gitlab.haskell.org/ghc/ghc/-/issues/24410">GHC currently is quadratic in the count of <code class="language-plaintext highlighter-rouge">TVar</code>s in a transaction</a>.
So if you find yourself doing <code class="language-plaintext highlighter-rouge">listT :: StmMap.Map k v -&gt; ListT STM [(k, v)]</code>, then you may want to consider an alternative structure.</p>

<h1 id="what-about-ioref-map-k-v">What about <code class="language-plaintext highlighter-rouge">IORef (Map k v)</code>?</h1>

<p>An <code class="language-plaintext highlighter-rouge">IORef (Map k v)</code> eliminates the possibility of livelock.
However, the <code class="language-plaintext highlighter-rouge">IORef</code> structure is not suitable for many writers.
To ensure a consistent view of the <code class="language-plaintext highlighter-rouge">Map</code>, you need to use <code class="language-plaintext highlighter-rouge">atomicModifyIORef</code>.
While a thread is doing an <code class="language-plaintext highlighter-rouge">atomicModifyIORef</code>, all other writes are blocked to the reference.
This means that an <code class="language-plaintext highlighter-rouge">IORef (Map k v)</code> is suitable if there are relatively few writes to the <code class="language-plaintext highlighter-rouge">IORef</code>, and the <code class="language-plaintext highlighter-rouge">IORef</code> write is mostly a complete replacement.
If you have many writers doing small writes, then threads will queue up and block on updating the entire <code class="language-plaintext highlighter-rouge">Map</code>.</p>

<p>In briefer code snippets,</p>

<ul>
  <li>Good: <code class="language-plaintext highlighter-rouge">atomicModifyIORef' (\oldMap -&gt; (Map.union newMap oldMap, ()))</code></li>
  <li>Bad: <code class="language-plaintext highlighter-rouge">atomicModifyIORef' (\oldMap -&gt; (Map.insert k v oldMap, ()))</code></li>
</ul>

<p><code class="language-plaintext highlighter-rouge">atomicModifyIORef</code> also requires the updating action to be <em>pure</em>.
If you need the modification to be effectful, <em>and</em> have a consistent view of the data, then you need to use either <code class="language-plaintext highlighter-rouge">stm-containers</code> or an <code class="language-plaintext highlighter-rouge">MVar</code>.</p>

<p><a href="https://github.com/parsonsmatt/prometheus-haskell/pull/1">In this PR to the <code class="language-plaintext highlighter-rouge">prometheus-haskell</code></a> library, I demonstrate that an <code class="language-plaintext highlighter-rouge">IORef (Map k v)</code> has significantly worse performance than an <code class="language-plaintext highlighter-rouge">stm-containers</code> <code class="language-plaintext highlighter-rouge">Map k v</code> as concurrency scales up.
This change made a big difference to the performance of our metric collection code.
<code class="language-plaintext highlighter-rouge">IORef (Map k v)</code> is not suitable because there are many frequent writes that are only concerned with a single <code class="language-plaintext highlighter-rouge">k</code>, and reading is done non-atomically - the value of <code class="language-plaintext highlighter-rouge">k0</code> changing does not impact the value of <code class="language-plaintext highlighter-rouge">k1</code>.</p>

<h1 id="what-about-mvar-map-k-v">What about <code class="language-plaintext highlighter-rouge">MVar (Map k v)</code>?</h1>

<p>An <code class="language-plaintext highlighter-rouge">MVar (Map k v)</code> avoids the problem of livelock, but introduces another potential problem: deadlock.
An <code class="language-plaintext highlighter-rouge">MVar</code> is a <em>locking</em> concurrency mechanism, where you can block other threads by <code class="language-plaintext highlighter-rouge">takeMVar</code> and making the <code class="language-plaintext highlighter-rouge">MVar</code> empty.
This allows updating threads to lock the value, update the <code class="language-plaintext highlighter-rouge">MVar</code>, and unlock the value.
However, if your code fails to fill up an <code class="language-plaintext highlighter-rouge">MVar</code>, or if two threads are waiting on mutually held <code class="language-plaintext highlighter-rouge">MVar</code>s, then your code cannot make progress, and will be deadlocked.</p>

<p>One advantage that this has to a <code class="language-plaintext highlighter-rouge">TVar</code> is fairness.
Threads that attempt to read or take from an <code class="language-plaintext highlighter-rouge">MVar</code> are enqueued, and guaranteed to operate in a first-in-first-out manner.
<code class="language-plaintext highlighter-rouge">TVar</code>, on the other hand, will start every thread at once, and the first one to complete wins while everyone else must retry.</p>

<p>If you have multiple <code class="language-plaintext highlighter-rouge">MVar</code> variables that you want to coordinate on, then you are increasing the risk of deadlock.</p>

<h1 id="when-is-tvar-map-k-v-safe">When is <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> safe?</h1>

<p>The fundamental problem with <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> is that any write to the <code class="language-plaintext highlighter-rouge">TVar</code> will invalidate and retry every transaction that reads from it.</p>

<p>For this variable to be safe, you need a single thread performing updates, and other threads are only doing reads.
Additionally, those updates should ideally be relatively rare - if the thread is constantly updating single keys in the map, that will invalidate transactions frequently.
Instead, you’ll want to batch updates to the <code class="language-plaintext highlighter-rouge">Map</code> and perform a whole replacement at once.</p>

<p>However, this is exactly the same limitation as <code class="language-plaintext highlighter-rouge">IORef (Map k v)</code> or an <code class="language-plaintext highlighter-rouge">MVar (Map k v)</code>.
If you only have a single <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> involved in your <code class="language-plaintext highlighter-rouge">STM</code> transactions, then you can simply switch to an <code class="language-plaintext highlighter-rouge">IORef</code> or <code class="language-plaintext highlighter-rouge">MVar</code> and enjoy increased performance.
If you have multiple <code class="language-plaintext highlighter-rouge">TVar</code> in your transaction, then you are at risk of livelock, and should use <code class="language-plaintext highlighter-rouge">stm-containers</code>.</p>

<p>If you <em>really need</em> <code class="language-plaintext highlighter-rouge">TVar</code> for <code class="language-plaintext highlighter-rouge">STM</code> transactionality, then I would highly recommend wrapping the <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code> in a <code class="language-plaintext highlighter-rouge">newtype</code> that forbids write operations, and only expose the underlying <code class="language-plaintext highlighter-rouge">TVar</code> in a manner that allows for a single thread to do infrequent writes.
This will avoid contention and livelock.</p>

<h1 id="when-is-tvar-map-k-tvar-v-safe">When is <code class="language-plaintext highlighter-rouge">TVar (Map k (TVar v))</code> safe?</h1>

<p><code class="language-plaintext highlighter-rouge">TVar (Map k (TVar v))</code> is slightly better than the above.
With this, any write to the top-level <code class="language-plaintext highlighter-rouge">TVar</code> will invalidate the transaction, but writes to the value <code class="language-plaintext highlighter-rouge">TVar</code>s will only cause contention on other transactions that are attempting to write to it.</p>

<p>For this to be safe, you must have a single thread that updates the <code class="language-plaintext highlighter-rouge">Map</code> <em>structure</em> (ie adding or removing keys).
Many threads can write on the map values and only experience the usual contention on a single variable.</p>

<p>However, this runs into the same problem as above: if you can safely refactor this to <code class="language-plaintext highlighter-rouge">IORef (Map k (TVar v))</code> (or <code class="language-plaintext highlighter-rouge">MVar</code>), then you are safe, otherwise, you are at risk of livelock and you should switch to <code class="language-plaintext highlighter-rouge">stm-containers</code>.</p>

<h1 id="the-semantics-of-a-reference-type">The Semantics of a Reference Type</h1>

<p>I love arguing semantics.
What else are we going to argue about, syntax?</p>

<p>If I say <code class="language-plaintext highlighter-rouge">IORef a</code>, I mean:</p>

<blockquote>
  <p>This is a reference to an <code class="language-plaintext highlighter-rouge">a</code>.
The reference may be read or written to from any thread.
The reference is very fast for these operations.
However, the only operation for atomic consistency is <code class="language-plaintext highlighter-rouge">atomicModifyIORef</code>.
So do not expect to be able to do transactions or blocking with this!</p>
</blockquote>

<p>We have a <em>very fast</em> reference with very few guarantees.
This is suitable for sharing information among many threads, where updates don’t happen very often, and updates are fast and pure.</p>

<p>Meanwhile, if I say <code class="language-plaintext highlighter-rouge">MVar a</code>, I mean:</p>

<blockquote>
  <p>This is a reference to an <code class="language-plaintext highlighter-rouge">a</code>, which may be full or empty.
The reference may be read or written to from any thread.
If the reference is empty, then other threads will queue up and wait for it to be full.
This means I can implement control, atomic updates, and fair access.
However, it also means that I can experience deadlock and blocking performance.</p>
</blockquote>

<p>Now, when writing code against an <code class="language-plaintext highlighter-rouge">MVar</code>, we have to be more careful: we can cause deadlock.
But we also have significantly more power over how the data is accessed and updated.
Particularly <em>fairness</em> and <em>coordination</em> are powerful advantages for the <code class="language-plaintext highlighter-rouge">MVar</code>.
However, transactions with multiple <code class="language-plaintext highlighter-rouge">MVar</code> are difficult to do correctly.</p>

<p>Where <code class="language-plaintext highlighter-rouge">IORef</code>s <code class="language-plaintext highlighter-rouge">atomicModifyIORef</code> requires a pure computation to avoid data races, you can <code class="language-plaintext highlighter-rouge">takeMVar</code> on an <code class="language-plaintext highlighter-rouge">MVar</code> to do an update - this allows you to do effects while computing your new value, and guarantees that consumers receive a consistent view of the value inside the <code class="language-plaintext highlighter-rouge">MVar</code>.</p>

<p>If I say <code class="language-plaintext highlighter-rouge">TVar a</code>, I mean:</p>

<blockquote>
  <p>This is a reference to an <code class="language-plaintext highlighter-rouge">a</code>.
The reference may be read or written to from any thread.
I want transactionality around <em>the entire structure of <code class="language-plaintext highlighter-rouge">a</code></em> - that is, if anyone touches any part of the structure of <code class="language-plaintext highlighter-rouge">a</code>, I want my whole transaction to be invalidated.</p>
</blockquote>

<p>This last bit is actually a very strong claim!
Are you sure you want transactionality around <em>the entire <code class="language-plaintext highlighter-rouge">a</code></em>?</p>

<p>For a <code class="language-plaintext highlighter-rouge">TVar Int</code>, the answer is <em>yes</em>.
But for a <code class="language-plaintext highlighter-rouge">TVar (Map k v)</code>, your transaction <em>probably</em> is only concerned with specific <em>parts</em> of the <code class="language-plaintext highlighter-rouge">Map</code>.
<code class="language-plaintext highlighter-rouge">stm-containers</code> uses a strategy similar to <code class="language-plaintext highlighter-rouge">TChan</code> and <code class="language-plaintext highlighter-rouge">TQueue</code> - rather than a <code class="language-plaintext highlighter-rouge">TVar</code> containing a recursive data structure, the actual recursive steps are themselves <code class="language-plaintext highlighter-rouge">TVar</code>s.
This allows modifications to the data structure to only invalidate transactions that are actually relevant.</p>

<p>Consider a SQL transaction.
If you have a query which selects a row from a table, you want to have some sort of locking to ensure that the row remains the same throughout the transaction.
Locking the <em>entire table</em> would be disastrous for performance - no one could do any work on the table until your transaction was complete!
Locking the entire row can also be quite bad for performance if the row is large.
But if you only select a handful of columns, ideally only <em>those</em> columns are locked by the query.</p>

<p><code class="language-plaintext highlighter-rouge">STM</code> is a wonderful mechanism for concurrency, but it isn’t foolproof.
We are still responsible for selecting the right data structures for good performance.
The core issue here is <em>coarse-grained transactional state</em>.
Finer transactionality gives us better performance.</p></div>
    </summary>
    <updated>2025-12-17T00:00:00Z</updated>
    <published>2025-12-17T00:00:00Z</published>
    <source>
      <id>https://www.parsonsmatt.org</id>
      <author>
        <name>Matt Parsons</name>
      </author>
      <link href="https://www.parsonsmatt.org" rel="alternate" type="text/html"/>
      <link href="https://www.parsonsmatt.org/feed.xml" rel="self" type="application/rss+xml"/>
      <title>parsonsmatt.org</title>
      <updated>2026-03-10T19:05:44Z</updated>
    </source>
  </entry>

  <entry xml:lang="en-US">
    <id>584219d403596e3099e0ee9b:58462c0e15d5db6feba171c0:68df0820fa5e8342ee3c3e89</id>
    <link href="https://mmhaskell.com/blog/2025/12/15/parsing-cdr-messages" rel="alternate" type="text/html"/>
    <title>Parsing CDR Messages</title>
    <summary type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Welcome to the 4th part of our series on parsing MCAP data from a ROS2 bag in Haskell. We’ve spent the previous parts <a href="https://mmhaskell.com/blog/2025/12/01/the-structure-of-an-mcap-file">understanding the record structure</a> in the bag and <a href="https://mmhaskell.com/blog/2025/12/08/schemas-channels-amp-messages-in-mcap">extracting the message data for particular topics</a>. Today we will parse that data into Haskell types! Unfortunately, this will actually require us to learn a bit about yet another data encoding (CDR), but luckily we’ll still be able to use some of our previous parsing code!</p>
<p>If you sign up for our <a href="https://academy.mondaymorninghaskell.com/p/solve-hs">Solve.hs course</a>, you’ll learn all about parsing techniques in Haskell in the 4th module! You’ll learn how to use the Megaparsec library, as well as other concepts like regular expressions.</p>
<h2>What is CDR?</h2>
<p>CDR stands for <a href="https://github.com/foxglove/cdr">Common Data Representation</a>, and it is a serialization format for primitive data. It is designed to work with an Interface Definition Language (IDL). ROS2 provides an IDL with its system of defining message types, and it uses CDR as the default encoding for message data in bags.</p>
<p>Now, most of the primitives we’ll see in our CDR data are encoded in a similar way to MCAP data. For example, fix-width integer values are still represented exactly how you would expect. Strings are similar in that they are size-prefixed by a 32-bit integer, but they are different in that CDR strings must have a null byte at the end.</p>
<p>What really distinguishes CDR though is the concept of <strong>alignment</strong>. Different primitive types must start at a position that is a certain multiple from the start of the data, and padding must be added if they would not.</p>
<p>The simplest example of this is to imagine we are encoding a boolean followed by a 64-bit integer (little endian). If we have the boolean <code>true</code> and the number <code>18</code>, we <em>could</em> represent this data with 9 bytes:</p>
<pre><code>0x01 0xf7 0x06 0x00 0x00 0x00 0x00 0x00 0x00</code></pre>
<p>The first byte (<code>0x01</code>) indicates the boolean <code>True</code>, the remaining 8 bytes give us the number 1783.</p>
<p>However, in CDR, a 64-bit integer <strong>must</strong> start at a position that is a multiple of 8. This means we would need to “pad” the boolean to have 7 additional null bytes:</p>
<pre><code>0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0xf7 0x06 0x00 0x00 0x00 0x00 0x00 0x00</code></pre>
<p>A 32-bit integer has an alignment multiple of 4, and since strings are prefixed by a 32-bit integer, this means they do as well. So to represent the boolean <code>true</code>, the string “Jacky” and the string “Carl”, we would have the following:</p>
<pre><code>0x01 0x00 0x00 0x00 # Bool `true` padded to 4
0x06 0x00 0x00 0x00 # Uint32 6 (size of “Jacky” + 1 for null byte)
0x4a 0x61 0x63 0x6b # “Jack”
0x79 0x00 0x00 0x00 # “y”, null byte to terminate, 2 bytes of padding
0x05 0x00 0x00 0x00 # Uint32 5 (size of “Carl” + 1 for null byte)
0x43 0x61 0x72 0x63 # “Carl”
0x00                # Null byte to terminate</code></pre>
<p>Now let’s start bringing this to Haskell.</p>
<h2>Representing our Messages</h2>
<p>First let’s define some Haskell types for our data. We’ve already done this with a <code>Simple</code> message type. Here is its message definition:</p>
<pre><code>int64 num
float64 value
string name</code></pre>
<p>And here is a Haskell type for it:</p>
<pre><code class="language-haskell">data SimpleMsg = SimpleMsg
  { num :: Int64
  , value :: Double
  , name :: String
  } deriving (Show, Eq)</code></pre>
<p>But since the first two types in here are aligned to 8, we could actually parse this without worrying about alignment! To make things more challenging (and show the range of data types available in ROS2), let’s make a <code>Complex</code> message like this:</p>
<pre><code>bool is_good
float64[] measurements
string complex_name
Simple msg1
Simple msg2</code></pre>
<p>This has a boolean, an array, and it references two instances of our <code>Simple</code> message above! Here’s how we could represent this type in Haskell:</p>
<pre><code class="language-haskell">data ComplexMsg = ComplexMsg
  { isGood :: Bool
  , measurements :: [Double]
  , complexName :: String
  , msg1 :: SimpleMsg
  , msg2 :: SimpleMsg
  } deriving (Show, Eq)</code></pre>
<p>Now let’s figure out how to organize our parsing code.</p>
<h2>A New Parser Monad</h2>
<p>So we won’t be parsing the message data in the context of the rest of the bag. Our bag parser organizes messages by topic, and we’ll parse the data after the fact. So while we still want a <code>ParsecT</code> monad, we don’t need <code>StateT TopicState</code>. Instead of tracking the full bag state, we need to track the number of bytes we’ve parsed so far so that we can perform alignment correctly. So let’s create this monad alias:</p>
<pre><code class="language-haskell">type CDRParser = ParsecT Void ByteString (StateT Word64 IO) a</code></pre>
<p>And now we can already write a crucial function to help us align our parser. We’ll call this <code>alignTo</code>, and it will move our parser forward until the saved alignment value matches the input:</p>
<pre><code class="language-haskell">alignTo :: Word64 -&gt; CDRParser ()
alignTo n = do
  align &lt;- lift get
  unless (align `mod` n == 0) $ do
    _ &lt;- anySingle
    lift $ modify (+ 1)
    alignTo n</code></pre>
<p>We inspect the current alignment value. If it is evenly divided by the input, then we are done. If not, we consume a single byte with <code>anySingle</code>, and then use <code>modify</code> to bump the saved alignment value by 1. Then we recurse so that we keep aligning until it’s done. We’ll use this function for each of our primitive parsers!</p>
<h2>Making our Monads Flexible</h2>
<p>Now ideally, we also want to make use of some of our prior parsers. After all, they use Megaparsec machinery. However, we’ve written them in the <code>Parser</code> monad, not the <code>CDRParser</code> monad.</p>
<p>Lucky for us, Megaparsec allows <code>ParsecT</code> to have a monad transformer class <code>MonadParsec</code> that enables us to reuse this code. We just have to change some type signatures. Let’s consider this original function:</p>
<pre><code class="language-haskell">parseUintLE :: (Bits a, Integral a) =&gt; Int -&gt; Parser (Word64, a)</code></pre>
<p>All we have to do to make this generalizable is add a <code>MonadParsec</code> constraint and use a generic monad type <code>m</code>:</p>
<pre><code class="language-haskell">parseUintLE :: (Bits a, Integral a, MonadParsec Void ByteString m) =&gt; Int -&gt; m (Word64, a)</code></pre>
<p>The <code>MonadParsec</code> class is still parameterized by the error type (<code>Void</code>) and the stream type (<code>ByteString</code>), so we need flexible contexts. We’ll do the same thing with our parsers for <code>Word64</code>, <code>Word32</code> and the string parser since we’ll use those in the next section:</p>
<pre><code class="language-haskell">parseUint64LE :: (MonadParsec Void ByteString m) =&gt; m (Word64, Word64)

parseUint32LE :: (MonadParsec Void ByteString m) =&gt; m (Word64, Word32)

parseString :: (MonadParsec Void ByteString m) =&gt; m (Word64, ByteString)</code></pre>
<p>One last thing we’ll want to generalize is <code>guard’</code>. This will allow us to do validation with our CDR parsers as well! We don’t need this to be <code>MonadParsec</code>. Any <code>MonadFail</code> will do.</p>
<pre><code class="language-haskell">guard' :: (MonadFail m) =&gt; String -&gt; Bool -&gt; m ()</code></pre>
<p>Now that these functions are generic, let’s apply them to our CDR case!</p>
<h2>Writing Primitive CDR Parsers</h2>
<p>First we’ll adapt our integer parsers. We can simply refer to our existing parsers, except we first align the parser and we update the alignment count.</p>
<pre><code class="language-haskell">parseCdrUint64LE :: CDRParser Word64
parseCdrUint64LE = do
  alignTo 8
  (len, n) &lt;- parseUint64LE
  lift $ modify (+ len)
  return n

parseCdrUint32LE :: CDRParser Word32
parseCdrUint32LE = do
  alignTo 4
  (len, n) &lt;- parseUint32LE
  lift $ modify (+ len)
  return n</code></pre>
<p>Our string parser is a similar story. However, we’ll do an extra check and manipulation because of the null byte. We want to use <code>guard’</code> to verify that the null byte is there, but then we use <code>init</code> later to remove it. We also convert it from <code>ByteString</code> to <code>String</code>.</p>
<pre><code class="language-haskell">import qualified Data.ByteString.Lazy.Char8 as BSC

parseCdrString :: CDRParser String
parseCdrString = do
  alignTo 4
  (len, str) &lt;- parseString
  lift $ modify (+ (fromIntegral len))
  guard' ("CDR String did not null terminate: " &lt;&gt; show str) (BS.last str == 0)
  return (init $ BSC.unpack str)</code></pre>
<p>To parse a <code>Double</code>, we’ll actually start by treating it like a 64-bit integer. Then we’ll just use <code>Data.Binary.IEEE754.coerceWordToDouble</code> to treat these exact bytes as a <code>Double</code> instead!</p>
<pre><code class="language-haskell">import Data.Binary.IEEE754 (wordToDouble)

parseCdrDoubleLE :: CDRParser Double
parseCdrDoubleLE = wordToDouble &lt;$&gt; parseCdrUint64LE</code></pre>
<p>Finally there’s boolean values. This one is easy with what we already know, as there’s no alignment necessary:</p>
<pre><code class="language-haskell">parseCdrBool :: CDRParser Bool
parseCdrBool = do
  b &lt;- anySingle
  lift $ modify (+ 1)
  return (b == 1)</code></pre>
<p>These are all the primitives we’ll need for our types, but there’s two more pieces we need. First, we need to be able to parse an <code>Array</code> of values. We’ll follow a similar pattern to the MCAP array parser by passing in <code>Parser a</code> as an argument. After that, things are much easier because CDR prefixes the array by the number of elements (32-bit integer), not the number of bytes.</p>
<pre><code class="language-haskell">parseCdrArray :: CDRParser a -&gt; CDRParser [a]
parseCdrArray parser = do
  n &lt;- parseCdrUint32LE
  forM [1..n] $ \_ -&gt; parser</code></pre>
<p>The nice part about this is that we’ve built alignment into all of our primitive parsers, so we don’t have to think about it when building macro parsers!</p>
<h2>Writing CDR Message Parsers</h2>
<p>Now it’s time to parse our messages. The good news is that we’ve done most of the hard work already! We just need one last helper, and that is for the CDR encapsulation header. This comes at the start of every message. It consists of 4 bytes, but we only care about the second byte. If this byte is <code>1</code>, then our data is little endian. Otherwise it is big endian. We’ll add a validation check that it’s little endian, since all our parsers assume this:</p>
<pre><code class="language-haskell">parseCdrHeader :: CDRParser ()
parseCdrHeader = do
  _ &lt;- anySingle
  leBit &lt;- anySingle
  _ &lt;- anySingle
  _ &lt;- anySingle
  guard' ("ROS2 CDR does not indicate little endian: " &lt;&gt; show leBit) (leBit == 1)</code></pre>
<p>This header comes at the start of every message, but we’ll write a parser for <code>SimpleMsg</code> that doesn’t include the header. We do this because there is <strong>not</strong> a separate header when <code>Complex</code> includes instances of <code>Simple</code>. This parser is quite simple, just using all our primitive parsers:</p>
<pre><code class="language-haskell">parseSimple :: CDRParser SimpleMsg
parseSimple = do
  n &lt;- parseCdrUint64LE
  v &lt;- parseCdrDoubleLE
  s &lt;- parseCdrString
  return $ SimpleMsg n v s</code></pre>
<p>Now we also include a version with the header:</p>
<pre><code class="language-haskell">parseSimpleMsg :: CDRParser SimpleMsg
parseSimpleMsg = do
  parseCdrHeader
  parseSimple</code></pre>
<p>Then <code>Complex</code> isn’t much harder. We just have more fields using our different primitives, plus <code>parseSimple</code>:</p>
<pre><code class="language-haskell">parseComplex :: CDRParser ComplexMsg
parseComplex = do
  b &lt;- parseCdrBool
  ms &lt;- parseCdrArray parseCdrDoubleLE
  n &lt;- parseCdrString
  s1 &lt;- parseSimple
  s2 &lt;- parseSimple
  return $ ComplexMsg b ms n s1 s2

parseComplexMsg :: CDRParser ComplexMsg
parseComplexMsg = do
  parseCdrHeader
  parseComplex</code></pre>
<p>Now we just need to call these from our main script!</p>
<h2>Parsing the Message Data</h2>
<p>There’s not much to this last part. Recalling from last time, our main function currently looks something like this:</p>
<pre><code class="language-haskell">parseBareRecordsFromFile :: FilePath -&gt; IO ()
parseBareRecordsFromFile fp = do
  input &lt;- BS.readFile fp
  (result, st) &lt;- runStateT (runParserT parseMcapFile' fp input) (TopicState HM.empty HM.empty HM.empty (HS.fromList ["/simple_topic", "/complex_topic"]) HS.empty)
  case result of
    Left e -&gt; print e
    Right recs -&gt; do
      forM_ recs $ \(_, rec) -&gt; printRec rec
      let simpleMessages = fromMaybe [] $ HM.lookup ("/simple_topic", "my_package/msg/Simple") (tsMessages st)
      forM_ simpleMessages $ \(MessageData t1 t2 msg) -&gt; do
        print (t1, t2, msg)
      let complexMessages = fromMaybe [] $ HM.lookup ("/complex_topic", "my_package/msg/Complex") (tsMessages st)
      forM_ complexMessages $ \(MessageData t1 t2 msg) -&gt; do
        print (t1, t2, msg)</code></pre>
<p>This will print the raw data for each message in the two topics <code>/simple_topic</code> and <code>/complex_topic</code>. Now, instead of just printing that data, we want to parse it! This involves invoking our <code>CDRParser</code> functions. We use <code>runParserT</code> on the raw data, while wrapping this in <code>evalStateT</code> with an initial alignment value of 0.</p>
<pre><code class="language-haskell">parseBareRecordsFromFile :: FilePath -&gt; IO ()
parseBareRecordsFromFile fp = do
      ...
      let simpleMessages = fromMaybe [] $ HM.lookup ("/simple_topic", "my_package/msg/Simple") (tsMessages st)
      forM_ simpleMessages $ \(MessageData t1 t2 msg) -&gt; do
        parseSimpleResult &lt;- evalStateT (runParserT parseSimpleMsg "Simple" msg) 0
        case parseSimpleResult of
          Left e -&gt; print e
          Right s -&gt; putStrLn $ "Parsed Simple Message: " &lt;&gt; show t1 &lt;&gt; " " &lt;&gt; show t2 &lt;&gt; " " &lt;&gt; show s
      let complexMessages = fromMaybe [] $ HM.lookup ("/complex_topic", "my_package/msg/Complex") (tsMessages st)
      forM_ complexMessages $ \(MessageData t1 t2 msg) -&gt; do
        parseComplexResult &lt;- evalStateT (runParserT parseComplexMsg "Complex" msg) 0
        case parseComplexResult of
          Left e -&gt; print e
          Right s -&gt; putStrLn $ "Parsed Complex Message: " &lt;&gt; show t1 &lt;&gt; " " &lt;&gt; show t2 &lt;&gt; " " &lt;&gt; show s</code></pre>
<p>And running this with our bag, we’ll see valid outputs!</p>
<pre><code>Parsed Simple Message: 1759374126013925786 1759374126013925786 SimpleMsg {num = 6, value = 6.3, name = "Goodbye"}
Parsed Simple Message: 1759374125991715245 1759374125991715245 SimpleMsg {num = 5, value = 4.2, name = "Hello"}
Parsed Complex Message: 1759374126014015513 1759374126014015513 ComplexMsg {isGood = False, measurements = [4.0,3.0,1.5,6.7,9.2], complexName = "Jo", simple1 = SimpleMsg {num = 12, value = 14.2, name = "Seymour"}, simple2 = SimpleMsg {num = 13, value = 15.1, name = "Sylvester"}}
Parsed Complex Message: 1759374126013960101 1759374126013960101 ComplexMsg {isGood = True, measurements = [1.0,2.0,2.5,4.5,6.1], complexName = "Jacky", simple1 = SimpleMsg {num = 10, value = 12.4, name = "Timothy"}, simple2 = SimpleMsg {num = 11, value = 13.3, name = "Jack"}}</code></pre>
<h2>Conclusion</h2>
<p>At long last, we have turned raw MCAP data into structured Haskell types! This is a great accomplishment, but there’s still more we can do. There’s a few different directions to go, but next time we’re going to try to optimize our parse so that we don’t have to scan the whole bag to find our topics. This will involve using indexes in the summary section, and we’ll have to re-think how we use Megaparsec.</p>
<p>If Megaparsec is still a bit confusing for you, you should sign up for <a href="https://academy.mondaymorninghaskell.com/p/solve-hs">Solve.hs</a>! In addition to teaching you about data structures, algorithms, and problem solving tricks in Haskell, you’ll also learn how to parse in Haskell! You’ll learn Megaparsec from the ground up, AND you’ll learn how to use regular expressions!</p></div>
    </summary>
    <updated>2025-12-15T12:30:00Z</updated>
    <published>2025-12-15T12:30:00Z</published>
    <author>
      <name>James Bowen</name>
    </author>
    <source>
      <id>https://mmhaskell.com/blog/</id>
      <link href="https://mmhaskell.com/blog/" rel="alternate" type="text/html"/>
      <link href="https://mmhaskell.com/blog?format=rss" rel="self" type="application/rss+xml"/>
      <title>Blog - Monday Morning Haskell</title>
      <updated>2025-10-05T20:31:39Z</updated>
    </source>
  </entry>
</feed>
