Planet Haskell

July 06, 2022

Mark Jason Dominus

Things I wish everyone knew about Git (Part II)

This is a writeup of a talk I gave in December for my previous employer. It's long so I'm publishing it in several parts:

It is really hard to lose stuff

A Git repository is an append-only filesystem. You can add snapshots of files and directories, but you can't modify or delete anything. Git commands sometimes purport to modify data. For example git commit --amend suggests that it amends a commit. It doesn't. There is no such thing as amending a commit; commits are immutable.

Rather, it writes a completely new commit, and then kinda turns its back on the old one. But the old commit is still in there, pristine, forever.

In a Git repository you can lose things, in the sense of forgetting where they are. But they can almost always be found again, one way or another, and when you find them they will be exactly the same as they were before. If you git commit --amend and change your mind later, it's not hard to get the old ⸢unamended⸣ commit back if you want it for some reason.

  • If you have the SHA for a file, it will always be the exact same version of the file with the exact same contents.

  • If you have the SHA for a directory (a “tree” in Git jargon) it will always contain the exact same versions of the exact same files with the exact same names.

  • If you have the SHA for a commit, it will always contain the exact same metainformation (description, when made, by whom, etc.) and the exact same snapshot of the entire file tree.

Objects can have other names and descriptions that come and go, but the SHA is forever.

(There's a small qualification to this: if the SHA is the only way to refer to a certain object, if it has no other names, and if you haven't used it for a few months, Git might discard it from the repository entirely.)

But what if you do lose something?

There are many good answers to this question but I think the one to know first is git-reflog, because it covers the great majority of cases.

The git-reflog command means:

“List the SHAs of commits I have visited recently”

When I run git reflog the top of the output says what commits I had checked out at recently, with the top line being the commit I have checked out right now:

    523e9fa1 HEAD@{0}: checkout: moving from dev to pasha
    5c31648d HEAD@{1}: pull: Fast-forward
    07053923 HEAD@{2}: checkout: moving from pr2323 to dev
    ...

The last thing I did was check out the branch named pasha; its tip commit is at 523e9f1a.

Before that, I did git pull and Git updated my local dev branch from the remote one, updating it to 5c31648d.

Before that, I had switched to dev from a different branch, pr2323. At that time, before the pull, dev referred to commit 07053923.

Farther down in the output are some commits I visited last August:

    ...
    58ec94f6 HEAD@{928}: pull --rebase origin dev: checkout 58ec94f6d6cb375e09e29a7a6f904e3b3c552772
    e0cfbaee HEAD@{929}: commit: WIP: model classes for condensedPlate and condensedRNAPlate
    f8d17671 HEAD@{930}: commit: Unskip tests that depend on standard seed data
    31137c90 HEAD@{931}: commit (amend): migrate pedigree tests into test/pedigree
    a4a2431a HEAD@{932}: commit: migrate pedigree tests into test/pedigree
    1fe585cb HEAD@{933}: checkout: moving from LAB-808-dao-transaction-test-mode to LAB-815-pedigree-extensions
    ...

Flux capacitor (magic time-travel doohickey) from “Back to the Future”

Suppose I'm caught in some horrible Git nightmare. Maybe I deleted the entire test suite or accidentally put my Small Wonder fanfic into a commit message or overwrote the report templates with 150 gigabytes of goat porn. I can go back to how things were before. I look in the reflog for the SHA of the commit just before I made my big blunder, and then:

    git reset --hard 881f53fa

Phew, it was just a bad dream.

(Of course, if my colleagues actually saw the goat porn, it can't fix that.)

I would like to nominate Wile E. Coyote to be the mascot of Git. Because Wile E. is always getting himself into situations like this one:

Wile E., a cartoon coyote has just fired a shotgun at Bugs Bunny.  For some reason the shotgun has fired backwards and blown his face off, as Git sometimes does.

But then, in the next scene, he is magically unharmed. That's Git.

Finding old stuff with git-reflog

  • git reflog by itself lists the places that HEAD has been
  • git reflog some-branch lists the places that some-branch has been
  • That HEAD@{1} thing in the reflog output is another way to name that commit if you don't want to use the SHA.
  • You can abbreviate it to just @{1}.
  • The following locutions can be used with any git command that wants you to identify a commit:

    • @{17} (HEAD as it was 17 actions ago)
    • @{18:43} (HEAD as it was at 18:43 today)
    • @{yesterday} (HEAD as it was 24 hours ago)
    • dev@{'3 days ago'} (dev as it was 3 days ago)
    • some-branch@{'Aug 22'} (some-branch as it was last August 22)

    (Use with git-checkout, git-reset, git-show, git-diff, etc.)

  • Also useful:

    git show dev@{'Aug 22'}:path/to/some/file.txt
    

    “Print out that file, as it was on dev, as dev was on August 22”

It's all still in there.

What if you can't find it?

Don't panic! Someone with more experience can probably find it for you. If you have a local Git expert, ask them for help.

And if they are busy and can't help you immediately, the thing you're looking for won't disappear while you wait for them. The repository is append-only. Every version of everything is saved. If they could have found it today, they will still be able to find it tomorrow.

(Git will eventually throw away lost and unused snapshots, but typically not anything you have used in the last 90 days.)

What if you don't like something you did?

Don't panic! It can probably put it back the way it was.

Git leaves a trail

When you make a commit, Git prints something like this:

    your-topic-branch 4e86fa23 Rework foozle subsystem

If you need to find that commit again, the SHA 4e86fa23 is in your terminal scrollback.

When you fetch a remote branch, Git prints:

       6e8fab43..bea7535b  dev        -> origin/dev

What commit was origin/dev before the fetch? At 6e8fab43. What commit is it now? bea7535b.

What if you want to look at how it was before? No problem, 6e8fab43 is still there. It's not called origin/dev any more, but the SHA is forever. You can still check it out and look at it:

    git checkout -b how-it-was-before 6e8fab43

What if you want to compare how it was with how it is now?

    git log 6e8fab43..bea7535b
    git show 6e8fab43..bea7535b
    git diff 6e8fab43..bea7535b

Git tries to leave a trail of breadcrumbs in your terminal. It's constantly printing out SHAs that you might want again.

A few things can be lost forever!

After all that talk about how Git will not lose things, I should point out the exceptions. The big exception is that if you have created files or made changes in the working tree, Git is unaware of them until you have added them with git-add. Until then, those changes are in the working tree but not in the repository, and if you discard them Git cannot help you get them back.

Good advice is Commit early and often. If you don't commit, at least add changes with git-add. Files added but not committed are saved in the repository, although they can be hard to find because they haven't been packaged into a commit with a single SHA id.

Some people automate this: they have a process that runs every few minutes and commits the current working tree to a special branch that they look at only in case of disaster.

The dangerous commands are git-reset and git-checkout

which modify the working tree, and so might wipe out changes that aren't in the repository. Git will try to warn you before doing something destructive to your working tree changes.

git-rev-parse

We saw a little while ago that Git's language for talking about commits and files is quite sophisticated:

            my-topic-branch@{'Aug 22'}:path/to/some/file.txt

Where is this language documented? Maybe not where you would expect: it's in the manual for git-rev-parse.

The git rev-parse command is less well-known than it should be. It takes a description of some object and turns it into a SHA. Why is that useful? Maybe not, but

The git-rev-parser man page explains the syntax of the descriptions Git understands.

A good habit is to skim over the manual every few months. You'll pick up something new and useful every time.

My favorite is that if you use the syntax :/foozle you get the most recent commit on the current branch whose message mentions foozle. For example:

    git show :/foozle

or

    git log :/introduce..:/remove

Coming next week (probably), a few miscellaneous matters about using Git more effectively.

by Mark Dominus (mjd@plover.com) at July 06, 2022 12:07 PM

July 04, 2022

Monday Morning Haskell

10 Steps to Understanding Data Structures in Haskell

(Skip to the steps)

Last year I completed Advent of Code, which presented a lot of interesting algorithmic challenges. One thing these problems really forced me to do was focus more clearly on using appropriate data structures, often going beyond the basics.

And looking back on that, it occurred to me that I hadn't really seen many tutorials on Haskell structures beyond lists and maps. Nor in fact, had I thought to write any myself! I've touched on sequences and arrays, but usually in the context of another topic, rather than focusing on the structure itself.

So after thinking about it, I decided it seemed worthwhile to start doing some material providing an overview on all the different structures you can use in Haskell. So data structures will be our "blog topic" starting in this month of July, and probably actually going into August. I'll be adding these overviews in a permanent series, so each blog post over the next few Mondays and Thursdays will link to the newest installment in that series.

But even beyond providing a basic overview of each type, I thought it would be helpful to come up with a process for learning new data structures - a process I could apply to learning any structure in any langugage. Going to the relevant API page can always be a little overwhelming. Where do you start? Which functions do you actually need?

So I made a list of the 10 steps you should take when learning the API for a data structure in a new language. The first of these have to do with reminding yourself what the structure is used for. Then you get down to the most important actions to get yourself started using that structure in code.

The 10 Steps

  1. What operations does it support most efficiently? (What is it good at?)
  2. What is it not as good at?
  3. How many parameters does the type use and what are their constraints.
  4. How do I initialize the structure?
  5. How do I get the size of the structure?
  6. How do I add items to the structure?
  7. How do I access (or get) elements from the structure?
  8. If possible, how do I delete elements from the structure?
  9. How do I combine two of these structures?
  10. How should I import the functions of this structure?

Each part of the series will run through these steps for a new structure, focusing on the basic API functions you'll need to know. To see my first try at using this approach where I go over the basic list type, head over to the first part of the series! As I update the series with more advanced structures I'll add more links to this post.

If you want to quickly see all the APIs for the structures we'll be covering, head to our eBooks page and download the FREE Data Structures at a Glance eBook!

by James Bowen at July 04, 2022 02:30 PM

July 02, 2022

ERDI Gergo

A small benchmark for functional languages targeting web browsers

I had an idea for a retro-gaming project that will require a MOS 6502 emulator that runs smoothly in the browser and can be customized easily. Because I only need the most basic of functionality from the emulation (I don't need to support interrupts, timing accuracy, or even the notion of cycles), I thought I'd just quickly write one. This post is not about the actual retro-gaming project that prompted this, but instead, my experience with the performance of the generated code using various functional-for-web languages.

As I usually do in situations like this, I started with a Haskell implementation to serve as a kind of executable specification, to make sure my understanding of the details of various 6502 instructions is correct. This Haskell implementation is nothing fancy: the outside world is modelled as a class MonadIO m => MonadMachine m, and the CPU itself runs in MonadMachine m => ReaderT CPU m, using IORefs in the CPU record for registers.

The languages

Ironing out all the wrinkles took a whole day, but once it worked well enough, it was time for the next step: rewriting it in a language that can then target the browser. PureScript seemed like an obvious choice: it's used a lot in the real world so it should be mature enough, and with how simple my Haskell code is, PureScript's idiosyncracies compared to Haskell shouldn't really come into play beyond the syntax level. The one thing that annoyed me to no end was that numeric literals are not overloaded, so all Word8s in my code had to be manually fromIntegral'd; and, in an emulator of an eight-bit CPU, there's a ton of Word8 literals...

The second contender was Idris 2. I've had good experience with Idris 1 for the web when I wrote the ICFP Bingo web app, but that project was all about the DOM manipulation and no computation. I was curious what performance I can get from Idris 2's JavaScript backend.

And then I had to include Asterius, a GHC-based compiler emitting WebAssembly. Its GitHub page states it is "actively maintained by Tweag I/O", but it's actually in quite a rough shape: the documentation on how to build it is out of date, so the only way to try it is via a 20G Docker container...

Notably missing from this list is GHCJS. Unfortunately, I couldn't find an up-to-date version of it; it seems the project, or at least work on integrating with standard Haskell tools like Stack, has died off.

To compare performances, I load the same memory image into the various emulators, set the program counter to the same starting point, and run it for 4142 instructions until a certain target instruction is reached. To paper over the browser's JavaScript JIT engine etc., each test runs for 100 times first as a warm-up, then 100 times measured.

Beside the PureScript, Idris 2, and GHC/Asterius implementations, I have also added a fourth version to serve as the baseline: vanilla JavaScript. Of course, I tried to make it as close to the functional versions as possible; I hope what I wrote is close to what could reasonably be expected as the output of a compiler.

Performance results

The following numbers come from the collected implementations in this GitHub repo. The PureScript and Idris 2 versions have been improved based on ideas from the respective Discord channels. For PureScript, using the CPS-transformed version of Reader helped; and in the case of Idris 2, Stefan Höck's changes of arguments instead of ReaderT, and using PrimIO when looping over instructions, improved performance dramatically.

Implementation Generated code size (bytes) Average time of 4142 instructions (ms)
JavaScript 12,877 2.12
Idris 2 60,379 6.38
PureScript 151,536 137.03
GHC/Asterius 1,448,826 346.73

So Idris 2 comes out way ahead of the pack here: unless you're willing to program in JavaScript, it's by far your best bet both for tiny deployment size and superb performance. All that remains to improve is to compile monad transformer stacks better so that the original ReaderT code works as well as the version using implicit parameters

To run the benchmark yourself, checkout the GitHub repo, run make in the top-level directory, and then use a web browser to open _build/index.html and use the JavaScript console to run await measureAll().

July 02, 2022 04:54 PM

July 01, 2022

Tweag I/O

Introducing Pirouette 2: formal methods for smart contracts

Writing software is hard, writing correct software is very hard; but as any hard thing, it becomes easier with the right tools. At Tweag we acknowledge this fact and make our tools sharper, in this blog post we introduce the second iteration of Pirouette: a tool for finding counterexamples for properties over Plutus smart contracts. For example, we can now mechanically produce a counterexample for the Minswap vulnerability:

💸 COUNTEREXAMPLE FOUND
{ __result ↦ True
  tx ↦ MkTxInfo [P (MkTxOutRef (Id 42) 0) (MkTxOut (A 0) (V (KV [])))] [] (V (KV [])) (V (KV [P "currency" (KV [P "token" 1, P "" 0])])) (Id 0) }

Pirouette’s goals are larger than contracts, though. We’ve built is as a framework for transforming, evaluating, and proving specifications for any System F-based language — read “any powerful functional language� — of which Plutus is an example. Pirouette’s main analysis uses a recent technique, incorrectness logic together with a symbolic evaluation engine, which allows us to reason in a necessity fashion, in addition to the more common sufficiency style. In this blogpost we will explain in a little more detail what this technique consists in and why it is useful for reasoning over Plutus smart contracts.

The Need for Tooling

At a very high level, a smart contract is a predicate which given some input — which in this case may represent some assets — and contextual information, decides whether the contract can be executed or not.

contract :: Transaction -> Bool

Note that returning True in this case has very real consequences, as it allows the respective transaction to go through and its assets to be transferred accordingly. Most attacks on smart contracts can in fact be characterized as finding a corner case which makes the predicate return True for an unintended transaction. Furthermore, smart contracts are often immutable, making them notoriously difficult to write.

Our job as auditors is to find those corner cases before a malicious party does. While manual audits can help increase our assurance that the smart contract is good, it is always a best-effort activity. Every other week we see contracts being hacked in the Ethereum space, even audited ones, and that happens in spite of the existence of a number of tools and years of experience by the Ethereum community. This is by no means circumscribed to Ethereum; if we keep using the same approaches, contracts running in the Cardano ecosystem will inevitably fall victim to the same fate.

We need to step up our game: audits are a best-effort activity, but they can become much more directed and successful with the right tools. This doesn’t give us a 100% percent guarantee — specifications can be wrong, the tools can have bugs, the platform itself may be exploitable — but machine-checked analysis is infinitely better than manual inspection. The development of these tools must be an essential part of the development of high-assurance contracts.

Pirouette 2

The development of new tools for reasoning about code poses an important question at its very inception: what of the dozens of well-known techniques should be implement? And what should be the input language to that tool? Our past experience as formal methods engineers showed us that in many cases the answer to the latter question is just a variation of another language, so it made sense to engineer Pirouette as a common core language which can be extended by each particular need. Currently, we have a need for more machine checked guarantees in our Plutus audits, hence, our first target was Plutus IR, which is based on System F, choosing a core syntax for System F which can be extended seemed like a natural choice. Pirouette can easily be extended to handle any other System F based language.

Necessity reasoning

Once we’ve introduced our general framework, let’s focus on one particular way in which we’ve used symbolic evaluation in Pirouette to better understand and check smart contracts, based on incorrectness logic, a technique introduced in 2020 by Peter O’Hearn.

Many techniques in the formal method space use sufficiency reasoning, which follows this general pattern:

  1. Assume that some property of the inputs, or in general of the starting state, holds.
  2. Figure out what happens to the state and the result after executing the function.
  3. Check that that final state satisfies whatever property we expect.

Take the following increment function,

increment :: Integer -> Integer
increment x = x + 1

Following that outline we see that if x > 1, then result > 1. Those steps correspond to (1) assume that x > 1, (2) figure out that result == x + 1, and (3) check that x > 1 && result == x + 1 ==> result > 1. (The particularities of each step don’t matter so much here, although step 3 is usually outsourced to an external reasoning engine, like an SMT solver.) What we have proven here is that x > 1 is a sufficient condition for result > 1 to hold, but this doesn’t tell us anything about x == 0.

In the space of smart contracts, sufficiency reasoning is not enough. Suppose we use some tool to prove that if the input conditions for the contract satisfy P, then contract returns True; this can be useful in our understanding, but doesn’t tell us whether our condition P is too strong, and there’s some space which a malicious party can exploit. Going back to our example, are there other cases which also lead to result > 1 but shouldn’t?

The solution is to switch to necessity reasoning, which is the name we’ve given to our variation of incorrectness logic. We get one particular mode of necessity reasoning by simply taking the steps for sufficiency reasoning, but in the opposite direction:

  1. Assume that some property of the result, or in general of the final state, holds.
  2. Figure out what happens to the state and input while executing the function.
  3. Check that the inputs, and in general the initial state, satisfy whatever property we expect.

That mode of reasoning rules out the x > 1 as a good specification, since by (1) assuming result > 1, and (2) figuring out that result == x + 1, it does not hold that (3) x > 1. In other words, necessity reasoning forces us to consider every possible input when writing a specification; this is extremely useful when describing the attack surface of a smart contract.

We have previously mentioned the MinSwap LP tokens vulnerability, and how it stemmed from an incorrect check that allowed minting other tokens on the side. We rewrote the vulnerable minting policy in Pirouette’s example language and were able to find a counterexample:

💸 COUNTEREXAMPLE FOUND
{ __result ↦ True
  tx ↦ MkTxInfo [P (MkTxOutRef (Id 42) 0) (MkTxOut (A 0) (V (KV [])))] [] (V (KV [])) (V (KV [P "currency" (KV [P "token" 1, P "" 0])])) (Id 0) }

The tx value represents a transaction, with its inputs, outputs, fees, and minted assets. If we look carefully we can see that a “rogue� minting is performed,

[P "currency" (KV [P "token" 1, P "" 0])]

since we have values for both our own token and another unexpected one. Since that would not be expected by the auditors, this would have launched a more sophisticated analysis of the code in that section.

Conclusion

We’re very happy to announce the new version of our Pirouette tool, which has been envisioned as a framework for transforming, analyzing, and verifying languages with a System F core. Over that foundation we’ve implemented a recent technique — incorrectness logic.

Smart contracts provide a prime example of the need to consider all inputs to a function which lead to the desired outcome. Incorrectness logic and necessity reasoning provides us the necessary logic foundation for that kind of specification.

July 01, 2022 12:00 AM

June 30, 2022

Philip Wadler

No, No, No


How to say no, a collection of templates that answer "no" to any possible question.

Maker's schedule, manager's schedule, an essay by Paul Graham explaining that meetings for makers and managers have very different costs.

Here's what it costs to say yes, an essay by Ryan Holiday to explain his calendar anorexia.

The last of these quotes Seneca:
No person would give up even an inch of their estate, and the slightest dispute with a neighbor can mean hell to pay; yet we easily let others encroach on our lives—worse, we often pave the way for those who will take it over. No person hands out their money to passers-by, but to how many do each of us hand out our lives! We’re tight-fisted with property and money, yet think too little of wasting time, the one thing about which we should all be the toughest misers.

by Philip Wadler (noreply@blogger.com) at June 30, 2022 09:06 PM

Monday Morning Haskell

A Brief Look at Asynchronous Exceptions

We've covered a lot of different exception mechanisms this month. Today we'll cover just one more concept that will serve as a teaser for some future exploration of concurrency. All the exception mechanisms we've looked at so are serial in nature. We call a certain IO function from our main thread, and we block the program until that operation finishes. However, exceptions can also happen in an asynchronous way. This can happen in a couple different ways.

First, as we'll learn later this year, we can fork different threads for our program to run, and it is possible to raise an exception in a different thread! If you're interested in exploring how this works, the relevant functions you should learn about are forkIO and throwTo:

forkIO :: IO () -> IO ThreadId

throwTo :: (Exception e) => ThreadId -> e -> IO ()

The first will take an IO action and run it on a new thread, returning the ID of that thread. The section function then allows you to raise an exception in that thread using the ID. We'll look into the details of this function at a later date.

However, there are also certain asynchronous exceptions that can occur no matter what we do. At any point, our program could theoretically run out of memory (StackOverflow or HeapOverflow), and our program will have to abort the execution of our program in an asynchronous manner. So even if we aren't specially forking new threads, we still might encounter these kinds of exceptions.

The mask function is a special utility that prevents an action from being interrupted by an asynchronous exception.

mask :: ((forall a. IO a -> IO a) -> IO b) -> IO b

However, just from looking at the type signature, this function is rather confusing. It takes one argument, a function that takes another function as its input! As we can read in the documentation though, the most common use case for this function is to protect a resource to make sure we release it even if an exception is thrown while performing a computation with it.

Lucky for us, the bracket function already handles this case, as we've discussed. So the chances that you'll have to manually use mask are not very high.

This wraps up our discussion of exceptions for now on Monday Morning Haskell! We'll be sure to touch on this subject again when we get to concurrency in a few months. If you want a summary of this month's articles in your inbox, there's still time to subscribe to our monthly newsletter! You won't want to miss what's coming up next week!

by James Bowen at June 30, 2022 02:30 PM

June 29, 2022

Mark Jason Dominus

Things I wish everyone knew about Git (Part I)

This is a writeup of a talk I gave in December for my previous employer. It's long so I'm publishing it in several parts:

How to approach Git; general strategy

Git has an elegant and powerful underlying model based on a few simple concepts:

  1. Commits are immutable snapshots of the repository
  2. Branches are named sequences of commits
  3. Every object has a unique ID, derived from its content

black and white white ink diagram of the elegant geometry of the floor plan of a cathedral

Built atop this elegant system is a flaming trash pile.

literal dumpster fire

The command set wasn't always well thought out, and then over the years it grew by accretion, with new stuff piled on top of old stuff that couldn't be changed because Backward Compatibility. The commands are non-orthogonal and when two commands perform the same task they often have inconsistent options or are described with different terminology. Even when the individual commands don't conflict with one another, they are often badly-designed and confusing. The documentation is often very poorly written.

What this means

With a lot of software, you can opt to use it at a surface level without understanding it at a deeper level:

“I don't need to know how it works.
I just want to know which commands to run.”

This is often an effective strategy, but

with Git, this does not work.

You can't “just know which commands to run” because the commands do not make sense!

To work effectively with Git, you must have a model of what the repository is like, so that you can formulate questions like “is the repo on this state or that state?” and “the repo is in this state, how do I get it into that state?”. At that point you look around for a command that answers your question, and there are probably several ways to do what you want.

But if you try to understand the commands without the model, you will suffer, because the commands do not make sense.

Just a few examples:

  • git-reset does up to three different things, depending on flags

  • git-checkout is worse

  • The opposite of git-push is not git-pull, it's git-fetch

  • etc.

If you try to understand the commands without a clear idea of the model, you'll be perpetually confused about what is happening and why, and you won't know what questions to ask to find out what is going on.

READ THIS

When I first used Git it drove me almost to tears of rage and frustration. But I did get it under control. I don't love Git, but I use it every day, by choice, and I use it effectively.

The magic key that rescued me was

John Wiegley's
Git From the Bottom Up

Git From the Bottom Up explains the model. I read it. After that I wept no more. I understood what was going on. I knew how to try things out and how to interpret what I saw. Even when I got a surprise, I had a model to fit it into.

You should read it too.

That's the best advice I have. Read Wiegley's explanation. Set aside time to go over it carefully and try out his examples. It fixed me.

If I were going to tell every programmer just one thing about Git, that would be it.

The rest of this series is all downhill from here.

But if I were going to tell everyone just one more thing, it would be:

It is very hard to permanently lose work.
If something seems to have gone wrong, don't panic.
Remain calm and ask an expert.

Many more details about that are in the followup article.

by Mark Dominus (mjd@plover.com) at June 29, 2022 05:16 PM

June 27, 2022

Monday Morning Haskell

Catching Before Production: Assert Statements in Haskell

We've spent a lot of time this month going over exceptions, which are ways to signal within our program that something unexpected has happened. These will often result in an early termination for our program even if we catch them. But by catching them, we can typically provide more helpful error messages and logs. Exceptions are intended for use in production code. You don't want them to ever go off, but they are there .

However, there are other bugs that you really want to catch before they ever make it into production. You don't want to formally recognize them in the type system because other parts of the program shouldn't have to deal with those possibilities. In these cases, it is common practice for programmers to use "assert" statements instead.

We'll start with a simple example in Python. We'll write a function to adjust a price, first by subtracting and second by taking the square root. Of course, you cannot take the square root of a negative number (and prices shouldn't be negative anyways). So we'll assert that the price is non-negative before we take the root.

def adjustPrice(input):
  adjustedDown = input - 400.0
  assert (adjustedDown >= 0)
  return $ sqrt(adjustedDown)

In Haskell we also have an assert function. It looks a bit like throw in the sense that its type signature looks pure, but can actually cause an error.

assert :: Bool -> a -> a

If the boolean input is "true", nothing happens. The function will return the second input as its output. But if the boolean is false, then it will throw an exception. This is useful because it will provide us with more information about where the error occurred. So let's rewrite the above function in Haskell.

adjustPrice :: Double -> Double
adjustPrice input = assert (adjustedDown >= 0.0) (sqrt adjustedDown)
  where
    adjustedDown = input - 400.0

If we give it a bad input, we'll get a helpful error message with the file and line number where the assertion occurred:

main :: IO ()
main = do
  let result = adjustPrice 325.0
  print result

...

>> stack exec my-program

my-program: Assertion failed
CallStack (from HasCallStack):
  assert, called at src/Lib.hs in MyProgram-0.1.0.0:Lib

Without using the asssert, our function would simply return NaN and continue on! It would be much harder for us to track down where the bug came from. Ideally, we would catch a case like this in unit testing. And it might indicate that our "adjustment" is too high (perhaps it should be 40.0 instead of 400.0).

For the sake of efficiency, assert statements are turned off in executable code. This is why it is imperative that you write a unit test to uncover the assertion problem. In order to run your program with assertions, you'll need to use the fno-ignore-asserts GHC option. This is usually off for executables, but on for test suites.

We have one more concept to talk about with exception handling, so get ready for that! If you want a summary of all the topics we talked about this month, make sure to subscribe to our monthly newsletter!

by James Bowen at June 27, 2022 02:30 PM

Gabriella Gonzalez

defaultable-map: An Applicative wrapper for Maps

defaultable-map: An Applicative wrapper for Maps

I’m announcing a small utility Haskell package I created that can wrap arbitrary Map-like types to provide Applicative and Alternative instances. You can find this package on Hackage here:

I can motivate why the Applicative and Alternative instances matter with a small example. Suppose that I define the following three Maps which are sort of like database tables:

import Defaultable.Map 

firstNames :: Defaultable (Map Int) String
firstNames = fromList [(0, "Gabriella"), (1, "Oscar"), (2, "Edgar")]

lastNames :: Defaultable (Map Int) String
lastNames = fromList [(0, "Gonzalez"), (2, "Codd"), (3, "Bryant")]

handles :: Defaultable (Map Int) String
handles = fromList [(0, "GabriellaG439"), (1, "posco"), (3, "avibryant")]

If you squint, you can think of these as analogous to database tables, where the primary key is an Int index:

> CREATE TABLE firstNames (id integer, firstName text);
> INSERT INTO firstNames (id, firstName) VALUES (0, 'Gabriella');
> INSERT INTO firstNames (id, firstName) VALUES (1, 'Oscar');
> INSERT INTO firstNames (id, firstName) VALUES (2, 'Edgar');
> SELECT * FROM firstNames;
id | firstName
---+----------
0 | Gabriella
1 | Oscar
2 | Edgar

> CREATE TABLE lastNames (id integer, lastName text);
> INSERT INTO lastNames (id, lastName) VALUES (0, 'Gonzalez');
> INSERT INTO lastNames (id, lastName) VALUES (2, 'Codd');
> INSERT INTO lastNames (id, lastName) VALUES (3, 'Bryant');
> SELECT * FROM lastNames;
id | lastName
---+---------
0 | Gonzalez
2 | Codd
3 | Bryant

> CREATE TABLE handles (id integer, handle text);
> INSERT INTO handles (id, handle) VALUES (0, 'GabriellaG439');
> INSERT INTO handles (id, handle) VALUES (1, 'posco');
> INSERT INTO handles (id, handle) VALUES (3, 'avibryant');
> SELECT * FROM handles;
id | handle
---+--------------
0 | GabriellaG439
1 | posco
3 | avibryant

The Defaultable (Map Int) type has a law-abiding Applicative instance, so we can safely “join” these “tables” using Applicative operations. For example, if we enable Haskell’s ApplicativeDo language extension then we can compute an “inner join” on tables like this:

{-# LANGUAGE ApplicativeDo #-}

innerJoin :: Defaultable (Map Int) (String, String)
innerJoin = do
firstName <- firstNames
lastName <- lastNames
return (firstName, lastName)

… and that evaluates to the following result:

Defaultable
(fromList
[ (0, ("Gabriella","Gonzalez"))
, (2, ("Edgar" ,"Codd" ))
]
)
Nothing

… which is the same result we would have gotten from doing an inner join in SQL:

> SELECT firstNames.id, firstName, lastName
> FROM firstNames INNER JOIN lastNames on firstNames.id = lastNames.id;
id | firstName | lastName
---+-----------+---------
0 | Gabriella | Gonzalez
2 | Edgar | Codd

The Defaultable (Map Int) type also has a law-abiding Alternative instance, which we can combine with the Applicative instance to compute “left/right/outer joins”. For example, this “left join”:

leftJoin :: Defaultable (Map Int) (String, Maybe String)
leftJoin = do
firstName <- firstNames
lastName <- optional lastNames
return (firstName, lastName)

… evaluates to:

Defaultable
(fromList
[ (0, ("Gabriella",Just "Gonzalez"))
, (1, ("Oscar" ,Nothing ))
, (2, ("Edgar" ,Just "Codd" ))
]
)
Nothing

… which is analogous to this SQL left join:

> SELECT firstNames.id, firstName, lastName
> FROM firstNames LEFT JOIN lastNames on firstNames.id = lastNames.id;
id | firstName | lastName
---+-----------+---------
0 | Gabriella | Gonzalez
1 | Oscar |
2 | Edgar | Codd

Since Haskell is a more fully-featured language than SQL, we can do more sophisticated things more easily than in SQL. For example, the following three-way join with some post-processing logic is much easier to express in Haskell than SQL:

display :: String -> Maybe String -> String -> String
display firstName Nothing handle =
firstName <> ": @" <> handle
display firstName (Just lastName) handle =
firstName <> " " <> lastName <> ": @" <> handle

interestingJoin :: Defaultable (Map Int) String
interestingJoin = do
firstName <- firstNames
lastName <- optional lastNames
handle <- handles
return (display firstName lastName handle)

… which evaluates to:

Defaultable
(fromList
[ (0, "Gabriella Gonzalez: @GabriellaG439")
, (1, "Oscar: @posco" )
]
)
Nothing

The Defaultable type constructor

The central data type exported by the package is the Defaultable type constructor, which has the following simple definition:

data Defaultable map value = Defaultable (map value) (Maybe value)

Here the map type parameter can be any Map-like type that includes the type of the key. For example, a typical instantiation of the Defaultable type constructor might be Defaultable (Map key) value or Defaultable IntMap value.

The first field of the type is the actual map that you want to wrap in order to get an Applicative and Alternative instance. The second field is an optional default value stored alongside the map that can be returned if a lookup does not find a matching key.

The default value is not required (it can be Nothing), but that default value is what makes the Applicative instance work. Specifically, without the ability to specify a default value there would be no way to implement pure for a Map-like type.

In case you’re curious, here is what the Applicative instance looks like:

instance (Apply map, forall a . Monoid (map a)) => Applicative (Defaultable map)
where
pure v = Defaultable mempty (pure v)

Defaultable fMap fDefault <*> Defaultable xMap xDefault =
Defaultable fxMap fxDefault
where
fxMap = (fMap <.> xMap) <> fFallback <> xFallback
where
fFallback =
case fDefault of
Nothing -> mempty
Just f -> fmap f xMap

xFallback =
case xDefault of
Nothing -> mempty
Just x -> fmap ($ x) fMap

fxDefault = fDefault <*> xDefault

The neat part of the above instance is the class constraint:

          ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
instance (Apply map, forall a . Monoid (map a)) => Applicative (Defaultable map)

The Defaultable type is set up in such a way that you can wrap any Map-like type that satisfies that constraint (which is basically all of them) and get a law-abiding Applicative instance (See the Appendix for a proof of the Applicative laws).

In particular, this constraint makes use of the QuantifiedConstraints language extension introduced in GHC 8.6. Without that instance then we wouldn’t be able to generalize this type to wrap arbitrary Maps and we’d have to hard-code the package to work with a specific Map like Data.Map.Map.

The Defaultable type also implements Alternative, too, although that instance is much simpler:

instance (Apply map, forall a . Monoid (map a)) => Alternative (Defaultable map) where
empty = Defaultable mempty empty

Defaultable lMap lDefault <|> Defaultable rMap rDefault =
Defaultable (lMap <> rMap) (lDefault <|> rDefault)

This instance is only possible because the Defaultable type constructor doesn’t require the default value to be present. If the default value were required then we could not sensibly define empty.

Prior art

I was surprised that something like this didn’t already exist on Hackage. The closest package I could find was this one:

However, that wasn’t exactly what I wanted, because it requires the default value to be present. That means that you can’t implement an Alternative instance for the TMap type from that package and you therefore can’t do things like left/right/outer joins as I mentioned above.

Also, more generally, sometimes you want a Map to have an Applicative instance without having to specify a default value. Requiring the default to always be present is not necessary to implement Applicative.

The other issue I had with that package is that it’s hard-coded to use Data.Map.Map under the hood, whereas I wanted an API that could be used in conjunction with any Map-like type.

Conclusion

The idea for this package originated from a LambdaConf presentation I gave a while ago where I brainstormed what a good “data science” ecosystem for Haskell might look like:

I sat on this idea for years without publishing anything to Hackage because my original vision was a bit too ambitious and included much more than just an Applicative Map type. However, recently I needed this Applicative Map type, so I settled for publishing a narrower and more focused package to Hackage.

The personal use case I have in mind for this package is no longer data science, but I hope that people interested in building a data science ecosystem for Haskell consider using this package as a building block since I believe it is well-suited for that purpose.

Appendix - Proof of the Applicative laws

These proofs require a few additional assumptions about the interaction between the Apply and Monoid constraint on the map type parameter to Defaultable. These assumptions hold for Map-like types.

The first assumption is that fmap is a Monoid homomorphism:

fmap f mempty = mempty

fmap f (x <> y) = fmap f x <> fmap f y

The second assumption is that f <.> is a Monoid homorphism:

f <.> mempty = mempty

f <.> (x <> y) = (f <.> x) <> (f <.> y)

The final assumption is specific to maps, which is:

-- Given:
mf :: map (a -> b)
mx :: map a
kf :: (a -> b) -> c
kx :: a -> c

(mf <.> mx) <> fmap kf mf <> fmap kx mx
= (mf <.> mx) <> fmap kx mx <> fmap kf mf

The intuition here is if that map is a Map-like type constructor then we can think of those three expressions as having a set of keys associated with them, such that:

-- Given:
keys :: map a -> Set key

keys (mf <.> mx) = keys (fmap kf mf) `intersection` keys (fmap kx mx)

So normally the following equality would not be true:

  fmap kf mf <> fmap kx mx
= fmap kx mx <> fmap kf mf

… because the result would change if there was a key collision. Then the order in which we union (<>) the two maps would change the result.

However, if you union yet another map (mf <.> mx) that shadows the colliding keys then result remains the same.

The proof below uses that assumption a bit less formally by just noting that we can commute a union operation if there is a downstream union operation that would shadow any colliding keys that might differ.

Proof of identity law
pure id <*> v

-- pure v = Defaultable mempty (pure v)
= Defaultable mempty (pure id) <*> v

-- Expand: v = Defaultable xMap xDefault
= Defaultable mempty (pure id) <*> Defaultable xMap xDefault

-- Definition of (<*>)
= Defaultable fxMap fxDefault
where
fxMap = (mempty <.> xMap) <> fFallback <> xFallback
where
fFallback =
case pure id of
Nothing -> mempty
Just f -> fmap f xMap

xFallback =
case xDefault of
Nothing -> mempty
Just x -> fmap ($ x) mempty

fxDefault = pure id <*> xDefault

-- mempty <.> xMap = mempty
= Defaultable fxMap fxDefault
where
fxMap = mempty <> fFallback <> xFallback
where
fFallback =
case pure id of
Nothing -> mempty
Just f -> fmap f xMap

xFallback =
case xDefault of
Nothing -> mempty
Just x -> fmap ($ x) mempty

fxDefault = pure id <*> xDefault

-- Simplify `case pure id of …`
= Defaultable fxMap fxDefault
where
fxMap = mempty <> fFallback <> xFallback
where
fFallback = fmap id xMap

xFallback =
case xDefault of
Nothing -> mempty
Just x -> fmap ($ x) mempty

fxDefault = pure id <*> xDefault

-- fmap id x = x
= Defaultable fxMap fxDefault
where
fxMap = mempty <> fFallback <> xFallback
where
fFallback = xMap

xFallback =
case xDefault of
Nothing -> mempty
Just x -> fmap ($ x) mempty

fxDefault = pure id <*> xDefault

-- fmap f mempty = mempty
= Defaultable fxMap fxDefault
where
fxMap = mempty < >fFallback <> xFallback
where
fFallback = xMap

xFallback =
case xDefault of
Nothing -> mempty
Just x -> mempty

fxDefault = pure id <*> xDefault

-- pure id <*> v = v
= Defaultable fxMap fxDefault
where
fxMap = mempty <> fFallback <> xFallback
where
fFallback = xMap

xFallback =
case xDefault of
Nothing -> mempty
Just x -> mempty

fxDefault = xDefault

-- Simplify
= Defaultable fxMap fxDefault
where
fxMap = mempty <> xMap <> mempty

fxDefault = xDefault

-- x <> mempty = x
-- mempty <> x = x
= Defaultable fxMap fxDefault
where
fxMap = xMap

fxDefault = xDefault

-- Simplify
= Defaultable xMap xDefault

-- Contract: v = Defaultable xMap xDefault
= v
Proof of the composition law
pure (.) <*> u <*> v <*> w

-- Expand:
-- u = Defaultable uMap uDefault
-- v = Defaultable vMap vDefault
-- w = Defaultable wMap wDefault
= pure (.)
<*> Defaultable uMap uDefault
<*> Defaultable vMap vDefault
<*> Defaultable wMap wDefault

-- pure v = Defaultable mempty (pure v)
= Defaultable mempty (pure (.))
<*> Defaultable uMap uDefault
<*> Defaultable vMap vDefault
<*> Defaultable wMap wDefault

… before continuing, it’s easiest to prove all eight possible combinations of:

  • uDefault is pure u or empty
  • vDefault is pure v or empty
  • wDefault is pure w or empty

To avoid lots of repetition, I’ll only prove the most difficult case (where all defaults are present), since the other proofs are essentially subsets of that proof where some subterms disappear because they become mempty.

Case:

  • uDefault = pure u
  • vDefault = pure v
  • wDefault = pure w
=       Defaultable mempty (pure (.))
<*> Defaultable uMap (pure u)
<*> Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)

-- Definition of (<*>)
= (Defaultable cuMap cuDefault
where
cuMap = (mempty <.> uMap) <> fmap (.) uMap <> fmap ($ u) mempty

cuDefault = pure (.) <*> pure u
)
<*> Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)

-- mempty <.> x = mempty
= (Defaultable cuMap cuDefault
where
cuMap = mempty <> fmap (.) uMap <> fmap ($ u) mempty

cuDefault = pure (.) <*> pure u
)
<*> Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)

-- fmap f mempty = mempty
= (Defaultable cuMap cuDefault
where
cuMap = mempty <> fmap (.) uMap <> mempty

cuDefault = pure (.) <*> pure u
)
<*> Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)

-- pure f <*> pure x = pure (f x)
= (Defaultable cuMap cuDefault
where
cuMap = mempty <> fmap (.) uMap <> mempty

cuDefault = pure ((.) u)
)
<*> Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)

-- x <> mempty = x
-- mempty <> x = x
= (Defaultable cuMap cuDefault
where
cuMap = fmap (.) uMap

cuDefault = pure ((.) u)
)
<*> Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)

-- Simplify
= (Defaultable (fmap (.) uMap) (pure (u .)))
<*> Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)

-- Definition of (<*>)
= (Defaultable cuvMap cuvDefault
where
cuvMap =
(fmap (.) uMap <.> vMap)
<> fmap (u .) vMap
<> fmap ($ v) (fmap (.) uMap)

cuvDefault = pure (u .) <*> pure v
)
<*> Defaultable wMap (pure w)

-- fmap f (fmap g x) = fmap (f . g) x
= (Defaultable cuvMap cuvDefault
where
cuvMap =
(fmap (.) uMap <.> vMap)
<> fmap (u .) vMap
<> fmap (. v) uMap

cuvDefault = pure (u .) <*> pure v
)
<*> Defaultable wMap (pure w)

-- ((.) u) = \v -> u . v
= (Defaultable cuvMap cuvDefault
where
cuvMap =
(fmap (.) uMap <.> vMap)
<> fmap (u .) vMap
<> fmap (. v) uMap

cuvDefault = pure (u .) <*> pure v
)
<*> Defaultable wMap (pure w)

-- pure f <*> pure x = pure (f x)
= (Defaultable cuvMap cuvDefault
where
cuvMap =
(fmap (.) uMap <.> vMap)
<> fmap (u .) vMap
<> fmap (. v) uMap

cuvDefault = pure (u . v)
)
<*> Defaultable wMap (pure w)

-- Definition of (<*>)
= Defaultable cuvwMap cuvwDefault
where
cuvwMap = (cuvMap <.> wMap) <> fmap (u . v) wMap <> fmap ($ w) cuvMap
where
cuvMap =
(fmap (.) uMap <.> vMap)
<> fmap (u .) vMap
<> fmap (. v) uMap

cuvwDefault = pure (u . v) <*> pure v

-- (f <> g) <.> x = (f <.> x) <> (g <.> x)
-- fmap f (x <> y) = fmap f x <> fmap f y
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(fmap (.) uMap <.> vMap <.> wMap)
<> (fmap (u .) vMap <.> wMap)
<> (fmap (. v) uMap <.> wMap)
<> fmap (u . v) wMap
<> fmap ($ w) (fmap (.) uMap <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u . v) <*> pure w

-- pure f <*> pure x = pure (f x)
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(fmap (.) uMap <.> vMap <.> wMap)
<> (fmap (u .) vMap <.> wMap)
<> (fmap (. v) uMap <.> wMap)
<> fmap (u . v) wMap
<> fmap ($ w) (fmap (.) uMap <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- fmap (.) u <.> v <.> w = u <.> (v <.> w)
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (fmap (u .) vMap <.> wMap)
<> (fmap (. v) uMap <.> wMap)
<> fmap (u . v) wMap
<> fmap ($ w) (fmap (.) uMap <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- fmap f (x <.> y) = fmap (f .) x <.> y
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (fmap (. v) uMap <.> wMap)
<> fmap (u . v) wMap
<> fmap ($ w) (fmap (.) uMap <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- x <.> fmap f y = fmap (. f) x <.> y
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap v wMap)
<> fmap (u . v) wMap
<> fmap ($ w) (fmap (.) uMap <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- fmap f (x <.> y) = fmap (f .) x <.> y
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap v wMap)
<> fmap (u . v) wMap
<> (fmap (($ w) .) (fmap (.) uMap) <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- fmap f (fmap g x) = fmap (f . g) x
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap v wMap)
<> fmap (u . v) wMap
<> (fmap ((($ w) .) . (.)) uMap <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- ((($ w) .) . (.))
-- = \u -> (($ w) .) ((.) u)
-- = \u -> (($ w) .) (u .)
-- = \u -> ($ w) . (u .)
-- = \u v -> ($ w) ((u .) v)
-- = \u v -> ($ w) (u . v)
-- = \u v -> (u . v) w
-- = \u v -> u (v w)
-- = \u v -> u (($ w) v)
-- = \u -> u . ($ w)
-- = (. ($ w))
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap v wMap)
<> fmap (u . v) wMap
<> (fmap (. ($ w)) uMap <.> vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- x <.> (f <$> y) = (. f) <$> x <.> y
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap v wMap)
<> fmap (u . v) wMap
<> (uMap <.> fmap ($ w) vMap)
<> fmap ($ w) (fmap (u .) vMap)
<> fmap ($ w) (fmap (. v) uMap)

cuvwDefault = pure (u (v w))

-- fmap f (fmap g x) = fmap (f . g) w
-- (f . g) = \x -> f (g x)
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap v wMap)
<> fmap (\w -> u (v w)) wMap
<> (uMap <.> fmap ($ w) vMap)
<> fmap (\v -> u (v w)) vMap
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- `fmap (\w -> u (v w)) wMap <> (uMap <.> fmap ($ w) vMap)` commutes because
-- the colliding keys are shadowed by `(uMap <.> (vMap <.> wMap))`
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap v wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap (\w -> u (v w)) wMap
<> fmap (\v -> u (v w)) vMap
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- `fmap u (vMap <.> wMap) <> (uMap <.> fmap v wMap)` commutes because the
-- colliding keys are sahdowed by `(uMap <.> (vMap <.> wMap))`
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (uMap <.> fmap v wMap)
<> fmap u (vMap <.> wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap (\w -> u (v w)) wMap
<> fmap (\v -> u (v w)) vMap
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- `fmap u (vMap <.> wMap) <> (uMap <.> fmap ($ w) vMap)` commutes because the
-- colliding keys are sahdowed by `(uMap <.> (vMap <.> wMap))`
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (uMap <.> fmap v wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap u (vMap <.> wMap)
<> fmap (\w -> u (v w)) wMap
<> fmap (\v -> u (v w)) vMap
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- \w -> u (v w) = u . v
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (uMap <.> (fmap v wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap u (vMap <.> wMap)
<> fmap (u . v) wMap)
<> fmap (\v -> u (v w)) vMap
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- fmap f (fmap g x) = fmap (f . g) x, in reverse
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (uMap <.> (fmap v wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap u (vMap <.> wMap)
<> fmap u (fmap v wMap)
<> fmap (\v -> u (v w)) vMap
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- \v -> u (v w)
-- = \v -> u (($ w) v)
-- = u . ($ w)
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (uMap <.> (fmap v wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap u (vMap <.> wMap)
<> fmap u (fmap v wMap)
<> fmap (u . ($ w)) vMap
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- fmap f (fmap g x) = fmap (f . g) x, in reverse
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (uMap <.> (fmap v wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap u (vMap <.> wMap)
<> fmap u (fmap v wMap)
<> fmap u (fmap ($ w) vMap)
<> fmap (\u -> u (v w)) uMap

cuvwDefault = pure (u (v w))

-- \f -> f x = ($ x)
= Defaultable cuvwMap cuvwDefault
where
cuvwMap =
(uMap <.> (vMap <.> wMap))
<> (uMap <.> (fmap v wMap)
<> (uMap <.> fmap ($ w) vMap)
<> fmap u (vMap <.> wMap)
<> fmap u (fmap v wMap)
<> fmap u (fmap ($ w) vMap)
<> fmap ($ (v w)) uMap

cuvwDefault = pure (u (v w))

-- f <.> (x <> y) = (f <.> x) <> (f <.> y), in reverse
-- fmap f (x <> y) = fmap f x <> fmap f y, in reverse
= Defaultable cuvwMap cuvwDefault
where
cuvwMap = (uMap <.> vwMap) <> fmap u vwMap <> fmap ($ (v w)) uMap
where
vwMap = (vMap <.> wMap) <> fmap v wMap <> fmap ($ w) vMap

cuvwDefault = pure (u (v w))

-- pure f <*> pure x = pure (f x), in reverse
= Defaultable cuvwMap cuvwDefault
where
cuvwMap = (uMap <.> vwMap) <> fmap u vwMap <> fmap ($ (v w)) uMap
where
vwMap = (vMap <.> wMap) <> fmap v wMap <> fmap ($ w) vMap

cuvwDefault = pure u <*> pure (v w)

-- Definition of (<*>), in reverse
= ( Defaultable uMap (pure u)
<*> (Defaultable vwMap vwDefault
where
vwMap = (vMap <.> wMap) <> fmap v wMap <> fmap ($ w) vMap

vwDefault = pure (v w)
)
)

-- pure f <*> pure x = pure (f x), in reverse
= ( Defaultable uMap (pure u)
<*> (Defaultable vwMap vwDefault
where
vwMap = (vMap <.> wMap) <> fmap v wMap <> fmap ($ w) vMap

vwDefault = pure v <*> pure w
)
)

-- Definition of (<*>), in reverse
= ( Defaultable uMap (pure u)
<*> ( Defaultable vMap (pure v)
<*> Defaultable wMap (pure w)
)
)

-- Contract:
-- u = Defaultable uMap uDefault
-- v = Defaultable vMap vDefault
-- w = Defaultable wMap wDefault
= u <*> (v <*> w)
Proof of homomorphism law
pure f <*> pure x

-- pure v = Defaultable mempty (pure v)
= Defaultable mempty (pure f) <*> Defaultable mempty (pure x)

-- Definition of (<*>)
= Defaultable fxMap fxDefault
where
fxMap = (mempty <.> mempty) <> fmap f mempty <> fmap ($ x) mempty

fxDefault = pure f <*> pure x

-- fmap f mempty = mempty
= Defaultable fxMap fxDefault
where
fxMap = (mempty <.> mempty) <> mempty <> mempty

fxDefault = pure f <*> pure x

-- mempty <.> x = mempty
= Defaultable fxMap fxDefault
where
fxMap = mempty <> mempty <> mempty

fxDefault = pure f <*> pure x

-- mempty <> x = x
= Defaultable fxMap fxDefault
where
fxMap = mempty

fxDefault = pure f <*> pure x

-- pure f <*> pure x = pure (f x)
= Defaultable fxMap fxDefault
where
fxMap = mempty

fxDefault = pure (f x)

-- Simplify
= Defaultable mempty (pure (f x))

-- pure v = Defaultable mempty (pure v), in reverse
= pure (f x)
Proof of interchange law
u <*> pure y

-- pure v = Defaultable mempty (pure v)
= u <*> Defaultable mempty (pure y)

-- Expand: u = Defaultable uMap uDefault
= Defaultable uMap uDefault <*> Defaultable mempty (pure y)

-- Definition of (<*>)
= Defaultable uyMap uyDefault
where
uyMap = (uMap <.> mempty) <> uFallback <> fmap ($ y) uMap
where
uFallback =
case uDefault of
Nothing -> mempty
Just u -> fmap y mempty

uyDefault = uDefault <*> pure y

-- fmap f mempty = mempty
= Defaultable uyMap uyDefault
where
uyMap = (uMap <.> mempty) <> uFallback <> fmap ($ y) uMap
where
uFallback =
case uDefault of
Nothing -> mempty
Just u -> mempty

uyDefault = uDefault <*> pure y

-- Simplify `case uDefault of …`
= Defaultable uyMap uyDefault
where
uyMap = (uMap <.> mempty) <> uFallback <> fmap ($ y) uMap
where
uFallback = mempty

uyDefault = uDefault <*> pure y

-- f <.> mempty = mempty
= Defaultable uyMap uyDefault
where
uyMap = mempty <> uFallback <> fmap ($ y) uMap
where
uFallback = mempty

uyDefault = uDefault <*> pure y

-- mempty <> x = x
= Defaultable uyMap uyDefault
where
uyMap = fmap ($ y) uMap

uyDefault = uDefault <*> pure y

-- u <*> pure y = pure ($ y) <*> u
= Defaultable uyMap uyDefault
where
uyMap = fmap ($ y) uMap

uyDefault = pure ($ y) <*> uDefault

-- pure f <*> x = fmap f x
= Defaultable uyMap uyDefault
where
uyMap = fmap ($ y) uMap

uyDefault = fmap ($ y) uDefault

-- Definition of `fmap`, in reverse
= fmap ($ y) (Defaultable uMap uDefault)

-- Contract: u = Defaultable uMap uDefault
= fmap ($ y) u

-- pure f <*> x = fmap f x, in reverse
= pure ($ y) <*> u

by Gabriella Gonzalez (noreply@blogger.com) at June 27, 2022 02:03 AM

June 23, 2022

Monday Morning Haskell

Resources and "Bracket"

During our focus on the IO Monad, we learned a few things about opening and closing file handles. One useful tidbit we learned from this process was the "bracket" pattern. This pattern allows us to manage the acquisition and release of resources in our system. The IO monad is very often concerned with external resources, whether files on our filesystem or operating system resources like thread locks and process IDs.

The general rule behind these kinds of resources is that we do not want our program to be in a state where they are unreachable by code. Another way of saying this is that any code that acquires a release must make sure to release it. For the example of file handles, we can acquire the resource with openFile and release it with hClose.

processFile :: FilePath -> IO ()
processFile fp = do
  -- Acquire resource
  fileHandle <- openFile fp ReadMode
  -- ... Do something with the file
  -- Release the resource
  hClose fileHandle

Now we might want to call this function with an exception handler so that our program doesn't crash if we encounter a serious problem:

main :: IO ()
main = do
  handle ioHandler (processFile "my_file.txt"
  ...
  where
    ioHandler :: IOError -> IO ()
    ioHandler e = putStr "Handled Exception: " >> print e

However, this error handler doesn't have access to the file handle. So it can't actually ensure the handle gets closed. So if our error occurs during the "do something" part of the function, this file handle will still be open.

But now suppose we need to do a second operation on this file that appends to it. If we still have a "Read Mode" handle open, we're not going to be able to open it for appending. So if our handler doesn't close the file, we'll encounter a potentially unnecessary error.

main :: IO ()
main = do
  handle ioHandler (processFile "my_file.txt")
  -- Might fail!
  appendOperation "my_file.txt"
  where
    ioHandler :: IOError -> IO ()
    ioHandler e = putStr "Handled Exception: " >> print e

The solution to this problem is to use the "bracket" pattern of resource usage. Under this pattern, our IO operation has 3 stages:

  1. Acquire the resource
  2. Use the resource
  3. Release the resource

The bracket function has three input arguments for these stages, though the order is 1 -> 3 -> 2:

bracket
  :: IO a -- 1. Acquire the resource
  -> (a -> IO b) -- 3. Release the resource
  -> (a -> IO c) -- 2. Use the resource
  -> IO c -- Final result

Let's add some semantic clarity to this type:

bracket
  :: IO resource -- 1. Acquire
  -> (resource -> IO extra) -- 3. Release
  -> (resource -> IO result) -- 2. Use
  -> IO result -- Result

The "resource" is often a Handle object, thread ID, or an object representing a concurrent lock. The "extra" type is usually the unit (). Most operations that release resources have no essential return value. So no other part of our computation takes this "extra" type as an input.

Now, even if an exception is raised by our operation, the "release" part of the code will be run. So if we rewrite our code in the following way, the file handle will get closed and we'll be able to perform the append operation, even if an exception occurs:

processFile :: FilePath -> IO ()
processFile fp = bracket
  (openFile fp ReadMode) -- 1. Acquire resource
  hClose -- 3. Release resource
  (\fileHandle -> do ... -- 2. Use resource)

main :: IO ()
main = do
  handle ioHandler (processFile "my_file.txt")
  appendOperation "my_file.txt"
  where
    ioHandler :: IOError -> IO ()
    ioHandler e = putStr "Handled Exception: " >> print e

As we discussed in May, the withFile helper does this for you automatically.

Now there are a few different variations you can use with bracket. If you don't need the resource as part of your computation, you can use bracket_ with an underscore. This follows the pattern of other monad functions like mapM_ and forM_ where the underscore indicates we don't use part of the result of a computation.

bracket_
  :: IO resource -- 1. Acquire
  -> (IO extra) -- 3. Release
  -> (IO result) -- 2. Use
  -> IO result -- Result

If you're passing around some kind of global manager object of data and state, this function may simplify your code.

There is also bracketOnError. This will only run the "release" action if an error is encountered. If the "do something" step succeeds, the release step is skipped. So it might apply more if you're trying to use this function as an alternative to handle.

bracketOnError
  :: IO resource -- 1. Acquire
  -> (resource -> IO extra) -- 3. Release if error
  -> (resource -> IO result) -- 2. Use the resource
  -> IO result -- Result

The last example is finally.

finally
  :: IO result
  -> IO extra
  -> IO result

This function is less directly related to resource management. It simply specifies a second action (returning "extra") that will be run once the primary computation ("result") is done, no matter if it succeeds or fails with an exception.

This might remind us of the pattern in other languages of "try/catch/finally". In Haskell, bracket will give you the behavior of the "finally" concept. But if you don't need the resource acquisition step, then you use the finally function.

Hopefully these tricks are helping you to write cleaner Haskell code. We just have a couple more exception patterns to go over this month. If you think you've missed something, you can scroll back on the blog. But you can also subscribe to our mailing list to get a summary at the end of every month!

by James Bowen at June 23, 2022 02:30 PM

Well-Typed.Com

GHC activities report: April-May 2022

This is the twelth edition of our GHC activities report, which describes the work on GHC and related projects that we are doing at Well-Typed. The current edition covers roughly the months of April and May 2022.

You can find the previous editions collected under the ghc-activities-report tag.

A bit of background: One aspect of our work at Well-Typed is to support GHC and the Haskell core infrastructure. Several companies, including IOHK and GitHub via the Haskell Foundation, are providing us with funding to do this work. We are also working with Hasura on better debugging tools. We are very grateful on behalf of the whole Haskell community for the support these companies provide.

If you are interested in also contributing funding to ensure we can continue or even scale up this kind of work, please get in touch.

Of course, GHC is a large community effort, and Well-Typed’s contributions are just a small part of this. This report does not aim to give an exhaustive picture of all GHC work that is ongoing, and there are many fantastic features currently being worked on that are omitted here simply because none of us are currently involved in them in any way. Furthermore, the aspects we do mention are still the work of many people. In many cases, we have just been helping with the last few steps of integration. We are immensely grateful to everyone contributing to GHC. Please keep doing so (or start)!

Team

The current GHC team consists of Ben Gamari, Andreas Klebinger, Matthew Pickering, Zubin Duggal and Sam Derbyshire.

Many others within Well-Typed, including Adam Gundry, Alfredo Di Napoli, Alp Mestanogullari, Douglas Wilson and Oleg Grenrus, are contributing to GHC more occasionally.

Releases

  • Ben, with help from Matt, did quite a bit of release wrangling on the 9.4 branch, producing two alpha releases with a third in the pipeline.

  • Matt has worked on the release management scripts so that all release artifacts are generated by a single CI pipeline which makes it easier to debug mistakes in the release process.

  • Zubin released GHC 9.2.3.

Typechecker

  • Sam helped new contributor CarrieMY improve the typechecking of record updates. This allows record updates involving existentials to work properly, fixing tickets #2595 #3632 #10808 #10856 #16501 #18311 #18802 #21158 and #21289.

  • Sam improved the typechecking of partially applied functions which have representation-polymorphic arguments, such as coerce, unboxed tuples and sums, and newtype constructors with -XUnliftedNewtypes, fixing #21544 and #21650.

  • Sam finished refactoring the representation-polymorphism checks in the typechecker, making use of a new flavour of metavariables (concrete metavariables). This simplifies the implementation and provides the final stepping stone to allow rewriting in RuntimeReps.

  • Matt fixed a compiler panic which was caused by not properly enforcing the Template Haskell level invariants for Typeable instances (!8264).

  • Matt debugged an apparent infinite loop in the compiler which just turned out to be the result of attempting to print out a very large coercion. Removing the print happily fixes the issue (!8263).

  • Matt wrote the Deep Subsumption proposal which aims to reduce the breakage from the simplified subsumption proposal.

Code generation

  • Ben finished his rework of the migration to a clang-based toolchain on Windows (#21019).

  • Ben fixed a calling-convention bug on Windows potentially leading to undefined behavior in programs using SIMD extensions on x86-64 (#21465).

  • Ben finished, committed, and backported his fix for #20959, a correctness bug in GHC’s CAF analysis.

  • Ben fixed the AArch64 NCG’s implementation of the IntMulMayOflo machop and extended the test-primops testsuite to exercise the operation (#21624).

  • Andreas changed dataToTag# code generation to take advantage of the tag inference. In short this means we can sometimes avoid a redundant eval on the argument to dataToTag#.

Core

  • Sam made several improvements to the treatment of casts in the simplifier and the simple optimiser. This was motivated by the work on allowing rewriting in RuntimeReps.

  • Sam fixed a long-standing bug in pushCoercionIntoLambda.

  • Ben continued work on his refactoring removing the somewhat-magical GHC.PrimOpWrappers module (#20155)

  • Matt and Andreas debugged a panic in the optimiser which was caused by doing an incorrect eta-expansion (#21694).

  • Andreas added logic to dump files in !7998 which means dumps for different ways will now be written to different files instead of silently overwriting each other. The old behaviour is still available by passing -fno-dump-with-ways.

  • Andreas fixed a bug where the SpecConstr pass could generate invalid core leading to segfaults in !8096.

  • Andreas worked on reworking some invariants around core unfoldings and call-by-value argument passing in #21497. This will fix #21472 and #21496 once implemented.

  • Andreas fixed #21685. A bug where shadowing at the core level caused the CSE pass to potentially produce invalid core or segfaults.

Runtime system

  • Ben fixed #21556, a bug in the Windows runtime system linker’s treatment of weak symbols.

  • Ben started debugging a bug in the Windows runtime system linker triggered by libc++ (#21618).

  • To aid in debugging #21618, Ben started introducing support for gdb’s JIT interface into GHC’s runtime linker (!8312).

  • Ben debugged and fixed a bug in the bytecode interpreter causing some programs having data constructors with strict fields to crash when run under the interpreter due to missing pointer tags (#21390).

  • Ben fixed a lurking bug in the runtime’s treatment of dead threads triggered when profiling is enabled (!8094, #21438).

  • Matt fixed the fallback path when clock_gettime is not available (!8424).

  • Andreas changed the runtime system to export symbols for utility macros useful for debugging in gdb (!8132).

Error messages

  • Andreas improved -dstg-lint in !7941 and !8012 which will now detect more uses of unsafeCoerce# which are guaranteed to cause segfaults.

Renamer

  • Sam fixed a bug with the computation of unused variables in mdo statements.

Driver

  • Ben introduced response file support (#16476) to GHC’s driver, eliminating a persistent source of headaches for users on Windows.

  • Ben removed GHC’s long-vestigal dependency on libtool and cleaned up some adjacent linking logic (#18826).

Foreign Function Interface

  • Sam strengthened the checks on the kind of Any appearing in foreign import and export declarations, avoiding GHC panics.

Template Haskell

  • Matt fixed the Template Haskell representation of OPAQUE pragmas to avoid a panic (!8133).

Profiling

  • Andreas added a new flag -fno-prof-manual which causes GHC to ignore user-written cost centre annotations. The intended use is to allow suppression of cost centre annotations in libraries far up the dependency chain which one might not care about.
  • Andreas fixed #21429, a bug where profiling cost centres in certain positions could cause segfaults through incorrect demand analysis.

Libraries

  • Ben introduced a meta-package, system-cxx-std-lib, giving users an easy way to bind to C++ dependencies by capturing the link dependencies necessary to link against the C++ standard library (#20010).

  • Ben fixed a long-standing but only recently-noticed bug in GHC’s treatment of file handle closure, allowing file flush failures to go unnoticed when run during handle finalization (#21336).

  • Ben fixed a memory soundness bug in the ghc-heap package’s treatment of weak pointer objects (#21622).

Compiler performance

  • Doug has been working on a “jsem” feature which will allow multiple instances of ghc --make to cooperate in sharing available cores (#19416).

  • Doug and Matt have been investigating parallelising the simplifier.

  • Andreas optimized a few cases where GHC used O(n^2) algorithms, replacing them with code running in O(n*log(n)). This should help in edge cases where modules have a very large number of exported functions.

Packaging

  • Matt has made a number of improvements to the creation of source distributions (!8371).

  • Zubin added CI jobs and modified the release script to generate and upload sources to aid in bootstrapping Hadrian to build GHC without a pre-existing cabal-install installation.

Runtime performance

  • Ben characterised the effect of !7986, a change to make IO performance more predictable in exchange for slightly higher overhead by dropping use of O_NONBLOCK on normal files. In his microbenchmark, he found the change to regress IO performance by roughly 5% (see #15153)

  • Inspired by his work on the O_NONBLOCK change, Ben rebased WJWH’s io_uring-based polling patch (!3794) and extended it to use io_uring for Handle read and write operations. This quickly ended up turning into a proof-of-concept io_uring-based IO manager implementation (!8073)

  • Ben spent some time thinking about a C– optimization to widen narrow (8- and 16-bit) integer expressions to avoid partial register stalls on x86-64.

Infrastructure

  • Ben investigated a bootstrapping failure of 9.4 with the make build system (#21188). Unfortunately, this ended up being attributable to a deficiency in the make build system’s handling of bootstrap dependencies, putting new urgency on the migration to Hadrian.

  • Matt refactored Hadrian to avoid the same bootstrap issues that have broken the make build system (!8339).

  • Ben refactored Hadrian’s treatment of build information from Cabal, fixing a good number of issues in the process (#20566, #20579).

  • Ben continued work on his branch removing the make build system from the tree in preparation for removal later this summer.

  • Ben investigated and fixed a rather perplexing set of Darwin CI failures involving bytestring’s use of ARM’s NEON extensions which ultimately were attributed to Rosetta’s unpredictable choice of execution platform (#21579).

  • Ben investigated another set of equally-bizarre CI failures on Linux, which ended up being due to a bug in Docker’s seccomp filters.

by ben, andreask, matthew, zubin, sam, adam, douglas at June 23, 2022 12:00 AM

Tweag I/O

Incremental builds for Haskell with Bazel

Building Haskell code with Bazel brings some benefits, like builds that are hermetic (i.e. easy to reproduce), caching which allows to switch branches in your version control system and still have fast rebuilds, and tracking of cross-language dependencies.

However, till recently, changes to the source code of a module in a library would require all of the modules in the library to be rebuilt, which could be a serious limitation when using Bazel to build frequent and small changes to the code. This was a consequence of Haskell rules only being able to express dependencies between libraries and binaries.

In this post we describe haskell_module, a new rule in rules_haskell, which allows to express the dependencies between modules. With this rule, Bazel has a higher resolution of the dependency graph and can skip building modules that are certainly not affected by a change.

Building libraries the old way

Suppose we wanted to build a library with only two modules. In Bazel+rules_haskell configuration this would be written as

haskell_library(
    name = "lib",
    srcs = ["A.hs", "B.hs"],
)

This rule produces a Bazel action that ends up calling ghc something like

ghc -no-link A.hs B.hs

which would then be followed by another action to do the linking.

The inputs to this action are the compiler itself and the source files. Bazel ensures that dependencies are not missing in the build configuration by only exposing declared dependencies to the action, this is a strategy most commonly known as sandboxing.

The action above produces as output the files A.hi, B.hi, A.o, and B.o. Unfortunately, Bazel does not allow us to declare these files as both outputs and inputs to the compile action since this would create a loop on the dependency graph. The main consequence, is that we cannot take advantage of ghc’s recompilation checker, which would give the action the chance of not rebuilding an unmodified module, should the action be called upon changes to either of the modules.

Building with haskell_module

With the new haskell_module rule, we can write instead

haskell_library(
    name = "lib",
    modules = [":A", ":B"],
)

haskell_module(
    name = "A",
    src = "A.hs",
)

haskell_module(
    name = "B",
    src = "B.hs",
)

Building with this configuration now creates the actions

ghc -c A.hs
ghc -c B.hs

which are then followed by the linking step. Because modules are built in different actions now, Bazel can distinguish that when only one of the modules has been modified, it doesn’t need to rerun the action to build the other module.

Further build parallelism

Now that the actions are broken down by module, it is possible for Bazel to run these actions in parallel. Consider a new Haskell module that is added to the library lib.

-- C.hs
module C where

import A
import B

We can express the dependencies between the modules with

haskell_module(
    name = "C",
    src = "C.hs",
    deps = [":A", ":B"]
)

haskell_library(
    name = "lib",
    modules = [":A", ":B", ":C"],
)

The dependency graph in Bazel now reflects the dependency graph implied by import declarations in Haskell modules. A first consequence of this, is that Bazel is now on equal footing with ghc to decide how to do parallel builds.

Furthermore, while ghc can build in parallel the modules of a library, Bazel is not limited by the library boundaries. Say module A comes from a library dependency instead of being in the same library as C.

haskell_module(
    name = "C",
    src = "C.hs",
    deps = [":B"],
    # rules_haskell slang to express that ":A" comes
    # from another library (the other library must be listed
    # in narrowed_deps of the enclosing haskell_library)
    cross_library_deps = [":A"],
)

haskell_library(
    name = "lib",
    modules = [":B", ":C"],
    # Other libraries with modules that might be referred
    # as dependencies in haskell_module rules.
    narrowed_deps = [":libA"],
)

haskell_library(
    name = "libA",
    modules = [":A"],
)

In this scenario, Bazel can still build both modules A and B in parallel, while most build tools for Haskell would insist on building library libA ahead of building any module in library lib. And this difference stands even if module C uses Template Haskell.

-- C.hs
{-# LANGUAGE TemplateHaskell #-}
module C where

import A
import B

$(return A.decls)

Now we tell to rules_haskell that C needs Template Haskell with the enable_th attribute.

haskell_module(
    name = "C",
    src = "C.hs",
    deps = [":B"],
    enable_th = True,
    cross_library_deps = [":A"],
)

When bazel tries to build C, it still can build modules A and B in parallel. The effect of enable_th is that the object files of A and B will be exposed to the build action.

ghc -c C.hs A.o B.o ...

Again, it isn’t necessary to build or link libA ahead of building the modules in lib. However, as far as I’m aware rules_haskell is the first implementation to support this.

Keeping source code and build configuration in sync

It would be pretty annoying to update the build configuration every time we add or remove import declarations in a source file. Suppose, module C no longer needs module A.

module C where

-- import A
import B

Now our build configuration is outdated since it incorrectly claims that module C depends on module A. While removing the dependency in the build configuration is not difficult, we get little help from rules_haskell to detect and fix these situations that are all too common when a project is under active development.

To streamline edits to the configuration, Tweag has developed gazelle_haskell_modules, a gazelle extension that scans the source code and updates the build configuration automatically. Whenever the import declarations change in any module of a project, a single invocation of a command will update the build configuration to match it.

bazel run //:gazelle_haskell_modules

gazelle_haskell_modules will discover Haskell modules, it will update the dependencies of library and haskell_module rules, it will update the enable_th attribute, it will add haskell_module rules when new source files are added, and it can remove haskell_module rules when source files are deleted or moved.

Limitations

When lowering the granularity of build actions, new kinds of phenomena enter the scene. If we start progressively diminishing the amount of work that each action does, eventually housekeeping tasks that we perform for each action will start having a cost comparable to that of the action itself.

One of the new outstanding overheads comes from sandboxing. There are different techniques to implement sandboxing, and it turns out that the sandboxing done by default in Bazel can account for near 20% of the builds when using haskell_module. This can be reduced by either putting the sandboxes in an in-memory file system (Bazel option --sandbox_base) or by reusing part of the setup from one sandbox to another (Bazel option --experimental_reuse_sandbox_directories).

Another source of overhead is the startup time of the compiler. When ghc is requested to build a module, it first needs to read the interface files of any imported modules, and may need to read interface files of other transitively imported modules. All this reading and loading needs to be done for every invocation, and when using haskell_module it can account for 25% of the build time once we have eliminated sandboxing overheads. The remedy to reduce the startup overhead is to use a compilation server or persistent worker in Bazel slang. By having a compiler running on a process that can receive multiple requests to compile modules, we only pay for the startup costs once. Unfortunately, implementing a persistent worker that handles sandboxing correctly poses a few challenges that still need to be solved.

Lastly, disabling ghcs recompilation checker is a limitation. If there is a change in a module M deep in the dependency graph, chances are that the change won’t become visible in the interface file of every module that imports M transitively. This is because interface files only account for some aspects of a module which are relevant to the modules that import them. At the point where interface files don’t change anymore, the recompilation checker would kick in if artifacts from earlier compilations are made available to the build process.

rules_haskell, however, has to provide the interface file of M as input to the actions that build every module that imports M transitively. This is because the compiler might need to read the interface file for M, and the build system can’t predict reliably whether it will be needed or not. If the interface file of M changes, then Bazel will arrange to rerun all those actions whether the interface files of the modules along the path in the dependency graph are changed or not. A remedy for this could be to use a feature of Bazel to report when an input hasn’t been used (i.e. unused_inputs_list), which at the time of this writing still needs to be investigated.

Performance assessment

In general, we found that haskell_module can build faster than haskell_library alone whether builds are incremental or not. The typical setup with haskell_library doesn’t try to use multicore support in ghc, and thus the finer-grained haskell_module rules would use more parallelism. For instance, building Cabal with haskell_library alone takes 2 minutes, whereas using haskell_module and 8 CPUs takes 1 minute.

When compared to stack or cabal-install, haskell_module can do faster on builds from scratch. This is, again, because Bazel can use more parallelism than the Haskell-specific tools, which usually would involve at most one CPU per package.

When doing incremental builds, though, both stack and cabal-install can use the recompilation checker, and for changes deep in the dependency graph with little propagation, haskell_module is not able to beat them yet. For changes near the build targets, or which force more recompilation, haskell_module would be more competitive.

Closing remarks

This project was possible thanks to the generous funding from Symbiont and their test bed for the initial implementation. In this post we showed how haskell_module and gazelle_haskell_module can be used to have incremental builds and more build parallelism than was possible with existing tools. In combination with gazelle_cabal, it is easier to migrate a Haskell project to use Bazel these days, and then to generate finely-grained haskell_module rules with gazelle_haskell_modules. We look forward to hearing from your experience with these tools and receiving your contributions!

June 23, 2022 12:00 AM

June 22, 2022

Oliver Charles

The list of monoids pattern

Hello! Yes, this blog is still alive. In this post, I want to share a small little pattern that I’ve found to have a surprisingly high quality-of-life improvement, and I call it the list of monoids pattern.

The idea is that whenever we have a monoidal value - a type that is an instance of the Monoid type class - we can sometimes produce a more ergonomic API if we change our functions to instead to a list of these monoidal values.

I recently proposed an instance of this pattern to lucid, and it was well received and ultimately merged as part of the new lucid2 package. To motivate this post, I’m going to start by reiterating my proposal.

lucid is a domain specific language for producing HTML documents. In lucid we have the Attribute type which represents a single key-value pairing. When we construct HTML elements, we supply a [Attribute] list. For example,

div_ :: [Attribute] -> Html a -> Html a

(Note that lucid has an overloading mechanism, and this is one possible type of div_).

The problem with this API is that it makes it difficult to abstract groups of attributes and reuse them.

Example 1

My motivation came from using the fantastic HTMX library, and wanting to group a common set of attributes that are needed whenever you connect an element with an end-point that serves server-sent events. More specifically, I wanted to “tail” an event stream, automatically scrolling the latest element of the stream into view (using Alpine.js). An example of the attributes are:

<div
   hx-sse="connect:/stream"
   hx-swap="beforeend"
   x-on:sse-message.camel="$el.scrollIntoView(false);">

In lucid, we can express this as:

div_
  [ makeAttribute "hx-sse" "connect:/stream"
  , makeAttribute "hx-swap" "beforeend"
  , makeAttribute "x-on:sse-message.camel" "$el.scrollIntoView(false);"
  ]

This is great, but my problem is wanting to re-use these attributes. If I have another page that I also want to have a stream in, I could copy these attributes, but the programmer in me is unhappy with that. Instead, I want to try and share this definition.

One option is:

tailSSE url =
  [ makeAttribute "hx-sse" $ "connect:" <> url
  , makeAttribute "hx-swap" "beforeend"
  , makeAttribute "x-on:sse-message.camel" "$el.scrollIntoView(false);"
  ]

But look what happens when I use this:

div_
  (concat [ tailSSE "/stream", [ class_ "stream-container" ] ])

Urgh! Just using this requires that I call concat, and to use more attributes I have to nest them in another list, and then I have to surround the whole thing in parenthesis. Worse, look what happens if we consider this code in the context of more “ordinary” HTML:

div_ [class_ "page"] do
  h1_ "Heading"
  div_
    [class_ "scroll"]
    do
      div_
        ( concat
            [ tailSSE "/stream",
              [ class_ "stream-container",
                id_ "stream-1"
              ]
            ]
        )

Our SSE attributes stand out like a sore thumb, ruining the nice DSL that lucid gives us.

At this point, we need to start thinking about ways to fix this.

Before we get to that, let’s look at one more example

Example 2

Continuing with lucid, I’m also a user of Tailwind for styling pages. In Tailwind, we combine primitive classes to style our elements. Sometimes, this styling needs to be conditional. When we we layout a list, we might want to emphasize a particular element:

ul_ do
  li_ [ class_ "p-4" ] "Item 1"
  li_ [ class_ "p-4 font-bold" ] "Item 2"
  li_ [ class_ "p-4" ] "Item 3"

Generally this list will come from another container which we want to enumerate over:

ul_ do
  for_ items \item ->
    li_ [ class_ $ if active item then "p-4 font-bold" else "p-4" ]

It’s unfortunate that we’ve had to repeat p-4 here. We could of course factor that out, but what I more generally want to do is define a common attribute for list items, and another attribute that indicates active. Then, for active items I can just conditionally add the “active” element:

ul_ do
  for_ items \item ->
    li_
      [ class_ "p-4"
      , if active item then class_ "font-bold" else ???
      ]
      (toHTML (caption item))

But what we are we going to put for ???? There isn’t really an “identity” attribute. A common hack is to add class_ "", but that is definitely a hack.

Solutions

If you see both of these problems, a natural reaction might be to make Attribute an instance of Monoid. We might change the type of

div_ :: Attributes -> Html a -> Html a

However, when we do this we momentarily make things a little worse. Starting with the second example 2:

ul_ do
  for_ items \item ->
    li_
      ( mconcat
          [ class_ "p-4",
          , if active item then class_ "font-bold" else mempty
          ]
      )
      (toHTML (caption item))

Our ??? becomes mempty which literally means “no attributes at all”. This solves our problem, but the cost is that overall the API has got more verbose.

How about our first example?

div_ (class_ "page") do
  h1_ "Heading"
  div_
    (class_ "scroll")
    do
      div_
        ( mconcat
            [ tailSSE "/stream",
            , class_ "stream-container",
            , id_ "stream-1"
            ]
        )

The result here is somewhat mixed. Applying a single attribute isn’t too bad, but my main objection to this was that it’s inconsistent, and here it’s even more inconsistent - a single attribute uses parethesis, but multiple attributes need a call to mconcat. It is nice though that our tailSSE call no longer sticks out, and just looks like any other attribute.

The list of monoids pattern

With that setup, I can now present my solution - the list of monoids pattern. As the name suggests, the trick is to simply change our Attributes argument to now be a [Attributes]. This is essentially a list-of-lists of key-value pairs, which is probably not our first instinct when creating this API. However, I think it pays of when we try and lay out HTML using lucid:

div_ [ class_ "page" ] do
  h1_ "Heading"
  div_
    [ class_ "scroll" ]
    do
      div_
        [ tailSSE "/stream",
        , class_ "stream-container",
        , id_ "stream-1"
        ]

We’re back to where we started! However, we’ve also retained a solution to the second example:

ul_ do
  for_ items \item ->
    li_
      [ class_ "p-4",
      , if active item then class_ "font-bold" else mempty
      ]
      (toHTML (caption item))

lucid could go further

Interestingly, once I had this observation I realised that lucid could actually go further. Html a is a Monoid, but notice that when we construct a div_ we supply a single Html a. This post suggests that an alternative API is instead:

div_ :: [Attributes] -> [Html a] -> Html a

Users of Elm might be getting a sense of déjà vu, as this is very similar to the type that Elm uses! I like this form because I think it makes HTML documents much more regular:

div_
  [ class_ "p-4 font-bold" ]
  [ p_ "Paragraph 1"
  , img_ [ src_ "haskell.gif" ]
  , p_ "More"
  ]

Elm falls short of the pattern advocated in this blog post, as both attributes and html elements lack an identity element, so while Elm uses lists, they aren’t lists of monoidal values.

optparse-applicative

I want to briefly compare this to the API in optparse-applicative. Gabriella postulated that optparse-applicative might be more approachable if it used records instead of its current monoidal based API. While I don’t disagree, I want to suggest that the list-of-monoids pattern here might also help.

When we use optparse-applicative, we often end up with code like:

flag
  True
  False
   ( long "no-extensions"
   <> short 'E'
   <> help "Don't show the possible extensions for physical files" 
   )

Here we’re using a <> like a list. Unfortunately, this has two problems:

  • Automatic code formatters already have a way to format lists, but they aren’t generally aware that we’re using <> as if it were constructing a list. This leads to either unexpected formatting, or special-casing within the formatter.

  • It’s less discoverable to new Haskell users that they can supply multiple values. Even an experienced Haskell user will likely have to look up the type and spot the Monoid instance.

If optparse-applicative instead used a list of monoids, the API would be a little more succinct for users, while not losing any functionality:

flag
  True
  False
  [ long "no-extensions"
  , short 'E'
  , help "Don't show the possible extensions for physical files"
  ]

Modifiers can be grouped and abstracted as before, and if we want to compute modifiers with an option to produce no modifiers at all, we can still return mempty. However users are no longer burdened with needing to combine modifiers using <>, and can instead lean on Haskell’s special syntax for lists.

Concluding thoughts

If at this point you’re somewhat underwhelmed by this blog post, don’t worry! This pattern is extremely simple - there are no complex tricks required, it’s just literally wrapping things in a list, moving a call to mconcat, and you’re done. However, I think the implications are fairly significant, and I highly recommend you give this a try.

by Oliver Charles at June 22, 2022 12:00 AM

June 20, 2022

Brent Yorgey

Swarm: status report

Swarm is a 2D programming and resource gathering game, written in Haskell. I announced it last September and gave an update one week after that, but haven’t written anything since then. However, that doesn’t mean development has stopped! Since last October, the repo has grown by an additional 4K lines of Haskell code (to 12K). Notable changes since then include:

  • Many UI improvements, like a main menu, ASCII art recipes, and mouse support
  • Many new entities and recipes (multiple types of motors, drills, and mines; new materials like iron, silver, quartz, and glass; new devices like circuits, clocks, cameras, comparators, compasses, …)
  • The game now supports scenarios, generically describing how to initialize the game and what the winning conditions should be: scenarios can be used to describe open-world games, tutorials, tricky programming challenges, … etc.
  • Improvements to the programming language, like a new dedicated robot type, and a “delay” type for things that should be evaluated lazily
  • Increased emphasis on exploration, with the ability to scan unknown entities in the world

Development has picked up considerably in the past few weeks, and we’re making good progress toward a planned alpha release (though no concrete plans in terms of a release date yet). If you’re interested in getting involved, check out our contribution guide, come join us on IRC (#swarm on Libera.Chat) and take a look at the list of issues marked “low-hanging fruit”—as of this writing there are 28 such issues, so plenty of tasks for everyone!

by Brent at June 20, 2022 04:26 PM

Monday Morning Haskell

Further Filtering

In our previous article we explored the functions catchJust and handleJust. These allow us to do some more specific filtering on the exceptions we're trying to catch. With catch and handle, we'll catch all exceptions of a particular type. And in many cases, especially where we have pre-defined our own error type, this is a useful behavior.

However, we can also consider cases with built-in error types, like IOError. There are a lot of different IO errors our program could throw. And sometimes, we only watch to catch a few of them.

Let's consider just two of these examples. In both actions of this function, we'll open a file and print its first line. But in the first example, the file itself does not exist. In the second example, the file exists, but we'll also open the file in "Write Mode" while we're reading it.

main :: IO ()
main = do
  action1
  action2
  where
    action1 = do
      h <- openFile "does_not_exist.txt" ReadMode
      firstLine <- hGetLine h
      putStrLn firstLine
      hClose h
    action2 = do
      h <- openFile "does_exist.txt" ReadMode
      firstLine <- hGetLine h
      putStrLn firstLine
      h2 <- openFile "does_exist.txt" WriteMode
      hPutStrLn h2 "Hello World"
      hClose h

These will cause two different kinds of IOError. But we can catch them both with a handler function:

main :: IO ()
main = do
  handle handler action1
  handle handler action2
  where
    action1 = do
      h <- openFile "does_not_exist.txt" ReadMode
      firstLine <- hGetLine h
      putStrLn firstLine
      hClose h
    action2 = do
      h <- openFile "does_exist.txt" ReadMode
      firstLine <- hGetLine h
      putStrLn firstLine
      h2 <- openFile "does_exist.txt" WriteMode
      hPutStrLn h2 "Hello World"
      hClose h

    handler :: IOError -> IO ()
    handler e = print e

And now we can run this and see both errors are printed.

>> stack exec my-program
does_not_exist.txt: openFile: does not exist (No such file or directory)
First line
does_exist.txt: openFile: resource busy (file is locked)

But suppose we only anticipated our program encountering the "does not exist" error. We don't expect a "resource busy" error, so we want our program to crash if it happens so we are forced to fix it. We need to filter the error types and use handleJust instead.

Luckily, there are many predicates on IOErrors like isDoesNotExistError. We can use this to write our own predicate:

-- Library function
isDoesNotExistError :: IOError -> Bool

-- Our predicate
filterIO :: IOError -> Maybe IOError
filterIO e = if isDoesNotExistError e
  then Just e
  else Nothing

Now let's quickly recall the type signatures of catchJust and handleJust:

catchJust :: Exception e =>
  (e -> Maybe b) ->
  IO a ->
  (b -> IO a) ->
  IO a

handleJust :: Exception e =>
  (e -> Maybe b) ->
  (b -> IO a) ->
  IO a ->
  IO a

We can rewrite our function now so it only captures the "does not exist" error. We'll add the predicate, and use it with handleJust, along with our existing handler.

main :: IO ()
main = do
  handleJust filterIO handler action1
  handleJust filterIO handler action2
  where
    action1 = do
      h <- openFile "does_not_exist.txt" ReadMode
      firstLine <- hGetLine h
      putStrLn firstLine
      hClose h
    action2 = do
      h <- openFile "does_exist.txt" ReadMode
      firstLine <- hGetLine h
      putStrLn firstLine
      h2 <- openFile "does_exist.txt" WriteMode
      hPutStrLn h2 "Hello World"
      hClose h

    handler :: IOError -> IO ()
    handler e = putStr "Caught error: " >> print e

    filterIO :: IOError -> Maybe IOError
    filterIO e = if isDoesNotExistError e
      then Just e
      else Nothing

When we run the program, we'll see that the first error is caught. We see our custom message "Caught error" instead of the program name. But in the second instance, our program crashes!

>> stack exec my-program
Caught error: does_not_exist.txt: openFile: does not exist (no such file or directory)
First line
my-program: does_exist.txt: openFile: resource busy (file is locked)

Hopefully this provides you with a clear and practical example of how you can use these filtering functions for handling your errors! Next time, we'll take a deeper look at the "bracket" pattern. We touched on this during IO month, but it's an important concept, and there are more helper functions we can incorporate with it! So make sure to stop back here later in the week. And also make sure to subscribe to our monthly newsletter if you haven't already!

by James Bowen at June 20, 2022 02:30 PM

June 15, 2022

Magnus Therning

Power-mode in Spacemacs

I just found the Power Mode for Emacs. If you want to try it out in Spacemacs you can make sure that your ~/.spacemacs contains the following

dotspacemacs-additional-packages
'(
  ...
  (power-mode :location (recipe
                         :fetcher github
                         :repo "elizagamedev/power-mode.el"))
  )

After a restart Power Mode can be turned on using SPC SPC power-mode.

Unfortunately I found that it slows down rendering so badly that Emacs isn't keeping up with my typing. Even though I removed it right away again it was fun to try it out, and I did learn how to add package to Spacemacs that aren't on MELPA.

A useful resource is this reference on the recipe format.

June 15, 2022 06:08 AM

June 10, 2022

Gabriella Gonzalez

The appeal of bidirectional type-checking

The appeal of bidirectional type-checking

In this post I hope to explain why bidirectional type-checking has a lot of cultural cachet within the programming language theory community. To be clear, I’m an amateur and I have no formal background in computer science or type theory. Nonetheless, I believe I’ve learned enough and compared notes with others to motivate bidirectional type-checking.

Subtyping

The fundamental problem that bidirectional type-checking solves well is subtyping. I will not explain why in this post, so for now take it as an article of faith that bidirectional type-checking provides a simple and consistent framework for supporting subtyping within a programming language’s type system.

Bidirectional type-checking does other things well, too (such as inferring the types of inputs from outputs), but subtyping is the thing that bidirectional type-checking does uniquely well. For example, you can infer inputs from outputs using unification instead of bidirectional type-checking, which is why I don't motivate bidirectional type-checking in terms of doing inference "in reverse".

By “subtyping”, I mean that a type A is a subtype of type B if any expression of type A is also a valid expression of type B.

For example, we could create a language that provides two numeric types:

  • Natural - The type of non-negative integers (ℕ)

  • Integer - The type of all integers (ℤ)

Furthermore, we could define Natural to be a subtype of Integer (i.e. ℕ ⊆ ℤ). In other words, if a scalar literal like 4 were a valid Natural number then 4 would also be a valid Integer, too. That would permit us to write something like this Fall-from-Grace code:

let x = 4 : Natural

in [ x, -5 ] : List Integer

… and the type-checker would not complain that we put a Natural number inside a List of Integers, because a Natural number is a subtype of an Integer.

Why subtyping matters

Now, automatic numeric coercions like that are convenient but in the grand scheme of things they are not a big deal from a language implementer’s point of view. The real appeal of subtyping is that subtyping appears in more places than you’d expect.

I’m not even talking about object-oriented subtyping like “The Dog class is a subtype of the Animal class”. Subtyping occurs quite frequently in even non-OOP languages, in the form of universal quantification (a.k.a. “polymorphism” or “generics”).

For example, we can define a polymorphic identity function, but then use the function as if it had a narrower (more specialized) type:

let id : forall (a : Type) . a -> a
= \x -> x

in id : Text -> Text

… and that works because forall (a : Type) . a -> a is a subtype of Text -> Text.

Yes, you read that right; I didn’t get the subtyping direction backwards. A polymorphic type is a subtype of a more specialized type.

In fact, forall (a : Type) . a -> a is a subtype of multiple types, such as:

  • Integer -> Integer
  • List Natural -> List Natural
  • { x: Bool } -> { x: Bool }

… and so on. This might be a bit counter-intuitive if you come from an OOP background where usually each type is a subtype of at most one other (explicitly declared) supertype.

This type specialization is implicit, meaning that we don’t need the type annotation to use id on a value of type Text. Instead, the specialization happens automatically, so we can use id on any value without annotations:

let id : forall (a : Type) . a -> a
= \x -> x

in id "ABC"

If our language didn’t support subtyping then we’d need to explicitly abstract over and apply type arguments. For example, this is how that same id example would work in Dhall (which does not support any form of subtyping):

let id : forall (a : Type) -> a -> a
= \(a : Type) -> \(x : a) -> x
-- ↑ Explicit type abstraction

in id Text "ABC"
-- ↑ Explicit type application

Typically I refer to a Dhall-style type system as “explicit type abstraction and type application”. Vice versa, I refer to a Grace-style language as “implicit type abstraction and type application”.

The real reason why subtyping matters is because you need to support subtyping in order to implement a language with “implicit type abstraction/application”. In other words, you need some way to automatically produce and consume polymorphic types without explicit types or type annotations from the end user and that is essentially a form of subtyping.

Bidirectional type-checking vs unification

Bidirectional type-checking is not the only way to implement a language with implicit type abstraction/application. For example, Hindley-Milner type inference is a type system that is based on unification instead of bidirectional type-checking type system and Hindley-Milner inference still supports implicit type abstraction/application.

The reason why Hindley Milner type inference works, though, is by exploiting a narrow set of requirements that do not generalize well to more sophisticated type systems. Specifically:

  • Hindley Milner type inference only permits “top-level polymorphism”

    In other words, Hindley Milner type inference only permits universal quantification at the top-level of a program or at the top-level of a let-bound expression

  • Hindley Milner type inference does not support any other form of subtyping

To expand on the latter point, language implementers often want to add other forms of subtyping to our languages beyond implicit type abstraction/application, such as:

… and if you try to extend Hindley Milner type inference to add any other language feature that requires subtyping then you run into issues pretty quickly. This is because Hindley Milner exploits a cute hack that does not generalize well.

In contrast, bidirectional type-checking is not just a method for handling implicit type abstraction/application. Rather, bidirectional type-checking is a framework for introducing any form of subtyping, of which implicit type abstraction/application is just a special case.

Conclusion

If you’d like to learn more about bidirectional type-checking then I recommend reading:

As I mentioned earlier, bidirectional type-checking has a lot of cultural cachet within the programming language theory community, so even if you don’t plan on using it, understanding it will help you speak the same language as many others within the community.

by Gabriella Gonzalez (noreply@blogger.com) at June 10, 2022 10:04 PM

June 09, 2022

Philip Wadler

Should PLDI join PACMPL?

Via a tweet, the PLDI steering community is asking whether PLDI should join PACMPL. Have your say! (My vote is yes.) 

Should @pldi join ICFP, OOPSLA and POPL in publishing its proceedings in the PACM-PL journal? The PLDI Steering Committee would appreciate your views. Please complete this short survey before the end of Thursday 16 June (AoE): https://forms.office.com/r/HjwYvq1CGw


by Philip Wadler (noreply@blogger.com) at June 09, 2022 09:38 PM

Gabriella Gonzalez

Folds are constructor substitution

Folds are constructor substitution

I notice that functional programming beginners and experts understand the word “fold” to mean subtly different things, so I’d like to explain what experienced functional programmers usually mean when they use the term “fold”. This post assumes a passing familiarity with Haskell.

Overview

A “fold” is a function that replaces all constructors of a datatype with corresponding expressions. “fold”s are not limited to lists, linear sequences, or even containers; you can fold any inductively defined datatype.

To explain the more general notion of a “fold”, we’ll consider three representative data structures:

  • lists
  • Maybe values
  • binary trees

… and show how we can automatically derive the “one true fold” for each data structure by following the same general principle.

Lists

Many beginners understand the word “fold” to be a way to reduce some collection of values (e.g. a list) to a single value. For example, in Haskell you can add up the elements of a list like this:

sum :: [Int] -> Int
sum xs = foldr (+) 0 xs

… where sum reduces a sequence of Ints to a single Int by starting from an initial accumulator value of 0 and then “folding” each element of the list into the accumulator using (+).

Haskell’s standard library provides at least two fold functions named foldl and foldr, but only foldr is the “canonical” fold for a list. By “canonical” I mean that foldr is the only fold that works by substituting list constructors.

We can more easily see this if we define our own linked list type with explicitly named constructors:

data List a = Cons a (List a) | Nil

… where instead of writing a list as [ 1, 2, 3 ] we instead will write such a list as:

example :: List Int
example = Cons 1 (Cons 2 (Cons 3 Nil))

This is a very unergonomic representation for a list, but bear with me!

We can implement the “canonical” fold for the above List type as a function that takes two arguments:

  • The first argument (named cons) replaces all occurrences of the Cons constructor
  • The second argument (named nil) replaces all occurrences of the Nil constructor

The implementation of the canonical fold looks like this:

fold :: (a -> list -> list) -> list -> List a -> list
fold cons nil (Cons x xs) = cons x (fold cons nil xs)
fold cons nil Nil = nil

You might not necessarily follow how that implementation works, so a more direct way to appreciate how fold works is to see how the function behaves on some sample inputs:

-- The general case, step-by-step
fold cons nil (Cons x (Cons y (Cons z Nil)))
= cons x (fold cons nil (Cons y (Cons z Nil)))
= cons x (cons y (fold cons nil (Cons z Nil)))
= cons x (cons y (cons z (fold cons nil Nil)))
= cons x (cons y (cons z nil))

-- Add up the elements of the list, but skipping more steps this time
fold (+) 0 (Cons 1 (Cons 2 (Cons 3 Nil)))
= (+) 1 ((+) 2 ((+) 3 0))
= 1 + (2 + (3 + 0))
= 6

-- Calculate the list length
fold (\_ n -> n + 1) 0 (Cons True (Cons False (Cons True Nil)))
= (\_ n -> n + 1) True ((\_ n -> n + 1) False ((\_ n -> n + 1) True 0))
= (\_ n -> n + 1) True ((\_ n -> n + 1) False 1)
= (\_ n -> n + 1) True 2
= 3

Notice that if we format the type of fold a bit we can see that the type of each argument to fold (sort of) matches the type of the corresponding constructor they replace:

fold
:: (a -> list -> list) -- Cons :: a -> List a -> List a
-> list -- Nil :: List a
-> List a
-> list

In the above type, list is actually a type variable and we could have used any name for that type variable instead of list, such as b. In fact, if we were to replace list with b, we would get essentially the same type as foldr for Haskell lists:

-- Our `fold` type, replacing `list` with `b`
fold
:: (a -> b -> b)
-> b
-> List a
-> b

-- Now compare that type to the `foldr` type from the Prelude:
foldr
:: (a -> b -> b)
-> b
-> [a]
-> b

We commonly use folds to reduce a List to a single scalar value, but folds are actually much more general-purpose than that and they can be used to transform one data structure into another data structure. For example, we can use the same fold function to convert our clumsy List type into the standard Haskell list type, like this:

fold (:) [] (Cons 1 (Cons 2 (Cons 3 Nil)))
= (:) 1 ((:) 2 ((:) 3 []))
= 1 : (2 : (3 : []))
= [ 1, 2, 3 ]

Maybe

Folds are not limited to recursive data types. For example, here is the canonical fold for Haskell’s Maybe type, which is not recursive:

data Maybe a = Nothing | Just a

fold :: maybe -> (a -> maybe) -> Maybe a -> maybe
fold nothing just Nothing = nothing
fold nothing just (Just x ) = just x

In fact, this function already exists in Haskell’s standard library by the name of maybe:

maybe :: b -> (a -> b) -> Maybe a -> b
maybe n _ Nothing = n
maybe _ f (Just x) = f x

Once you think of folds in terms of constructor substitution you can quickly spot these canonical folds for other types.

Binary trees

What about more complex data structures, like the following binary Tree type?

data Tree a = Node a (Tree a) (Tree a) | Leaf

This sort of fold is still straightforward to write, by applying the same principle of constructor substitution:

fold :: (a -> tree -> tree -> tree) -> tree -> Tree a -> tree
fold node leaf (Node x l r) = node x (fold node leaf l) (fold node leaf r)
fold node leaf Leaf = leaf

We only need to keep recursively descending over the Tree, replacing constructors as we go.

We can use this fold to reduce the Tree to a single value, like this:

-- Add up all the nodes in the tree
fold (\x l r -> x + l + r) 0 (Node 1 (Node 2 Leaf Leaf) (Node 3 Leaf Leaf))
= (\x l r -> x + l + r) 1
((\x l r -> x + l + r) 2 0 0)
((\x l r -> x + l + r) 3 0 0)
= (\x l r -> x + l + r) 1
(2 + 0 + 0)
(3 + 0 + 0)
= (\x l r -> x + l + r) 1
2
3
= 1 + 2 + 3
= 6

… or we can use the same fold function to transform the Tree into another data structure, like a list:

-- List `Tree` elements in pre-order
fold (\x l r -> x : l ++ r) [] (Node 1 (Node 2 Leaf Leaf) (Node 3 Leaf Leaf))
= (\x l r -> x : l ++ r) 1
((\x l r -> x : l ++ r) 2 [] [])
((\x l r -> x : l ++ r) 3 [] [])
= (\x l r -> x : l ++ r) 1
(2 : [] ++ [])
(3 : [] ++ [])
= (\x l r -> x : l ++ r) 1
[2]
[3]
= (\x l r -> x : l ++ r) 1
[2]
[3]
= 1 : [2] ++ [3]
= [1, 2, 3]

… even use the fold to reverse the tree:

fold (\x l r -> Node x r l) Leaf (Node 1 (Node 2 Leaf Leaf) (Node 3 Leaf Leaf))
= (\x l r -> Node x r l) 1
((\x l r -> Node x r l) 2 Leaf Leaf)
((\x l r -> Node x r l) 3 Leaf Leaf)
= (\x l r -> Node x r l) 1
(Node 2 Leaf Leaf)
(Node 3 Leaf Leaf)
= Node 1 (Node 3 Leaf Leaf) (Node 2 Leaf Leaf)

Generality

At this point you might be wondering: “what can’t a fold do?”. The answer is: you can do essentially anything with a fold, although it might not necessarily be the most efficient solution to your problem. You can think of a fold as the most general-purpose interface for consuming a data structure because the fold interface is a “lossless” way to process a data structure.

To see why a fold is a “lossless” interface, let’s revisit the fold function for Trees and this time we will pass in the Node and Leaf constructors as the inputs to the fold. In other words, we will replace all occurrences of Node with Node and replace all occurrences of Leaf with Leaf:

fold Node Leaf (Node 1 (Node 2 Leaf Leaf) (Node 3 Leaf Leaf))
= Node 1 (Node 2 Leaf Leaf) (Node 3 Leaf Leaf)

This gives us back the original data structure, demonstrating how we always have the option for a fold to recover the original pristine input. This is what I mean when I say that a fold is a lossless interface.

by Gabriella Gonzalez (noreply@blogger.com) at June 09, 2022 02:23 AM

June 06, 2022

Jeremy Gibbons

Richard Bird, 1943-2022

My mentor, colleague, and friend Richard Bird died in April 2022 after a long battle with cancer. I wrote an obituary of him for The Guardian, his favoured newspaper; this post is a hybrid of that obituary and a eulogy I delivered at his funeral.

Richard was born in 1943 in London. His parents Kay and Jack were landlords of a series of pubs in South London. Richard attended St Olave’s Grammar School, then matriculated to read mathematics at Cambridge in 1960. After graduation, he had a brief spell working in sales for International Computers and Tabulators, the mainframe computer company that eventually became ICL. He was evidently effective in this role, being as personable as he was; but it involved flying around the country in small planes, which Richard did not enjoy at all. No amount of money or approval could persuade him to stay, so he left after a year to do postgraduate study at the University of London Institute of Computer Science.

Richard met Norma Lapworth at a friend’s 21st birthday party. Norma trained as an ecologist, later becoming a teacher, education advisor, and Ofsted schools inspector. Richard and Norma married in 1967.

After a year lecturing in Vancouver 1971-1972, Richard and Norma returned to the UK for Richard to take up a lectureship at the University of Reading. He was given the time finally to complete his PhD thesis, on Computational Complexity on Register Machines, in 1973. But Reading was a rather linguistically oriented department, which did not suit Richard’s mathematical temperament. So he leapt at the opportunity to join Tony Hoare’s expanding Programming Research Group at Oxford University Computing Laboratory in 1983. He was appointed as a University Lecturer in Computation, with a non-tutorial fellowship at St Cross College, assuming responsibility as Director of the MSc in Computation. He moved to a Tutorial Fellowship at Lincoln College in 1988. Richard stayed at the PRG for the remainder of his career, being promoted to Reader in Computation in 1990 and then to Professor in 1996, serving as Director of OUCL from 1998 to 2003, and finally retiring in 2008 after 25 years.

Richard’s research area was functional programming—an approach to computer programming centred around obeying the conventions of traditional mathematics. To Richard, it was self-evident that programs are mathematical entities, and that you can manipulate them in precisely the same way as you do quadratic equations. No immediate need for consideration of the capabilities of computer hardware; no entanglement in the vagaries of specific programming languages. Just algebra, pure and simple.

His PhD thesis in the early 1970s was already concerned with program transformation: the rules by which one program is shown to be equivalent to another. He wasn’t so interested in the program you actually end up with, so much as the process by which you got there, and how you know you ended up in the right place, and what alternative directions you didn’t follow en route. Specifically, he was interested in the calculus of equivalences with which you could conduct equational reasoning on programs, just as in high-school algebra.

In 1980 he was invited to participate in IFIP Working Group 2.1 on Algorithmic Languages and Calculi. This was the group that had designed the seminal programming language Algol68, and that continues to research into notations and techniques for designing and describing algorithms today. Richard had a very fruitful collaboration in WG2.1 with Lambert Meertens from the CWI at Amsterdam, developing what came to be known as the Bird-Meertens Formalism, or Squiggol to its friends. This was a concise and somewhat cryptic algebraic program notation with a powerful accompanying corpus of theorems, supporting the equational reasoning style that Richard sought.

Richard is known and respected around the world for his elegant writing. He published about 100 scientific papers in his life—not especially prolific for a scientist, but every one of those papers was very well polished. He also wrote or co-wrote seven books, starting with Programs and Machines (1976), which featured among other constructions a machine called NoRMa, the Number-Theoretic Register Machine.

I won’t go through all seven books, much as I would like to; I will just pick out a few highlights. Richard’s second book, Introduction to Functional Programming (1988, with Phil Wadler), set out his vision for teaching functional programming. Many people love this book. Graham Hutton wrote to me to say:

Richard set the standards for elegance and clarity, and the original Bird & Wadler is my all-time favourite book. It still stands up today as one of the best programming books ever written—every page is a gem.

Richard’s next book, Algebra of Programming (1996, with Oege de Moor), was much more research-oriented, extending his Squiggol agenda from functions to relations. Tim Sears tweeted:

This book blew my mind regarding what the relationship could be between math, programming and systems. I never met Richard, but his work has had a profound impact on me.

Richard’s fifth book Pearls of Functional Algorithm Design collected and repolished some of the Functional Pearls he had published over the years. These Pearls emulated Jon Bentley’s famous Programming Pearls column in CACM in the 1980s. Bentley explained:

Just as natural pearls grow from grains of sand that have irritated oysters, these programming pearls have grown from real problems that have irritated programmers. The programs are fun, and they teach important programming techniques and fundamental design principles.
Richard established and was the founding editor of the Functional Pearls column, from the very first issue in 1991 of the Journal of Functional Programming until he retired in 2008. He characterized Functional Pearls as “polished, elegant, instructive, entertaining”. He had high standards for them: he advised reviewers to
stop reading when you get bored, or the material gets too complicated, or too much specialist knowledge is needed, or the writing is bad.
He gave a keynote lecture about Pearls in 2006, where he observed that “most of them need more time in the oyster”.

Functional Pearls are the epitome of Richard’s writing style. Erik Meijer called him “the poet laureate of functional programming”. Greg Fitzgerald tweeted that

When life got hard, I’d read his pearls to settle my mind. His proofs read like poetry.
Ed Kmett commented that Richard’s pearls book
is the only real clear exposition of how to think like a functional programmer that I’ve ever seen

I was honoured to be Richard’s co-author on his final book, Algorithm Design with Haskell, written during his illness and published in 2020. Working with him was always a joy.

Despite the global influence of his publications, Richard always saw teaching as his main duty, and research as something he did for relaxation. Andrew Ker thanked Richard for the way he had taught him as a first-year undergraduate—a way which Andrew has passed on to his own students, and which he now sees them repeating in turn. Andrew wrote:

I’ve come to see that passing on our knowledge to the next generation is the most important thing we do, even more than generating new knowledge through research. And the ideas and attitudes persist through the generations. Your influence and example extends far beyond what you know, and countless students are grateful for it.

My own favourite Richard story is a teaching one. Richard was giving a tutorial about algorithm analysis: how many steps does this program take—for example, something about processing hierarchical tree-shaped data? You’d expect a question about counting steps to have an answer in whole numbers; but in fact, the answer often involves logarithms. One student asked why this was. Richard thought for a moment, then said “logs come from trees”. The students broke out in laughter, and it took Richard a while to work out why. In fact, I think this was Richard’s own favourite Richard story too, because I know that he continued to tell it in tutorials for years afterwards.

I want to close by mentioning Richard’s openness, considerateness, and egalitarianism. The departmental secretaries always looked forward to him coming in, because he treated them with the same friendliness that he treated the most eminent of his colleagues. Several once-overawed students have written to me to express gratitude for his welcome—“no Herr Professor Doktor Doktor here, we are all colleagues”—and for helping them to find their own way.

Richard’s collaborator Lambert Meertens says:

His lectures were a joy to attend, because of their clarity and the entertaining delivery. But I’ll remember him at least as much for his kindness, his generosity, his warm interest in people’s wellbeing—expressed not only in words, but also deeds.
Cezar Ionescu remembers being apprehensive about first meeting Richard at WG2.1—you should never meet your heroes—but says he
was exactly the way I had imagined him to be: very sharp, funny, wise. We talked about opera and Shakespeare, and fought (and won!) a hard battle for tap water with the hotel staff.

My own involvement with Richard started when I applied in 1987 to do a DPhil at Oxford, on a project that turned out to have finished by the time I arrived. I rattled around the Lab for a year, not even seeing my official supervisor. But Richard invited me to join his weekly Problem Solving Club. He took me under his wing, and I have never looked back. He became my DPhil supervisor, then some years later hired me into my current position, and I am privileged to have been able to call him my colleague and my friend. I basically owe him my whole career, and I have no words sufficient to convey my gratitude to him. I will miss him greatly.

by jeremygibbons at June 06, 2022 04:00 PM

June 03, 2022

Holden Karau

Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption

 Making Hibernate work on Ubuntu 22.04 (jammy) on the Framework Laptop w/full disk encryption


The 0th step is disabling secure boot in the bios (reboot, press F2 to get into the BIOS, disable secure boot).

The first step is to make a swapfile large enough for your device to suspend to.


sudo swapoff /swapfile

sudo dd if=/dev/zero of=/swapfile bs=$(cat /proc/meminfo | awk '/MemTotal/ {print $2}') count=1024 conv=notrunc

sudo mkswap /swapfile

sudo swapon /swapfile



Then make sure it's enabled on reboot, e.g: echo '/swapfile swap swap defaults 0 0' | sudo tee -a /etc/fstab


If there is another swap partition or file in /etc/fstab you may need to take it out.


Find the device your swapfile is located on (e.g. findmnt  -no source -T /swapfile), note here some guides recommend using the UUID, in my experience, this resulted in some errors.


To find the resume offset of your file run filefrag -v /swapfile and then look at the first value under physical_offset.


Set `GRUB_CMDLINE_LINUX_DEFAULT="splash resume=${RESUME_DEVICE} resume_offset=${RESUME_OFFSET} nvme.noacpi=1"` in `/etc/default/grub` and then update grub with sudo update-grub


If you want to enable hibernation on lid close set `HandleLidSwitch=hibernate` in `/etc/systemd/logind.conf`.


by Holden Karau (noreply@blogger.com) at June 03, 2022 07:49 PM

Mark Jason Dominus

Disabling the awful Macbook screen lock key

(The actual answer is at the very bottom of the article, if you want to skip my complaining.)

My new job wants me to do my work on a Macbook Pro, which in most ways is only a little more terrible than the Linux laptops I am used to. I don't love anything about it, and one of the things I love the least is the Mystery Key. It's the blank one above the delete key:

Upper right corner of a MacBook Pro, with keys as described.

This is sometimes called the power button, and sometimes the TouchID. It is a sort of combined power-lock-unlock button. It has something to do with turning the laptop on and off, putting it to sleep and waking it up again, if you press it in the right way for the right amount of time. I understand that it can also be trained to recognize my fingerprints, which sounds like something I would want to do only a little more than stabbing myself in the eye with a fork.

If you tap the mystery button momentarily, the screen locks, which is very convenient, I guess, if you have to pee a lot. But they put the mystery button right above the delete key, and several times a day I fat-finger the delete key, tap the corner of the mystery button, and the screen locks. Then I have to stop what I am doing and type in my password to unlock the screen again.

No problem, I will just turn off that behavior in the System Preferences. Ha ha, wrong‑o. (Pretend I inserted a sub-article here about the shitty design of the System Preferences app, I'm not in the mood to actually do it.)

Fortunately there is a discussion of the issue on the Apple community support forum. It was posted nearly a year ago, and 316 people have pressed the button that says "I have this question too". But there is no answer. YAAAAAAY community support.

Here it is again. 292 more people have this question. This time there is an answer!

practice will teach your muscle memory from avoiding it.

Two business men in suits and ties.  One is saying “Did you just tll me to go fuck myself?” and the other is replying “I believe I did, Bob.”

This question was tough to search for. I found a lot of questions about disabling touch ID, about configuring the touch ID key to lock the screen, basically every possible incorrect permutation of what I actually wanted. I did eventually find what I wanted on Stack Exchange and on Quora — but no useful answers.

There was a discussion of the issue on Reddit:

How do you turn off the lock screen when you press the Touch ID button on MacBook Pro. Every time I press the Touch ID button it locks my screen and its super irritating. how do I disable this?

I think the answer might be my single favorite Reddit comment ever:

My suggestion would be not to press it unless you want to lock the screen. Why do you keep pressing it if that does something you don't want?

Two business men in suits and ties.  One is saying “Did you just tll me to go fuck myself?” and the other is replying “I believe I did, Bob.”

Victory!

I did find a solution! The key to the mystery was provided by Roslyn Chu. She suggested this page from 2014 which has an incantation that worked back in ancient times. That incantation didn't work on my computer, but it put me on the trail to the right one. I did need to use the defaults command and operate on the com.apple.loginwindow thing, but the property name had changed since 2014. There seems to be no way to interrogate the meaningful property names; you can set anything you want, just like Unix environment variables. But The current developer documentation for the com.apple.loginwindow thing has the current list of properties and one of them is the one I want.

To fix it, run the following command in a terminal:

    defaults write com.apple.loginwindow DisableScreenLockImmediate -bool yes

They documenation claims will it will work on macOS 10.13 and later; it did work on my 12.4 system.

Something something famous Macintosh user experience.

[ The cartoon is from howfuckedismydatabase.com. ]

by Mark Dominus (mjd@plover.com) at June 03, 2022 01:45 AM

June 02, 2022

Tweag I/O

Smooth, non-invasive Haskell Stack and Nix shell integration

This blog post is for developers who want to build their Haskell project with Stack and provide system dependencies and tools with Nix.

TL;DR: create your project with stack new myproject github:tweag/nix-integration.

Stack

Stack is one of the two popular build tools for Haskell projects. It takes care of setting up a build environment by providing the GHC compiler and works hand-in-hand with Stackage snapshots: a specific version of GHC paired with a subset of packages from Hackage, where each package’s version was chosen to make the set self-consistent, relieving developers of the need of finding compatible Haskell package versions.

This post is about Stack as this is our main Haskell build tool at Tweag. Without entering into an emacs-vs-vi kind of debate, a similar post could be written about Cabal, whose interaction with Nix is a bit different.

Nix

Nix is a package manager that provides — among other benefits — the nix-shell, a sort of virtual environment for everything, including:

  • system dependencies, e.g. zlib or your favorite database driver
  • compilers, e.g. ghc or javac

Nix is popular among Haskell developers, and at Tweag. The nix-shell, in particular, is a solution to the dreaded “it works on my machine� class of problems often encountered by teams working on a project. However, there are many reasons why your coworkers or other contributors would not use Nix:

  • they are on Windows and don’t want to or can’t use Windows Subsystem for Linux
  • they can’t install Nix on their machine (e.g. if they don’t have root access)
  • they don’t know how

So while Nix might be great for you, it’s helpful if it’s optional for others. Therefore, any solution combining Nix and Stack should be non-invasive and also work for those who don’t use Nix.

Everything will just work, right?

This is where things get awkward: Stack (via snapshots) and Nix (via nix-shell) partially overlap! Remember, they can both provide GHC. This is unfortunately not a simple “pick one� situation, as we will see below.

We want a solution where:

  • Both Nix users and non-Nix users can work on the project with Stack (non-invasive)
  • Nix users get all the system packages and tools they need from nix-shell (Nix integration)

In this blog post we will:

  • look at common solutions and their pitfalls
  • build a solution that fits all our goals
  • provide templates so that you can easily reuse this solution in your projects

The optimistic solution: Add Stack to Nix shell

This is the very first solution I tried back when I originally had this problem.

The idea is:

  1. nix-shell provides Stack for Nix users
  2. Stack will take care of providing GHC
  3. Profit
# shell.nix
let
  pkgs = ...;
in
pkgs.mkShell {
  buildInputs = [ pkgs.stack ];
}

Let’s see how this fares on a small Haskell project. Say, a project using the excellent Servant library:

# package.yaml
[...]
dependencies:
  - base
  - servant-server

Let’s compile this before we can start hacking on our project:

$ nix-shell
[nix-shell]$ stack build
[...]
zlib > configure
zlib > Configuring zlib-0.6.2.3...
zlib > Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.7: Missing dependency on a foreign
zlib > library:
zlib > * Missing (or bad) header file: zlib.h
zlib > * Missing (or bad) C library: z
zlib > This problem can usually be solved by installing the system package that
zlib > provides this library (you may need the "-dev" version). [...]

Oh, right, Servant depends on the zlib system package. It’s ok, we add it to our shell.nix:

# shell.nix
let
  pkgs = ...;
in
pkgs.mkShell {
  buildInputs = [
    pkgs.stack
    pkgs.zlib
  ];
}

And then we recompile:

$ nix-shell
[nix-shell]$ stack build
[...]
zlib > configure
zlib > Configuring zlib-0.6.2.3...
zlib > Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.7: Missing dependency on a foreign
zlib > library:
zlib > * Missing (or bad) header file: zlib.h
zlib > * Missing (or bad) C library: z
zlib > This problem can usually be solved by installing the system package that
zlib > provides this library (you may need the "-dev" version). [...]

Huh. What is happening here? Is it time to blame a cache? 🤔

The problem is that the zlib package is not visible to the GHC provided by Stack!

Another problem is that if other GHC-backed tools like Haskell Language Server or Hoogle are provided by Nix, then they will use a different GHC from the one used to build the project, leading to all kinds of weird errors.

You can test for yourself that it does not work with the full code of this section.

Key takeaway: For Nix users, GHC should be provided by Nix, and Stack should use that GHC.

The invasive solution: Use Stack-Nix integration

Stack conveniently provides Nix integration. This is exactly what we were looking for! In particular, Stack will use the Nix-provided GHC, and we can specify extra Nix packages like zlib that Stack will use during compilation.

This integration can take two forms:

  • Lightweight, by passing configuration flags in stack.yaml or in the command-line
  • Full, by passing a configuration flag pointing at a Nix file

For the remainder of this post, we will only use the “full� form, but every solution described below also works in the “lightweight� form.

In essence:

# shell.nix
let
  pkgs = ...;
in
pkgs.mkShell {
  buildInputs = [ pkgs.stack ];
  # Note that Stack relies on the `NIX_PATH` environment variable to discover
  # the `nixpkgs` and obtain the right `ghc`.
  NIX_PATH = "nixpkgs=" + pkgs.path;
}
# stack.yaml
resolver: lts-18.28
packages:
  - .
nix:
  enable: true
  pure: false
  # See https://docs.haskellstack.org/en/stable/nix_integration/#using-a-custom-shellnix-file
  shell-file: nix/stack-integration.nix

Lo and behold, our project now compiles! �

But in the process, Nix concerns have now leaked into the non-Nix file stack.yaml! It means that non-Nix users are no longer able to build our project and contribute �

A less invasive alternative is to pass all those Nix flags in the command line: non-Nix users would not be impacted. But then Nix users would need to pass --nix --no-nix-pure --nix-shell-file=nix/stack-integration.nix options on every stack command! Not only is this inconvenient, but it is also easy to forget, and in that case, you would have weird or confusing errors.

You can test for yourself with the full code of this section.

Key takeaways:

  • We need a way to pass Stack-Nix integration flags only when Stack is used by Nix-users
  • We must not modify non-Nix files (e.g. we must not modify stack.yaml or package.yaml)

The solution

We want to:

  • change some behavior for Nix users only, so the solution must happen in the Nix files
  • change the behavior of Stack in particular, and it is provided by shell.nix, so the solution must happen in shell.nix
  • provide some Nix flags to the stack command-line interface by default, so we need a way to “overrideâ€�, “aliasâ€� or “wrapâ€� the stack executable.

A great candidate for this job is Nix’s wrapProgram shell function from the makeWrapper Nix package!

We can wrap stack into a new executable (conveniently named stack) where the Nix flags are set, and provide this wrapped, enhanced stack rather than the default one in the shell:

let
  pkgs = ...;

  stack-wrapped = pkgs.symlinkJoin {
    name = "stack";
    paths = [ pkgs.stack ];
    buildInputs = [ pkgs.makeWrapper ];
    postBuild = ''
      wrapProgram $out/bin/stack \
        --add-flags "\
          --nix \
          --no-nix-pure \
          --nix-shell-file=nix/stack-integration.nix \
        "
    '';
  };

in
pkgs.mkShell {
  # Do NOT use `stack`, otherwise system dependencies like `zlib` are missing at compilation
  buildInputs = [ stack-wrapped ];
  NIX_PATH = "nixpkgs=" + pkgs.path;
}

Here I chose to use the full Stack-Nix integration by pointing Stack at a nix/stack-integration.nix file, but this would work similarly with the lightweight integration, by passing all Stack-Nix flags to wrapProgram, e.g., --nix-packages zlib.

In both cases, the NIX_PATH environment variable customization is also required by Stack-Nix.

Here is how the implementation works: symlinkJoin creates a copy (with symbolic links) of the stack output in the Nix store; after this is done, wrapProgram renames stack to .stack-wrapped and names the wrapper stack. This way, if you need the vanilla stack, without the extra arguments (e.g. for debugging purposes), it is available in the nix-shell as .stack-wrapped.

You can see the full code for yourself.

Stack template

Besides the full code linked above, I also prepared a Stack template, so all you need to do is:

stack new myproject github:tweag/nix-integration

and you will have a Haskell project with full Stack-Nix integration!

Conclusion

This blog post was as much about giving a decent solution to Stack + Nix integration as it was about explaining how one can solve this kind of problem, taking into account both human constraints (e.g. not everyone uses Nix) and technical constraints (e.g. it must compile!).

Don’t hesitate to open an issue to either repository (code examples and the template) if you have questions, remarks, suggestions!

June 02, 2022 12:00 AM

June 01, 2022

Well-Typed.Com

Well-Typed Advanced Track at ZuriHac 2022

At this year’s ZuriHac that will take place 11–13 June 2022, we will once again offer an Advanced Track, comprising two independent workshops about topics related to Haskell. Please do not let the label “Advanced Track” fool you: Both workshops are targeted at wide audiences, and you do not at all have to be a Haskell expert to attend. More about the prerequisites for each session is explained below.

We would be delighted to welcome you at these sessions or to discuss any interesting topics with you at ZuriHac. This year, ZuriHac and therefore the workshops will happen live again. We therefore do not yet know if the workshops will be recorded. We will make the materials available later, though. ZuriHac is free to attend; you can still register. No separate registration for the workshops is required.

If you cannot make it to ZuriHac but are still interested in our courses or other services, check our Training page, Consultancy page, or just send us an email.

Note that the times have been changed.

Armando Santos: IOSim

Saturday, 11 June 2022, 1430–1700 CEST

IOSim is a simulator and drop-in replacement for the IO Monad developed by the networking team at IOHK. The io-sim and io-classes packages enable programmers to write code that can handle both real and simulated IO. Because it exposes a detailed execution trace, which can then be used in property tests, this tool has already proven to be very useful in the context of testing complex components of the ouroboros-network package.

In this workshop, Armando Santos will look at some examples of how to use io-sim and io-classes, as well as how to develop our own programs and use the tool to test and detect faults in the code.

This session should be useful for Haskell programmers of all levels of experience, however attendees should know about IO and ideally have some familiarity with basic concurrency primitives (async is sufficient) and STM.

Andres Löh: An Introduction to Type Systems and Operational Semantics

Sunday, 12 June 2022, 1330–1630 CEST

In this workshop, Andres Löh will introduce some simple type systems, starting from the Simply Typed Lambda Calculus and moving towards System Fω, which forms the basis of GHC’s internal Core language.

We will learn how to read and understand type rules, what desirable properties type systems (should) have, and how to implement a type checker for such systems.

The amount of Haskell knowledge you need in order to participate in this workshop is actually rather small.

Although this session is part of the so-called “Advanced Track,” it is actually rather introductory and will not teach you anything new if you are already familiar with reading papers about type systems. But if you are primarily a user of Haskell and have never really looked at the theoretical foundations, then this session may be just what you have been looking for in order to demystify some of the more theoretically focused papers and blog posts in the Haskell world.

by andres, armando, adam at June 01, 2022 12:00 AM

May 31, 2022

Gabriella Gonzalez

Generate web forms from pure functions

Generate web forms from pure functions

This is an announcement post for my “Grace browser” project, which you can find here:

This project is a web page which can dynamically convert a wide variety of functional programming expressions to their equivalent HTML. This conversion can even auto-generate interactive web forms from functions, which means that people without web programming knowledge can use the Grace browser to create and share simple and interactive web pages.

Demo

You can quickly get the gist of this project by visiting the following page:

The Grace browser begins with a code editor where you can input a purely functional expression and the above link uses the following example code:

\input ->
{ "x or y" : input.x || input.y
, "x and y": input.x && input.y
}

The interpreter then infers the type of the input expression, which for the above example is:

{ "x": Bool, "y": Bool } -> { "x or y": Bool, "x and y": Bool }

… and you can read that type as saying:

This is a function whose input is a record and whose output is also a record. The input record has two boolean-valued fields named “x” and “y” and the output record has two boolean-valued fields named “x or y” and “x and y”.

Here is the novel bit: the interpreter then generates a web interface appropriate for that type. In this case, the equivalent web interface is a form with two inputs:

  • a checkbox labeled “x”
  • a checkbox labeled “y”

… and the form also produces two outputs:

  • a read-only checkbox labeled “x or y”
  • a read-only checkbox labeled “x and y”

The result looks like this:

Screen Shot 2022-05-28 at 11 12 23 AM

Moreover, the generated form is reactive: as you check or uncheck the two input checkboxes the two output checkboxes immediately update in response. In particular:

  • the “x or y” box will be checked whenever either input box is checked
  • the “x and y” box will be checked whenever both input boxes are checked

Screen Shot 2022-05-28 at 11 19 15 AM

This also all runs entirely client side, meaning that all the computation happens in your browser. Specifically:

  • compiling the functional code to an interactive form is done client-side
  • updating form outputs in response to inputs is also done client-side

Intelligent forms

Here’s another example that might further pique your interest:

The above example is a pure function to increment each element of a list:

List/map (\x -> x + 1)

The List/map function is unsaturated, meaning that we’re missing the final argument: the actual list to transform. So the interpreter infers the following type for the function:

List Natural -> List Natural

… and you can read that type as saying:

This is a function whose input is a list of natural numbers and whose output is also a list of natural numbers.

So what kind of interactive form will that generate?

The generated input to the form begins with a blue “+” button that you can click to add elements to the input list. Each time you click the button the form creates a numeric input for that list element alongside a red “-” button that you can click to delete the corresponding list element. As you add or remove elements from the input list the reactive form update will also add or remove elements from the output list, too.

Screen Shot 2022-05-28 at 11 14 45 AM

Moreover, each element in the input list will be a numeric input. As you adjust each input element the matching output element will automatically be set to a number that is one greater than the input number.

The interpreter also sets the permitted range of the numeric inputs based on the inferred type. Since the default numeric type is Natural (i.e. non-negative) numbers, the numeric inputs will forbid negative inputs. However, if you were to add a type annotation to specify an Integer element type:

List/map (\x -> x + 1 : Integer)

… then the generated form will change to permit negative inputs and outputs because then the inferred type would be:

List Integer -> List Integer

Shareable forms

The “Grace browser” is based on the Fall-from-Grace functional programming language (or “Grace” for short), which I previously announced here:

… and one of the features of this language is the ability to import arbitrary expressions by URL, including functions!

That means that if you were to create a useful pure function for others to use you could host your code anywhere that you can serve plain text (such as a GitHub repository or gist) and anybody could turn that function into the corresponding form.

For example, the following gist contains a pure Grace function to compute US federal income taxes for 2022:

… so if you paste the URL for the raw text of the gist into the Grace browser you’ll get a shareable form for computing your taxes:

Screen Shot 2022-05-28 at 11 15 45 AM

This provides a lightweight way to publish, share, and consume utility code.

Plain JSON data

The Grace browser also works for plain data, too. In fact, Grace is a superset of JSON so the Grace browser is also a JSON browser.

For example, I can render the JSON served at https://api.github.com as HTML by pasting that URL into the Grace browser, which produces this result:

Screen Shot 2022-05-28 at 11 16 39 AM

If I’m only interested in one field of the JSON output, I can project out the field of interest like this:

( https://api.github.com/ ).starred_gists_url

Screen Shot 2022-05-28 at 11 17 08 AM

In other words, you can use Grace browser sort of like a “jq but with HTML output”.

Teaching tool

You can also use the Grace browser to teach functional programming concepts, too. For example, you can illustrate the difference between Haskell/Rust-style error-handling and Go-style error-handling by entering this input into the code editor:

{ "Error handling in Haskell and Rust":
\input -> input : < Failure : Text | Success : Natural >
, "Error handling in Go":
\input -> input : { "Failure" : Optional Text, "Success": Optional Natural }
}

… and the form will illustrate how:

  • In Haskell and Rust, the failure data and success data are mutually exclusive

    The generated form illustrates this by using mutually exclusive radio buttons for the input.

  • In Go, failure data and success data are not mutually exclusive

    The generated form illustrates this by using checkboxes to independently enable or disable the failure and success data.

Screen Shot 2022-05-28 at 11 18 06 AM

Semantic web

Grace is currently very limited in its current incarnation, meaning that the language only provides a small set of built-in functions and operators. The reason why is because I originally created Grace to serve as a simple reference implementation of how to create a functional programming language and I intended people to fork the language to add any additional language features they needed.

However, if Grace were more featureful then you could imagine creating a “semantic web” of purely functional expressions that could reference each other by URL and be visualized using an intelligent client like the Grace browser. For example, you could have some pure data hosted at https://example.com/students.ffg:

[ { "Name": "John Doe"       , "Grade": 95 }
, { "Name": "Mary Jane" , "Grade": 98 }
, { "Name": "Alice Templeton", "Grade": 90 }
]

… and then you could create a “view” into that data that adds up all the grades by hosting another expression at https://example.com/total.ffg which could contain:

let sum = https://raw.githubusercontent.com/Gabriella439/grace/main/prelude/natural/sum.ffg

in sum (List/map (\student -> student."Grade") ./students.ffg)

… and whenever you would update the data hosted at students.ffg the view at total.ffg would automatically update, too. You could then generate a web page for either view of the data using something like the Grace browser.

Conclusion

If this interests you, the website contains a tutorial that you can try out that partially overlaps with the examples from this post:

Just click the “Try the tutorial” button to give it a whirl.

If you want a deeper dive into what the Grace language can do, then I recommend reading the original announcement post:

… or reading the README from the corresponding GitHub project:

Also, I set things up so that if you do fork the language you can easily generate your own “Grace browser” for your fork of the language that’s just a bunch of static assets you can host anywhere. No server-side computation necessary!

by Gabriella Gonzalez (noreply@blogger.com) at May 31, 2022 04:26 PM

May 28, 2022

Mark Jason Dominus

“Llaves” and other vanishing consonants

Lately I asked:

Where did the ‘c’ go in llave (“key”)? It's from Latin clavīs

Several readers wrote in with additional examples, and I spent a little while scouring Wiktionary for more. I don't claim hat this list is at all complete; I got bored partway through the Wiktionary search results.

Spanish English Latin antecedent
llagar to wound plāgāre
llama flame flamma
llamar to summon, to call clāmāre
llano flat, level plānus
llantén plaintain plantāgō
llave key clavis
llegar to arrive, to get, to be sufficient   plicāre
lleno full plēnus
llevar to take levāre
llorar to cry out, to weep plōrāre
llover to rain pluere

I had asked:

Is this the only Latin word that changed ‘cl’ → ‘ll’ as it turned into Spanish, or is there a whole family of them?

and the answer is no, not exactly. It appears that llave and llamar are the only two common examples. But there are many examples of the more general phenomenon that

(consonant) + ‘l’ → ‘ll’

including quite a few examples where the consonant is a ‘p’.

Spanish-related notes

  • Eric Roode directed me to this discussion of “Latin CL to Spanish LL” on the WordReference.com language forums. It also contains discussion of analogous transformations in Italian. For example, instead of plānusllano, Italian has → piano.

  • Alex Corcoles advises me that Fundéu often discusses this sort of issue on the Fundéu web site, and also responds to this sort of question on their Twitter account. Fundéu is the Foundation of Emerging Spanish, a collaboration with the Royal Spanish Academy that controls the official Spanish language standard.

  • Several readers pointed out that although llave is the key that opens your door, the word for musical keys and for encryption keys is still clave. There is also a musical instrument called the claves, and an associated technical term for the rhythmic role they play. Clavícula (‘clavicle’) has also kept its ‘c’.

  • The connection between plicāre and llegar is not at all clear to me. Plicāre means “to fold”; English cognates include ‘complicated’, ‘complex’, ‘duplicate’, ‘two-ply’, and, farther back, ‘plait’. What this has to do with llegar (‘to arrive’) I do not understand. Wiktionary has a long explanation that I did not find convincing.

  • The levārellevar example is a little weird. Wiktionary says "The shift of an initial 'l' to 'll' is not normal".

  • Llaves also appears to be the Spanish name for the curly brace characters { and }. (The square brackets are corchetes.)

Not related to Spanish

  • The llover example is a favorite of the Universe of Discourse, because Latin pluere is the source of the English word plover.

  • French parler (‘to talk’) and its English descendants ‘parley’ and ‘parlor’ are from Latin parabola.

  • Latin plōrāre (‘to cry out’) is obviously the source of English ‘implore’ and ‘deplore’. But less obviously, it is the source of ‘explore’. The original meaning of ‘explore’ was to walk around a hunting ground, yelling to flush out the hidden game.

  • English ‘autoclave’ is also derived from clavis, but I do not know why.

  • Wiktionary's advanced search has options to order results by “relevance” and last-edited date, but not alphabetically!

Thanks

  • Thanks to readers Michael Lugo, Matt Hellige, Leonardo Herrera, Leah Neukirchen, Eric Roode, Brent Yorgey, and Alex Corcoles for hints clues, and references.

[ Addendum: Andrew Rodland informs me that an autoclave is so-called because the steam pressure inside it forces the door lock closed, so that you can't scald yourself when you open it. ]

by Mark Dominus (mjd@plover.com) at May 28, 2022 03:21 PM

May 27, 2022

GHC Developer Blog

GHC 9.2.3 is now available

GHC 9.2.3 is now available

Zubin Duggal - 2022-05-27

The GHC developers are very happy to at announce the availability of GHC 9.2.3. Binary distributions, source distributions, and documentation are available at downloads.haskell.org.

This release includes many bug-fixes and other improvements to 9.2.2 including:

As some of the fixed issues do affect correctness users are encouraged to upgrade promptly.

We would like to thank Microsoft Azure, GitHub, IOG, the Zw3rk stake pool, Tweag I/O, Serokell, Equinix, SimSpace, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

Happy compiling,

  • Zubin

by ghc-devs at May 27, 2022 12:00 AM

Lysxia's blog

Formalizing finite sets

Combinatorics studies mathematical structures by counting. Counting may seem like a benign activity, but the same rigor necessary to prevent double- or under-counting mistakes arguably underpins all of mathematics.1

Combining my two favorite topics, I’ve always wanted to mechanize combinatorics in Coq.2 An immediate challenge is to formalize the idea of “set”.3 We have to be able to define the set of things we want to count. It turns out that there are at least two ways of encoding sets in type theory: sets as types, and sets as predicates. They are suitable for defining different classes of operations: sums (disjoint union) are a natural operation on types, while unions and intersections are naturally defined on predicates.

The interplay between these two notions of sets, and finiteness, will then let us prove the standard formula for the cardinality of unions, aka. the binary inclusion-exclusion formula:

#|X ∪ Y| = #|X| + #|Y| - #|X ∩ Y|
Imports and options
From Coq Require Import ssreflect ssrbool.

Set Implicit Arguments.

Sets as types

The obvious starting point is to view a type as the set of its inhabitants.

How do we count its inhabitants? We will say that a set A has cardinality n if there is a bijection between A and the set {0 .. n-1} of natural numbers between 0 and n-1.

Bijections

A bijection is a standard way to represent a one-to-one correspondence between two sets, with a pair of inverse functions. We define the type bijection A B as a record containing the two functions and a proof of their inverse relationship.

Record is_bijection {A B} (to : A -> B) (from : B -> A) : Prop :=
  { from_to : forall a, from (to a) = a
  ; to_from : forall b, to (from b) = b }.

Record bijection (A B : Type) : Type :=
  { bij_to : A -> B
  ; bij_from : B -> A
  ; bij_is_bijection :> is_bijection bij_to bij_from }.

Infix "<-->" := bijection (at level 90) : type_scope.

We say that A and B are isomorphic when there exists a bijection between A and B. Isomorphism is an equivalence relation: reflexive, symmetric, transitive.4

Definition bijection_refl {A} : A <--> A.
Admitted. (* Easy exercise *)

Definition bijection_sym {A B} : (A <--> B) -> (B <--> A).
Admitted. (* Easy exercise *)

Definition bijection_trans {A B C} : (A <--> B) -> (B <--> C) -> (A <--> C).
Admitted. (* Easy exercise *)

Infix ">>>" := bijection_trans (at level 40).

Finite sets

Our “bijective” definition of cardinality shall rely on a primitive, canonical family of finite types {0 .. n-1} that is taken for granted. We can define them as the following sigma type, using the familiar set comprehension notation, also known as ordinal in math-comp:

Definition fin (n : nat) : Type := { p | p < n }.

An inhabitant of fin n is a pair of a p : nat and a proof object of p < n. Such proofs objects are unique for a given p and n, so the first component uniquely determines the second component, and fin n does have exactly n inhabitants.5

Finiteness

We can now say that a type A has cardinality n if there is a bijection between A and fin n, i.e., there is an inhabitant of A <--> fin n. Note that this only defines finite cardinalities, which is fine for doing finite combinatorics. Infinity is really weird so let’s not think about it.

As a sanity check, you can verify the cardinalities of the usual suspects, bool, unit, and Empty_set.

Definition bijection_bool : bool <--> fin 2.
Admitted. (* Easy exercise *)

Definition bijection_unit : unit <--> fin 1.
Admitted. (* Easy exercise *)

Definition bijection_Empty_set : Empty_set <--> fin 0.
Admitted. (* Easy exercise *)

A type A is finite when it has some cardinality n : nat. When speaking informally, it’s common to view finiteness as a property, a thing that a set either is or is not. To prove finiteness is merely to exhibit the relevant data: a number to be the cardinality, and an associated bijection (which we call an enumeration of A, enum for short). Hence we formalize “finiteness” as the type of that data.

Record is_finite (A : Type) : Type :=
  { card : nat
  ; enum : A <--> fin card }.

Further bundling is_finite A proofs with their associated set A, we obtain a concept aptly named “finite type”.6 A finite type is a type A paired with a proof of is_finite A.

Record finite_type : Type :=
  { ft_type :> Type
  ; ft_is_finite :> is_finite ft_type }.

We leverage coercions (indicated by :>) to lighten the notation of expressions involving finite_type.

The first coercion ft_type lets us use a finite_type as a Type. So if E : finite_type, we can write the judgement that “e is an element of E” as e : E, which implicitly expands to the more cumbersome e : ft_type E.

Similarly, the second coercion ft_is_finite lets us access the evidence of finiteness without naming that field. In particular, we can write the cardinality of E : finite_type as card E, as if card were a proper field of E rather than the nested record it actually belongs to. This is a convenient mechanism for overloading, letting us reuse the name card(inality) even though records technically cannot have fields with the same name. With that, we define #|A| as sugar for card A:

Notation "'#|' A '|'" := (card A).
Some notation boilerplate
Declare Scope fintype_scope.
Delimit Scope fintype_scope with fintype.
Bind Scope fintype_scope with finite_type.

Uniqueness of cardinality

The phrase “cardinality of a set” suggests that cardinality is an inherent property of sets. But now we’ve defined “finite type” essentially as a tuple where the cardinality is just one component. What’s to prevent us from putting a different number there, for the same underlying type?

We can prove that this cannot happen. Cardinality is unique: any two finiteness proofs for the same type must yield the same cardinality.

(The proof is a little tedious and technical.)

Theorem card_unique {A} (F1 F2 : is_finite A) : card F1 = card F2.
Admitted. (* Intermediate exercise *)

A slightly more general result is that isomorphic types (i.e., related by a bijection) have the same cardinality. It can first be proved in terms of is_finite, from which a corollary in terms of finite_type follows.

Theorem card_bijection {A B} (FA : is_finite A) (FB : is_finite B)
  : (A <--> B) -> card FA = card FB.
Admitted. (* Like card_unique *)

Theorem card_bijection_finite_type {A B : finite_type}
  : (A <--> B) -> #|A| = #|B|.
Proof.
  apply card_bijection.
Qed.

The converse is also true and useful: two types with the same cardinality are isomorphic.

Theorem bijection_card {A B} (FA : is_finite A) (FB : is_finite B)
  : card FA = card FB -> (A <--> B).
Admitted. (* Easy exercise *)

Theorem bijection_card_finite_type {A B : finite_type}
  : #|A| = #|B| -> (A <--> B).
Proof.
  apply bijection_card.
Qed.

Operations on finite sets

Sum

The sum of sets is also known as the disjoint union.

Inductive sum (A B : Type) : Type :=
| inl : A -> A + B
| inr : B -> A + B
where "A + B" := (sum A B) : type_scope.

sum is a binary operation on types. We must work to make it an operation on finite types.

There is a bijection between fin n + fin m (sum of sets) and fin (n + m) (sum of nats).

Definition bijection_sum_fin {n m} : fin n + fin m <--> fin (n + m).
Admitted. (* Intermediate exercise *)

The sum is a bifunctor.

Definition bijection_sum {A A' B B'}
  : (A <--> B) -> (A' <--> B') -> (A + A' <--> B + B').
Admitted. (* Easy exercise *)

Combining those facts, we can prove that the sum of two finite sets is finite (finite_sum), and the cardinality of the sum is the sum of the cardinalities (card_sum).

Definition is_finite_sum {A B} (FA : is_finite A) (FB : is_finite B)
  : is_finite (A + B) :=
  {| card := #|FA| + #|FB|
  ;  enum := bijection_sum (enum FA) (enum FB) >>> bijection_sum_fin |}.

Definition finite_sum (A B : finite_type) : finite_type :=
  {| ft_type := A + B ; ft_is_finite := is_finite_sum A B |}.

Infix "+" := finite_sum : fintype_scope.
Theorem card_sum {A B : finite_type} : #|(A + B)%fintype| = #|A| + #|B|.
Proof.
  reflexivity.
Qed.

Product

The cartesian product has structure dual to the sum.

Inductive prod (A B : Type) : Type :=
| pair : A -> B -> A * B
where "A * B" := (prod A B) : type_scope.
  • There is a bijection fin n * fin m <--> fin (n * m).
  • The product is a bifunctor.
  • The product of finite sets is finite.
  • The cardinality of the product is the product of the cardinalities.
Coq code
Definition bijection_prod_fin {n m} : fin n * fin m <--> fin (n * m).
Admitted. (* Intermediate exercise *)

Definition bijection_prod {A A' B B'}
  : (A <--> B) -> (A' <--> B') -> (A * A' <--> B * B').
Admitted. (* Easy exercise *)

Definition is_finite_prod {A B} (FA : is_finite A) (FB : is_finite B)
  : is_finite (A * B) :=
  {| card := #|FA| * #|FB|
  ;  enum := bijection_prod (enum FA) (enum FB) >>> bijection_prod_fin |}.

Definition finite_prod (A B : finite_type) : finite_type :=
  {| ft_type := A * B ; ft_is_finite := is_finite_prod A B |}.

Infix "*" := finite_prod : fintype_scope.

Theorem card_prod {A B : finite_type} : #|(A * B)%fintype| = #|A| * #|B|.
Proof.
  reflexivity.
Qed.

Sets as predicates

Two other common operations on sets are union and intersection. However, those operations don’t fit in the view of sets as types. While set membership x ∈ X is a proposition, type inhabitation x : X is a judgement, which is a completely different thing,7 so we need a different approach.

The idea of set membership x ∈ X as a proposition presumes that x and X are entities that exist independently of each other. This suggests that there is some “universe” that elements x live in, and the sets X under consideration are subsets of that same universe. We represent the universe by a type A, and sets (i.e., “subsets of the universe”) by predicates on A.

Definition set_of (A : Type) := (A -> bool).

Hence, if x : A is an element of the universe, and X : set A is a set (subset of the universe), we will denote set membership x ∈ X simply as X x (x satisfies the predicate X).

Those predicates are boolean, i.e., decidable. This is necessary in several constructions and proofs here, notably to prove that the union or intersection of finite sets is finite. We rely on a coercion to implicitly convert booleans to Prop: is_true : bool >-> Prop, which is exported by ssreflect.

Union, intersection, complement

Those common set operations correspond to the usual logical connectives.

Section Operations.

Context {A : Type}.

Definition union' (X Y : set_of A) : set_of A := fun a => X a || Y a.
Definition intersection' (X Y : set_of A) : set_of A := fun a => X a && Y a.
Definition complement' (X : set_of A) : set_of A := fun a => negb (X a).

End Operations.

Define the familiar infix notation for union and intersection.

Declare Scope set_of_scope.
Delimit Scope set_of_scope with set_of.
Bind Scope set_of_scope with set_of.

Infix "∪" := union' (at level 40) : set_of_scope.
Infix "∩" := intersection' (at level 40) : set_of_scope.

Finiteness

Again, we will characterize finite sets using bijections to fin n. We first transform the set X into a type to_type X, so we can form the type of bijections to_type X <--> fin n. Like fin, we define to_type A as a sigma type. Thanks to the predicate X being boolean, there is at most one proof p : X a for each a, so the type { a : A | X a } has exactly one inhabitant for each inhabitant a : A satisfying X a.

Definition to_type {A : Type} (X : set_of A) : Type := { a : A | X a }.

Coercion to_type : set_of >-> Sortclass.

We obtain a notion of finite set by imitating the structure of finite_type. The set-as-predicate X is finite if the set-as-type to_type X is finite.

Record finite_set_of (A : Type) : Type :=
  { elem_of :> set_of A
  ; fso_is_finite :> is_finite (to_type elem_of)
  }.

Similarly, a finite_type_of can be coerced to a finite_type.

Definition to_finite_type {A} (X : finite_set_of A) : finite_type :=
  {| ft_type := elem_of X
  ;  ft_is_finite := X |}.

Coercion to_finite_type : finite_set_of >-> finite_type.

Finite unions and intersections

We then prove that the union and intersection of finite sets are finite. This is actually fairly challenging, since proving finiteness means to calculate the cardinality of the set and to construct the associated bijection. Unlike sum and product, there is no simple formula for the cardinality of union and intersection. One candidate may seem to be the binary inclusion-exclusion formula:

#|X ∪ Y| = #|X| + #|Y| - #|X ∩ Y|

But that only gives the cardinality of the union in terms of the intersection, or vice versa, and we don’t know either yet.

Rather than constructing the bijections directly, we decompose the proof. The intuition is that X ∪ Y and X ∩ Y can easily be “bounded” by known finite sets, namely X + Y and X respectively. By “bounded”, we mean that there is an injection from one set to the other.

The standard definition of injectivity is via an implication f x = f y -> x = y. However, a better definition for our purposes comes from a one-sided inverse property: a function f : A -> B is a section if there exists another function g : B -> A (called a retraction) such that g (f x) = x. Every section is an injection, but the converse requires the law of excluded middle.

Record is_section {A B} (to : A -> B) (from : B -> A) : Prop :=
  { s_from_to : forall a, from (to a) = a }.

Record section (A B : Type) : Type :=
  { s_from : A -> B
  ; s_to : B -> A
  ; s_is_section : is_section s_from s_to }.

The point is that, given a section to a finite set, section A (fin n), we can construct a bijection A <--> fin m for some m, that is smaller than n. We formalize this result with a proof-relevant sigma type.

Definition section_bijection (A : Type) (n : nat)
  : section A (fin n) -> { m & A <--> fin m }.
Admitted. (* Hard exercise *)

This construction is rather involved. It is much more general than when we were looking specifically at union and intersection, but at the same time it is easier to come up with as it abstracts away the distracting details of those operations.

Now there is a section from X ∪ Y to X + Y, and from X ∩ Y to X.

Definition section_union {A} (X Y : set_of A)
  : section (X ∪ Y)%set_of (X + Y).
Admitted. (* Easy exercise *)

Definition section_intersection {A} (X Y : set_of A)
  : section (X ∩ Y)%set_of X.
Admitted. (* Easy exercise *)

We can then rely on the finiteness of X and X + Y to extend those sections to fin n for some n via the following theorem:

Theorem section_extend (A B C : Type)
  : section A B -> (B <--> C) -> section A C.
Admitted. (* Easy exercise *)

Definition section_union' {A} (X Y : finite_set_of A)
  : section (X ∪ Y)%set_of (fin (#|X| + #|Y|)).
Proof.
  eapply section_extend.
  - apply section_union.
  - apply is_finite_sum.
Qed.

Definition section_intersection' {A} (X Y : finite_set_of A)
  : section (X ∩ Y)%set_of (fin #|X|).
Proof.
  eapply section_extend.
  - apply section_intersection.
  - apply enum.
Qed.

Finally, by section_bijection, we obtain finiteness proofs of union' and intersection', which let us define union and intersection properly as operations on finite sets.

Theorem is_finite_union {A} {X Y : set_of A}
    (FX : is_finite X) (FY : is_finite Y)
  : is_finite (X ∪ Y)%set_of.
Proof.
  refine {| enum := projT2 (section_bijection _) |}.
  eapply (section_extend (B := X + Y)%type).
  - apply section_union.
  - apply (is_finite_sum FX FY).
Qed.

Theorem is_finite_intersection {A} {X Y : set_of A}
    (FX : is_finite X) (FY : is_finite Y)
  : is_finite (X ∩ Y)%set_of.
Proof.
  refine {| enum := projT2 (section_bijection _) |}.
  eapply section_extend.
  - apply section_intersection.
  - apply (enum FX).
Qed.

Definition union {A} (X Y : finite_set_of A) : finite_set_of A :=
  {| fso_is_finite := is_finite_union X Y |}.

Definition intersection {A} (X Y : finite_set_of A) : finite_set_of A :=
  {| fso_is_finite := is_finite_intersection X Y |}.
Declare Scope fso_scope.
Delimit Scope fso_scope with fso.
Bind Scope fso_scope with finite_set_of.

Infix "∪" := union (at level 40) : fso_scope.
Infix "∩" := intersection (at level 40) : fso_scope.

Hereafter, and will denote finite unions and intersections.

#[local] Open Scope fso_scope.

Inclusion-exclusion

#|X ∪ Y| = #|X| + #|Y| - #|X ∩ Y|

To prove that formula, it’s probably not a good idea to look at how and compute their cardinalities. A better idea is to construct a bijection, which implies an equality of cardinalities by card_bijection.

To start, subtractions are bad, so we rewrite the goal:

#|X ∪ Y| + #|X ∩ Y| = #|X| + #|Y|

Now we look for a bijection (X ∪ Y) + (X ∩ Y) <--> X + Y. It gets a bit tricky because of the dependent types.

Definition inclusion_exclusion_bijection {A} (X Y : finite_set_of A)
  : (X ∪ Y)%set_of + (X ∩ Y)%set_of <--> X + Y.
Admitted. (* Hard exercise *)

Isomorphic sets have the same cardinality (by theorem card_bijection_finite_type). The resulting equation simplifies to the binary inclusion-exclusion identity, because #|A + B| equals #|A| + #|B| definitionally. So the proof consists simply of applying that theorem with the above bijection.

Theorem inclusion_exclusion {A} (X Y : finite_set_of A)
  : #|X ∪ Y| + #|X ∩ Y| = #|X| + #|Y|.
Proof.
  apply (@card_bijection_finite_type ((X ∪ Y) + (X ∩ Y)) (X + Y)).
  apply inclusion_exclusion_bijection.
Qed.

Conclusion

To formalize mathematics, it’s often useful to revisit our preconceptions about fundamental concepts. To carry out even basic combinatorics in type theory, it’s useful to distinguish two views of the naive notion of set.

For example, when we say “union”, we really mean one of two things depending on the context. Either the sets are obviously disjoint, so we really mean “sum”: this corresponds to viewing sets as types. Or we implicitly know that the two sets contain the same “type” of elements a priori, so the overlap is something we have to worry about explicitly: this corresponds to viewing sets as predicates on a given universe.


  1. Ironically, when making restaurant reservations, I still occasionally forget to count myself.↩︎

  2. The code from this post is part of this project I’ve started here. Also check out Brent Yorgey’s thesis: Combinatorial Species and Labelled Structures (2014).↩︎

  3. Speaking of sets, it’s important to distinguish naive set theory from axiomatic set theory. Naive set theory is arguably what most people think of when they hear “set”. It is a semi-formal system for organizing mathematics: there are sets, they have elements, and there are various operations to construct and analyze sets, but overall we don’t think too hard about what sets are (hence, “semi-formal”). When this blog post talks about sets, it is in the context of naive set theory. Axiomatic set theory is formal, with rules that are clear enough to be encoded in a computer. The name “axiomatic set theory” is a stroke of marketing genius, establishing it as the “standard” way of formalizing naive set theory, and thus, all of mathematics, as can be seen in most introductory courses on formal logic. Historically, Zermelo’s set theory was formulated at around the same time as Russell’s type theory, and type theory is at the root of currently very active areas of programming language theory and formal methods.↩︎

  4. Bijections actually form a groupoid (a “proof-relevant equivalence relation”).↩︎

  5. We could also have defined fin as the inductive type of bounded naturals, which is named Fin.t in the standard library. Anecdotal experience suggests that the sigma type is more beginner-friendly. But past a certain level of familiarity, I think they are completely interchangeable.

    Inductive fin' : nat -> Type :=
    | F0 : fin' 1
    | FS : forall n, fin' n -> fin' (S n).

    The definition of fin as a sigma type relies on details of the definition of the order relation _ < _. Other definitions may allow the proposition p < n to be inhabited by multiple proof objects, causing fin n to have “more” than n inhabitants unless they are collapsed by proof irrelevance.↩︎

  6. math-comp has a different but equivalent definition of fintype.↩︎

  7. … if you know what those words mean.↩︎

by Lysxia at May 27, 2022 12:00 AM

May 26, 2022

Mark Jason Dominus

Quick Spanish etymology question

Where did the ‘c’ go in llave (“key”)? It's from Latin clavīs, like in “clavicle”, “clavichord”, “clavier” and “clef”.

Is this the only Latin word that changed ‘cl’ → ‘ll’ as it turned into Spanish, or is there a whole family of them?

[ Addendum 20220528: There are more examples. ]

by Mark Dominus (mjd@plover.com) at May 26, 2022 12:43 PM

Tweag I/O

Reproducible probabilistic programming environments

The development of user-friendly and powerful probabilistic programming libraries (PPLs) has been making Bayesian data analysis easily accessible: PPLs allow a statistician to define a statistical model for their data programatically in either domain-specific or common popular programming languages and provide powerful inference algorithms to sample posterior- and other probability distributions. Under the hood, many PPLs make use of big libraries such as TensorFlow, JAX or Theano / Aesara (a fork of Theano) that provide fast, vectorized array and matrix computations and, for gradient-based inference algorithms, automatic differentiation. In practice, if a user wants to use a PPL, they have to make sure these dependencies (and their dependencies!) are installed, too, which often can be difficult and/or irreproducible: want to compile your compute graph to C code? Then you need a C compiler. But what if you can’t just install the required C compiler version, because there’s already a different, incompatible version on your system? Or you want to run your work code on your private laptop, whose Python version is too low. And so on and so on…

We recently packaged and fixed a couple of PPLs and their related dependencies for use with the Nix package manager. In this blog post, we will showcase Nix and how it gives you easy access to a fully reproducible and isolated development environment in which PPLs and all their dependencies are pre-installed.

A really brief introduction to Nix

Nix assumes a functional approach to package management: building a software component is regarded as a pure, deterministic function that has as inputs the component’s source code, a list of dependencies, possibly a compiler, build instructions, etc. — in short, everything you need to build a software component. This concept is implemented very strictly in Nix. For example, a Python application packaged in Nix has not only its immediate Python dependencies exactly declared (with specific versions and all their dependencies), but also the Python version it is supposed to work with and any system dependencies (think BLAS, OpenCV, C compiler, glibc, …).

Nix and its ecosystem is an extremely large and active open source project, as indicated by the Nix package collection containing build instructions for over 80,000 packages. All these packages can be made available to developers in completely reproducible shells that are guaranteed to provide exactly the same software environment on your laptop, your AWS virtual machine or your continous integration (CI) runner. A limitation of Nix is that it runs only on Linux-like systems and that it requires sudo privileges for installation. You can also use Nix in Windows Subsystem for Linux (WSL) and on MacOS, but support for the latter is not as good as for standard Linux systems. After you installed Nix, you can get a shell in which, for example, Python 3.9, numpy, and OpenCV are available simply by typing:

$ nix-shell -p python39Packages.numpy open-cv

You can now check where this software is stored:

$ python -c "import numpy; print(numpy.__file__)"
/nix/store/bhs02rwyhgsdwriw9f1amkx9020zpir5-python3.9-numpy-1.21.2/lib/python3.9/site-packages/numpy/__init__.py
$ which opencv_version
/nix/store/hyni6hs71dphfy2s5yk8w1h3gzh90a44-opencv-4.5.4/bin/opencv_version

You see that all software provided by Nix is stored in a single directory (the “Nix store”) and is identified by a unique hash. This feature allows you to have several versions of the same software installed side-by-side, without the risk of collision and without any modification to the state of your system: want Python 3.8 and Python 3.9 available in the same shell? Sure, just punch in a nix-shell -p python38 python39!

Example: reproducible development Nix shell with PyMC3

Now that we hopefully made you curious about Nix, let’s finally mix Nix and probabilistic programming. As mentioned in the introduction, we made a couple of probabilistic programming-related libraries available in the Nix package collection. More specifically, we fixed and updated the packaging of PyMC3 and TensorFlow Probability and newly packaged the Theano fork Aesara, which is a crucial dependency for the upcoming PyMC 4 release.

To get a Nix shell that makes, for example, PyMC3 available, you could just run nix-shell -p python39Packages.pymc3, but if you want to add more packages and later recreate this environment, you want to have some permanent definition of your Nix shell. To achieve this, you can write a small file shell.nix in the functional Nix language declaring these dependencies. The shell.nix file essentially describes a function that takes a certain Nix package collection as an input and returns a Nix development shell. An example could look as follows:

let
  # use a specific (although arbitrarily chosen) version of the Nix package collection
  default_pkgs = fetchTarball {
    url = "http://github.com/NixOS/nixpkgs/archive/nixpkgs-unstable.tar.gz";
    # the sha256 makes sure that the downloaded archive really is what it was when this
    # file was written
    sha256 = "0x5j9q1vi00c6kavnjlrwl3yy1xs60c34pkygm49dld2sgws7n0a";
  };
# function header: we take one argument "pkgs" with a default defined above
in { pkgs ? import default_pkgs { } }:
with pkgs;
let
  # we create a Python bundle containing Python 3.9 and a few packages
  pythonBundle =
    python39.withPackages (ps: with ps; [ pymc3 matplotlib numpy ipython ]);
# this is what the function returns: the result of a mkShell call with a buildInputs
# argument that specifies all software to be made available in the shell
in mkShell { buildInputs = [ pythonBundle ]; }

You can then enter the Nix shell defined by this file by simply typing nix-shell in the file’s directory. When doing that for the first time, Nix will download all necessary dependencies from a cache server and perhaps rebuild some dependencies from scratch, which might take a minute. But once all dependencies are built or downloaded, they will be cached in the /nix/store/ directory and available instantaneously for later nix-shell calls.

Now let’s see what we can do with our Nix shell! We first check where Python and PyMC3 are from and then run a small Python script that does a Bayesian linear regression on made-up sample data:

$ nix-shell
[... lots of output about downloading dependencies]
[nix-shell:~/some/dir:]$ which python
/nix/store/rhc1yh5dvhll2db9n8qywpg6ysdv6yif-python3-3.9.10-env/bin/python
# *Not* your system Python, but a Python from your Nix store
[nix-shell:~/some/dir:]$ python -c "import pymc3; print(pymc3.__file___)"
/nix/store/rhc1yh5dvhll2db9n8qywpg6ysdv6yif-python3-3.9.10-env/lib/python3.9/site-packages/pymc3/__init__.py
# PyMC3 is in your Nix store, too. The state of your system Python installation is unchanged
[nix-shell:~/some/dir:]$ python pymc3_linear_regression.py
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [noise, intercept, slope]
Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 5 seconds.

png

[nix-shell:~/some/dir:]$ exit
# now you're back to your normal shell...
$ which python
/usr/bin/python
# and back to your system Python

As you can see, you entered an isolated development shell that provides the dependencies you specified and that allows you to run PyMC3 without changing the state of your system Python installation. And if you now run the same sequence of commands on a different machine with Nix installed, it will just work just the same way as above! Just put the shell.nix file in the same VCS repository as your code and voilà - you’re sharing not only your code, but also the software environment it was developed in.

This is not specific to PyMC3 at all: a reproducible and isolated software environment containing TensorFlow Probability or Aesara can be defined and used similarly; just replace pymc3 by tensorflow-probability or aesara in your Python bundle.

On the face of it, that all might seem not so different from a Python virtual environment. But we saw the crucial difference above: Python virtual environments manage only Python dependencies, but no dependencies further down the “dependency tree”. Nix, on the other hand, behaves thus rather a bit like Conda or, although it’s quite a stretch, like a Docker image: it provides system dependencies, too. A detailed comparison of these alternatives to Nix is beyond the scope of this post, though.

Conclusion

In this blog post, we gave a short introduction to the Nix package manager and its development shell feature and demonstrated how to use it to obtain a reproducible software environment that contains PyMC3 and a few other Python dependencies. We also showed that these software environments don’t mess with your system state and thus allow you to fearlessly experiment and try out new software without breaking anything. In this regard, Nix provides an alternative to Docker or Conda, but it can do much, much more — in fact, there is even a whole Linux distribution (NixOS) that is based on the Nix package manager!

You can find the shell.nix file and the PyMC3 example script in Tweag’s blog post resources repository. If you would like to learn more about Nix, visit the Nix website for more resources, browse the Nix Discourse or pop in to #nix:nixos.org on Matrix or #nixos on the Libera.Chat IRC network.

May 26, 2022 12:00 AM

May 24, 2022

GHC Developer Blog

GHC 9.4.1-alpha2 released

GHC 9.4.1-alpha2 released

bgamari - 2022-05-24

The GHC developers are happy to announce the availability of the second alpha release of the GHC 9.4 series. Binary distributions, source distributions, and documentation are available at downloads.haskell.org.

This major release will include:

  • A new profiling mode, -fprof-late, which adds automatic cost-center annotations to all top-level functions after Core optimisation has run. This incurs significantly less performance cost while still providing informative profiles.

  • A variety of plugin improvements including the introduction of a new plugin type, defaulting plugins, and the ability for typechecking plugins to rewrite type-families.

  • An improved constructed product result analysis, allowing unboxing of nested structures, and a new boxity analysis, leading to less reboxing.

  • Introduction of a tag-check elision optimisation, bringing significant performance improvements in strict programs.

  • Generalisation of a variety of primitive types to be levity polymorphic. Consequently, the ArrayArray# type can at long last be retired, replaced by standard Array#.

  • Introduction of the \cases syntax from GHC proposal 0302

  • A complete overhaul of GHC’s Windows support. This includes a migration to a fully Clang-based C toolchain, a deep refactoring of the linker, and many fixes in WinIO.

  • Support for multiple home packages, significantly improving support in IDEs and other tools for multi-package projects.

  • A refactoring of GHC’s error message infrastructure, allowing GHC to provide diagnostic information to downstream consumers as structured data, greatly easing IDE support.

  • Significant compile-time improvements to runtime and memory consumption.

  • On overhaul of our packaging infrastructure, allowing full traceability of release artifacts and more reliable binary distributions.

  • … and much more. See the release notes for a full accounting.

Note that, as 9.4.1 is the first release for which the released artifacts will all be generated by our Hadrian build system, it’s possible that there will be packaging issues. If you enounter trouble while using a binary distribution, please open a ticket. Likewise, if you are a downstream packager, do consider migrating to Hadrian to run your build; the Hadrian build system can be built using cabal-install, stack, or the in-tree bootstrap script.

We would like to thank Microsoft Azure, GitHub, IOG, the Zw3rk stake pool, Tweag I/O, Serokell, Equinix, SimSpace, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

Happy testing,

  • Ben

by ghc-devs at May 24, 2022 12:00 AM

May 23, 2022

GHC Developer Blog

Mid 2022 Release Plans

Mid 2022 Release Plans

Matthew Pickering - 2022-05-23

This post sets out our current plans for the upcoming releases in the next few months.

9.4.1

The next stage of the 9.4.1 release series (alpha2) will be released within the next few days. The main changes in alpha2 relative to alpha1 are improvements to the packaging and release process which have fixed a number of packaging bugs present in alpha1 due to moving to bindists built by hadrian.

The final 9.4.1 release is scheduled for late July.

The release manager for this release is Ben Gamari.

9.2.3

The 9.2.3 release is scheduled to immediately follow the 9.4.1-alpha1. This release fixes some packaging issues on Windows and a few bugs, notably:

  • a panic involving unbound cycle-breaker variables that prevented several libraries from compiling, such as massiv-io (#20231),
  • a typechecker regression in which GHC refused to use recursive equalities involving type families (#21473, #21515).

The release manager for this release is Zubin Duggal.

9.0.* series

We do not intend to produce any more releases in the 9.0.* series. From our perspective there are no serious bugs affecting the 9.0.2 release. It is advised that users start using the 9.2 series, which we intend to stabilise in the same manner as the 8.10 series. We have made this decision for the following reasons:

  • The 9.2 series does not contain significant breakage (when upgrading from 9.0)
  • Anecdotal evidence suggests that many companies are upgrading straight from 8.10.7 to 9.2.2 and skipping the 9.0.* releases.
  • We do not currently have capacity to manage 4 active branches.

Conclusion

This post summarises the latest state of the release cycles and our intent within the next few months. If you have any questions or comments then please post the next few months. If you have any questions or comments then please be in touch via mailto:ghc-devs@haskell.org.

by ghc-devs at May 23, 2022 12:00 AM

May 18, 2022

Gabriella Gonzalez

Namespaced De Bruijn indices

Namespaced De Bruijn indices

In this post I share a trick I use for dealing with bound variables in Dhall that I thought might be useful for other interpreted programming languages. I have no idea if this trick has been introduced before but if it has then just let me know and I’ll acknowledge any prior art here.

Edit: Todd Wilson points out that Mark-Oliver Stehr’s CINNI originally introduced this idea.

The brief explanation of the trick is: instead of choosing between a named or a nameless representation for bound variables you can get the best of both worlds by namespacing De Bruijn indices by variable names. This simplifies the implementation and in some cases improves the end user’s experience.

The rest of this post is a longer explanation of the above summary, starting with an explanation of the trick and followed by a review of the benefits of this approach.

Background

I’d like to first explain what I mean by “named” and “nameless” representations before I explain the trick.

A named representation of the lambda calculus syntax tree typically looks something like this:

data Syntax
= Variable String
| Lambda String Syntax
| Apply Syntax Syntax

For example, if the user wrote the following Haskell-like code:

\f -> \x -> f x

… then that would correspond to this syntax tree:

example :: Syntax
example = Lambda "f" (Lambda "x" (Apply (Variable "f") (Variable "x")))

The named representation has the nice property that it preserves the original variable names … well, sort of. This representation definitely preserves the variable names when you initially parse the code into the syntax tree, but if you β-reduce an expression you can potentially run into problems.

For example, consider this expression:

\x -> (\y -> \x -> y) x

… which corresponds to this syntax tree:

Lambda "x" (Apply (Lambda "y" (Lambda "x" (Variable "y"))) (Variable "x"))

If you try to β-reduce (\y -> \x -> y) x without renaming any variables then you get the following incorrect result:

\x -> \x -> x

This bug is known as “name capture” and capture-avoiding substitution requires renaming one of the variables named x so that the inner x does not shadow the outer x. For example, we could fix the problem by renaming the outer x to x1 like this:

\x1 -> \x -> x1

A nameless representation tries to work around these name capture issues by replacing the variable names with numeric indices (known as De Bruijn indices):

data Syntax
= Variable Int
| Lambda Syntax
| Apply Syntax Syntax

For example, code like this:

\f -> \x -> f x

… corresponds to this nameless representation:

example :: Syntax
example = Lambda (Lambda (Apply (Variable 1) (Variable 0)))

Carefully note that the Lambda constructor now has no field for the bound variable name, so it’s as if the user had instead written:

\ -> \ -> @1 @0

… using @n to represent the variable whose De Bruijn index is n.

The numeric De Bruijn indices refer to bound variables. Specifically, the numeric index 0 refers to the “closest” or “innermost” variable bound by a lambda:

--                This 0 index …
-- ↓
\ -> \ -> @1 @0
-- ↑ … refers to the variable bound by this lambda

… and incrementing the index moves to the next outermost lambda:

--             This 1 index …
-- ↓
\ -> \ -> @1 @0
-- ↑ … refers to the variable bound by this lambda

De Bruijn indices avoid name collisions between bound variables, but they require you to do additional work if you wish to preserve the original variable names. There are several ways to do so, and I’ll present my preferred approach.

The trick - Part 1

We can get the best of both worlds by combining the named and nameless representations into a hybrid representation like this:

data Syntax
= Variable String Int
| Lambda String Syntax
| Apply Syntax Syntax

I call this representation “namespaced De Bruijn indices”.

This is almost the exact same as our named representation, except that we have now added an Int field to the Variable constructor. This Int field is morally the same as the De Bruijn index in the nameless representation, except that this time the De Bruijn index is “namespaced” to a specific variable name.

The easiest way to explain this is with a few examples.

The following expression:

\x -> \y -> \x -> x@0

… corresponds to this syntax tree:

Lambda "x" (Lambda "y" (Lambda "x" (Variable "x" 0)))

… and this curried function returns the third function argument:

--                    This …
-- ↓
\x -> \y -> \x -> x@0
-- ↑ … refers to this bound variable

… because that is the innermost bound variable named x.

Similarly, the following expression:

\x -> \y -> \x -> y@0

… corresponds to this syntax tree:

Lambda "x" (Lambda "y" (Lambda "x" (Variable "y" 0)))

… which returns the second function argument:

--                    This …
-- ↓
\x -> \y -> \x -> y@0
-- ↑ … refers to this bound variable

… because that is the innermost bound variable named y.

Carefully note that our variable still has a De Bruijn index of 0, but we ignore the innermost bound variable named x because we also pair our De Bruijn index with name of the variable we are referring to (y) so we only count bound variables named y when resolving the De Bruijn index.

Finally, the following expression:

\x -> \y -> \x -> x@1

… corresponds to this syntax tree:

Lambda "x" (Lambda "y" (Lambda "x" (Variable "x" 1)))

… which returns the first function argument:

--                    This …
-- ↓
\x -> \y -> \x -> x@1
-- ↑ … refers to this bound variable

The De Bruijn index is 1, which means that it refers to the second innermost (0-indexed) bound variable named x.

Notice how this representation lets us refer to shadowed variables by their index. These De Bruijn indices are not an internal implementation detail, but are actually available to the user as part of the surface syntax of the language.

However, we want to avoid littering the code with these De Bruijn indices, which brings us to the second part of the trick.

The trick - Part 2

The next step is to add syntactic sugar to the language by allowing users to omit the index in the source code, which defaults the index to 0. This means that an expression that never references shadowed variables never needs to specify a De Bruijn index.

For example, instead of writing this:

\x -> \y -> \x -> x@0

… we can elide the index to simplify the code to:

\x -> \y -> \x -> x

… which will still parse as:

Lambda "x" (Lambda "y" (Lambda "x" (Variable "x" 0)))

Similarly, we can simplify this:

\x -> \y -> \x -> y@0

… to this:

\x -> \y -> \x -> y

… which will still parse as:

Lambda "x" (Lambda "y" (Lambda "x" (Variable "y" 0)))

However, we cannot use this syntactic sugar to simplify the final example:

\x -> \y -> \x -> x@1

… since the index is non-zero. Any code that references a shadowed variable still needs to use an explicit De Bruijn index to do so.

Vice versa, we also omit zero indices when pretty-printing code. When we pretty-print this syntax tree:

Lambda "x" (Lambda "y" (Lambda "x" (Variable "x" 0)))

… we don’t include the index:

\x -> \y -> \x -> x

This syntactic sugar ensures that most users do not need to be aware that indices exist at all when writing code. The user only encounters the indices in two scenarios:

  • The user wishes to explicitly reference a shadowed variable

    For example, in the following expression:

    \x -> \y -> \x -> x@1

    … the user might prefer to use the built-in language support for disambiguating variables of the same name rather than renaming one of the two variables named x.

  • The indices appear in a β-reduced result

    For example, this expression has no user-visible De Bruijn indices:

    \x -> (\y -> \x -> y) x

    … but if you β-reduce the expression (I’ll cover how in the Appendix) and pretty-print the β-reduced expression then the result will introduce a non-zero De Bruijn index to disambiguate the two variables named x:

    \x -> \x -> x@1

In fact, the latter scenario is the reason I originally adopted this trick: I wanted to be able to display β-reduced functions to the end user while preserving the original variable names as much as possible.

Note that De Bruijn indices don’t appear when a β-reduced expression does not reference any shadowed variables. For example, if you β-reduce this expression:

(\f -> f f) (\x -> x)

… the result has no De Bruijn index (because the index is 0 and is therefore elided by the pretty-printer):

\x -> x

The trick - Part 3

One of the benefits of the traditional nameless representation using (non-namespaced) De Bruijn indices is that you get α-equivalence for free. Two nameless expressions are α-equivalent if they are syntactically identical. We can build upon this useful property to derive a compact algorithm for α-equivalence of “namespaced De Bruijn indices”.

The trick is to recognize that namespaced De Bruijn indices reduce to ordinary De Bruijn indices in the degenerate case when you rename all variables to the same name. I’ll call this renaming process “α-reduction”.

For example, if we α-reduce the following expression by renaming all of the: variables to _:

\x -> \y -> \x -> x@1

… then we get this result:

\_ -> \_ -> \_ -> _@2

See the Appendix for the α-reduction algorithm.

Equipped with α-reduction, then we can derive α-equivalence: two expressions are α-equivalent if their α-reduced forms are syntactically identical.

For example, this expression:

\x -> x

… and this expression:

\y -> y

… both α-reduce to:

\_ -> _

… so they are α-equivalent.

Benefits

There are a few benefits of using this trick that motivate me to use this in all of my interpreted languages:

  • This trick improves the readability of β-reduced functions

    β-reduced functions preserve the original variable names and this trick doesn’t suffer from the rename-related name pollution that plagues other capture-avoiding substitution algorithms. In particular, β-reduced expressions only display De Bruijn indices when absolutely necessary (if they reference a shadowed variable) and they otherwise use the original pristine variable names.

  • This trick simplifies the internal implementation

    You don’t need to maintain two separate syntax trees for a named and nameless representation. You can use the same syntax tree for both since any named syntax tree can be α-reduced to give the equivalent nameless syntax tree.

  • This trick enables userland support for referencing shadowed variables

    I know some people think that referencing shadowed variable names is a misfeature. However, I personally feel that resolving name collisions by adding ' or _ characters to the end of variable names is less principled than having language support for resolving name collisions using optional De Bruijn indices.

  • (Not shown) This trick can sometimes improve type errors

    To be precise, this trick improves the inferred types displayed in error messages when using explicit universal quantification.

    Type variables also have to avoid name collisions, so if you use the same namespaced De Bruijn representation for your types then you avoid polluting your inferred types and error messages with junk type variables like a14.

    This post doesn’t cover the equivalent type-level trick, but you can refer to the Dhall standard if you need an example of a language that uses this trick.

Conclusion

I believe that namespaced De Bruijn indices are most appropriate for languages that are (A) strongly normalizing (like Dhall) and (B) interpreted, because such languages tend to support pretty-printing β-reduced functions.

I think this trick is also useful to a lesser extent for all interpreted languages, if only because the implementation is (in my opinion) simpler and more elegant than other algorithms for capture-avoiding substitution (See the Appendix below).

On the other hand, compiled languages will likely not benefit much from this trick since they typically have no need to preserve the original variable names and they also will use an intermediate representation that is very different from the concrete syntax tree.

Appendix - Implementation

This section provides Haskell code specifying how to α-reduce and β-reduce a syntax tree that uses namespaced De Bruijn indices.

This reference implementation is not the most efficient implementation, but it’s the simplest one which I use for pedagogical purposes. If you’re interested in efficiency then check out my Grace project, which mixes this trick with the more efficient normalization-by-evaluation algorithm.

I also don’t include code for the parser or pretty-printer, because the only interesting part is the syntactic sugar for handling variables with a De Bruijn index of 0. Again, check out Grace if you want to refer to a more complete implementation.

-- | Syntax tree
data Syntax
= Variable String Int
| Lambda String Syntax
| Apply Syntax Syntax
deriving (Eq, Show)

{-| Increase the index of all bound variables matching the given variable name

This is modified from the Shifting definition in Pierce's \"Types and
Programming Languages\" by adding an additional argument for the namespace
to shift
-}
shift
:: Int
-- ^ The amount to shift by
-> String
-- ^ The variable name to match (a.k.a. the namespace)
-> Int
-- ^ The minimum bound for which indices to shift
-> Syntax
-- ^ The expression to shift
-> Syntax
shift offset namespace minIndex syntax =
case syntax of
Variable name index -> Variable name index'
where
index'
| name == namespace && minIndex <= index = index + offset
| otherwise = index

Lambda name body -> Lambda name body'
where
minIndex'
| name == namespace = minIndex + 1
| otherwise = minIndex

body' = shift offset namespace minIndex' body

Apply function argument -> Apply function' argument'
where
function' = shift offset namespace minIndex function

argument' = shift offset namespace minIndex argument

{-| Substitute the given variable name and index with an expression

This is modified from the Substitution definition in Pierce's \"Types and
Programming Languages\" by adding an additional argument for the variable
index
-}
substitute
:: Syntax
-- ^ The expression to substitute into
-> String
-- ^ The name of the variable to replace
-> Int
-- ^ The index of the variable to replace
-> Syntax
-- ^ The expression to substitute in place of the given variable
-> Syntax
substitute expression name index replacement =
case expression of
Variable name' index'
| name == name' && index == index' -> replacement
| otherwise -> Variable name' index'

Lambda name' body -> Lambda name' body'
where
index'
| name == name' = index + 1
| otherwise = index

shiftedBody = shift 1 name' 0 replacement

body' = substitute body name index' shiftedBody

Apply function argument -> Apply function' argument'
where
function' = substitute function name index replacement

argument' = substitute argument name index replacement

-- | β-reduce an expression
betaReduce :: Syntax -> Syntax
betaReduce syntax =
case syntax of
Variable name index -> Variable name index

Lambda name body -> Lambda name body'
where
body' = betaReduce body

Apply function argument ->
case function' of
Lambda name body -> body'
where
shiftedArgument = shift 1 name 0 argument

substitutedBody = substitute body name 0 shiftedArgument

unshiftedBody = shift (-1) name 0 substitutedBody

body' = betaReduce unshiftedBody

_ -> Apply function' argument'
where
function' = betaReduce function

argument' = betaReduce argument

-- | α-reduce an expression
alphaReduce :: Syntax -> Syntax
alphaReduce syntax =
case syntax of
Variable name index -> Variable name index

Lambda name body -> Lambda "_" body'
where
shiftedBody = shift 1 "_" 0 body

substitutedBody = substitute shiftedBody name 0 (Variable "_" 0)

unshiftedBody = shift (-1) name 0 substitutedBody

body' = alphaReduce unshiftedBody

Apply function argument -> Apply function' argument'
where
function' = alphaReduce function

argument' = alphaReduce argument

-- | Returns `True` if the two input expressions are α-equivalent
alphaEquivalent :: Syntax -> Syntax -> Bool
alphaEquivalent left right = alphaReduce left == alphaReduce right

Appendix - History

I actually first introduced this feature in Morte, not Dhall. The idea originated from the discussion on this issue.

by Gabriella Gonzalez (noreply@blogger.com) at May 18, 2022 01:56 PM

May 12, 2022

Tweag I/O

Comparing strict and lazy

This blog post covers essentially the same material as the talk I gave at Haskell Exchange 2020 (time truly flies). If you prefer watching it in a talk format, you can watch the recording. Or you can browse the slides.

I first conceived of writing (well, talking, initially) on this subject after one too many person told me “lazy is better than strict because it composes”. You see, this is a sentence that simply doesn’t make much sense to me, but it is oft repeated.

Before we get started, let me make quite specific what we are going to discuss: we are comparing programming languages, and whether by default their function calls are lazy or strict. Strict languages can (and do) have lazy data structures and lazy languages can have strict data structures (though it’s a little bit harder, in GHC, for instance, full support has only recently been released).

In the 15 years that I’ve been programming professionally, the languages in which I’ve written the most have been Haskell and Ocaml. These two languages are similar enough, but Haskell is lazy and Ocaml is strict. I’ll preface my comparison by saying that, in my experience, when switching between Ocaml and Haskell, I almost never think about laziness or strictness. It comes up sometimes. But it’s far from being a central consideration. I’m pointing this out to highlight that lazy versus strict is really not that important; it’s not something that’s worth the very strong opinions that you can see sometimes.

Locks

With these caveats established, I’d like to put to rest the statement that laziness composes better. Consider the following piece of Haskell

atomicPut :: Handle -> String -> IO ()
atomicPut h line =
  withMVar lock $ \_ -> do
    hPutStrLn h line

This looks innocuous enough: it uses a lock to ensure that the line argument is printed without being interleaved with another call to atomicPut. It also has a severe bug. Don’t beat yourself up if you don’t see why: it’s pretty subtle; and this bit of code existed in a well-used logging library for years (until it broke production on a project I was working on and I pushed a fix). The problem, here, is that line is lazy, hence can contain an arbitrary amount of computation, which is subsequently run by hPutStrLn. Running arbitrary amounts of computation within a locked section is very bad.

The fix, by the way, is to fully evaluate the line before entering the lock

atomicPut :: Handle -> String -> IO ()
atomicPut h line = do
  evaluate $ force line
  withMVar lock $ \_ -> do
    hPutStrLn h line

It goes to show, though, that laziness doesn’t compose with locks. You have to be quite careful too: for each variable used within the locked section, you need to evaluate it at least as much as the locked code will before entering the lock.

Shortcutting fold

When people claim that lazy languages compose better, what they think about is something like this definition

or :: [Bool] -> Bool
or =  foldr (||) False

This is truly very clever: because this implementation will stop traversing the list as soon as it finds a True element. To see why, let’s look at the definition of foldr

foldr            :: (a -> b -> b) -> b -> [a] -> b
foldr _ z []     =  z
foldr f z (x:xs) =  f x (foldr f z xs)

When we call foldr recursively, we do that as an argument to f, but since f is lazy, the recursive call is not evaluated until f itself asks for the evaluation. In or, f is (||), which doesn’t evaluate its second argument when the first is True, so the recursive call never happens in this case.

It’s absolutely possible to do the same thing in a strict language. But it requires quite a bit more setup:

(* val fold_right_lazily : ('a -> 'b Lazy.t -> 'b Lazy.t) -> 'a List.t -> 'b Lazy.t -> 'b Lazy.t *)
let rec fold_right_lazily f l accu =
  match l with
  | [] -> accu
  | a::l' -> f a (lazy (Lazy.force (fold_right_lazily f l' accu)))

(* val or_ : bool List.t -> bool *)
let or_ l = Lazy.force (fold_right_lazily (fun x y -> x || (Lazy.force y)) l (Lazy.from_val false))

But, honestly, it’s not really worth it. GHC takes a lot of care to optimise lazy evaluation, since it’s so central in its evaluation model. But Ocaml doesn’t give so much attention to lazy values. So fold_right_lazily wouldn’t be too efficient. In practice, Ocaml programmers will rather define or manually

(* val or_ : bool List.t -> bool *)
let rec or_ = function
  | [] -> false
  | b::l -> b || (or_ l) (* || is special syntax which is lazy in the
                           second argument*)

Applicative functors

Another, probably lesser known, way in which laziness shines is applicative functors. At its core, an applicative functor, is a data structure which supports zipWith<N> (or map<N> in Ocaml) for all N. For instance, for lists:

zipWith0 :: a -> [a]
zipWith1 :: (a -> b) -> [a] -> [b]
zipWith2 :: (a -> b -> c) -> [a] -> [b] -> [c]
zipWith3 :: (a -> b -> c -> d) -> [a] -> [b] -> [c] -> [d]
zipWith4 :: (a -> b -> c -> d -> e) -> [a] -> [b] -> [c] -> [d] -> [e]
zipWith5 :: (a -> b -> c -> d -> e -> f) -> [a] -> [b] -> [c] -> [d] -> [e] -> [f]

Of course, that’s infinitely many functions, and we can’t define them all. Though probably, it’s enough to define 32 of them, but even that would be incredibly tedious. The applicative functor abstraction very cleverly finds a way to summarise all these functions as just three functions:

pure :: a -> [a]
(<$>) :: (a -> b) -> [a] -> [b]
(<*>) :: [a -> b] -> [a] -> [b]

(in Haskell, this would be the Applicative instance of the ZipList type, but let’s not be distracted by that)

Then, zipWith5 is derived simply as:

zipWith5 :: (a -> b -> c -> d -> e -> f) -> [a] -> [b] -> [c] -> [d] -> [e] -> [f]
zipWith5 f as bs cs ds es = f <$> as <*> bs <*> cs <*> ds <*> es

The definition is so simple that you never need to define zipWith5 at all: you just use the definition inline. But there’s a catch: were you to use this definition on a strict data structure, the performance would be abysmal. Indeed, this zipWith5 would allocate 5 lists: each call to (<*>) allocates an intermediate list. But a manual implementation of zipWith5 requires a single list to be allocated. This is very wasteful.

For lazy list it’s alright: you do allocate all 5 lists as well, but in a very different pattern. You first allocate the first cons cell of each of the 5 lists, and discard all 4 intermediate results. Then you allocate the second cons cell of each list, etc… This means that at any point in time, the memory overhead is constant. This is the sort of load that a garbage collector handles very very well.

Now, this is really about lazy data structures versus strict data structures, which I said at the beginning that I wouldn’t discuss. But I think that there is something here: in a strict language, you will usually be handling strict data structures. If you want them to support an efficient applicative interface, you will need a lazy copy of the same data structure. This is a non-trivial amount of extra code. I imagine it could be mitigated by the language letting you derive this lazy variant of your data structure. But I don’t think any language does this yet.

Matching lazy data is weird

That being said, lazy languages come with their lot of mind-boggling behaviour. Pattern-matching can be surprisingly counter-intuitive.

Consider the following

f :: Bool -> Bool -> Int
f _    False = 1
f True False = 2
f _    _     = 3

This second clause may seem completely redundant: after all anything matched by the second clause is already matched by the first. However it isn’t: it forces the evaluation of the first argument. So the mere presence of this clause changes the behaviour of f (it makes it that f undefined False = undefined). But f can never return 2.

This and more examples can be found in Simon Peyton Jones’s keynote talk at Haskell Exchange 2019 Revisiting Pattern Match Overlap Checks as well as in an older paper GADTs meet their match, by Karachalias, Schrijvers, Vytiniotis, and Peyton Jones.

Memory

Consider the following implementation of list length:

length :: [a] -> Integer
length [] = 0
length (_:as) = 1 + length as

It’s not a good implementation: it will take <semantics>O(n)<annotation encoding="application/x-tex">O(n)</annotation></semantics>O(n) stack space, which for big lists, will cause a stack overflow, even though length really only require <semantics>O(1)<annotation encoding="application/x-tex">O(1)</annotation></semantics>O(1) space. It’s worth noting that we are consuming stack space, here, because (+) is strict. Otherwise, we could have been in a case like the or function from earlier, where the recursive call was guarded by the lazy argument and didn’t use space.

For such a strict recursion, the solution is well-known: change the length function to be tail recursive with the help of an accumulator:

length :: [a] -> Integer
length = go 0
  where
    go :: Integer -> [a] -> Integer
    go acc [] = acc
    go acc (_:as) = go (acc+1) as

This transformation is well-understood, straightforward, and also incorrect. Because while it is true that we are no longer using stack space, we have traded it for just as much heap space. Why? Well, while (+) is strict, Integer is still a lazy type: a value of type Integer is a thunk. During our recursion we never evaluate the accumulator; so what we are really doing is creating a sort of copy of the list in the form of thunks which want to evaluate acc + 1.

This is a common trap of laziness (see also this blog post by Neil Mitchell). It’s much the same as why you typically want to use foldl' instead of foldl in Haskell. The solution is to evaluate intermediate results before making recursive calls, for instance with bang patterns1:

length :: [a] -> Integer
length = go 0
  where
    go :: Integer -> [a] -> Integer
    go !acc [] = acc
    go !acc (_:as) = go (acc+1) as

This is an instance of a bigger problem: it’s often very difficult to reason about memory in lazy languages. The question “do these few lines of code leak memory” sometimes provokes very heated discussions among seasoned Haskell programmers.

On a personal note I once fixed a pretty bad memory leak. The fix was pushed in production. When I came back to work the next day, the memory leak was still there. What had happened is that there were actually two memory leaks, one was caused by laziness: my reproducer forced the guilty thunks, so masked that second leak. I only fixed one. I got burnt by the fact that the memory usage of applications with a lot of laziness changes when you observe them.

Lazy deserialisation

The flip side, though, is that if you don’t need a whole data structure, you won’t allocate the unnecessary parts without having to think about it. Where this shines the brightest, in my opinion, is in lazy deserialisation.

The scenario is you get (from disk, from the network,…) a datastructure in the form of a serialised byte-string. And you convert it to a linked data structure for manipulation within your application. If the data structure is lazy, you can arrange it so that forcing part of the data structure performs the relevant deserialisation (for instance Json can be deserialised lazily).

This is very nice because linked data structures weigh strongly on the garbage collector which needs to traverse them again and again, while byte-strings are essentially free.

In this scenario you can, guilt-free, convert your byte-string to a value of your data structure. And deserialisation will happen implicitly on demand.

This can be done in a strict language with a lazy data structure, but this requires more care and more boilerplate, so it can get in the way sometimes.

Conclusion

Comparing laziness and strictness, it’s difficult to find a clear winner. I could have given more points of comparisons (in fact, there are a few more in my talk), but the pattern continues. Of course, among the different advantages and disadvantages of laziness and strictness, there may be some which count more for you. This could make you take a side. But others will want to make a different trade-off.

From where I stand, laziness is not a defining feature of Haskell. In fact, Purescript, which is undeniably a Haskell dialect, is a strict language. Things like type classes, higher-order type quantification, and purity are much more relevant in distinguishing Haskell from the rest of the ML family of languages.

In my opinion, the main role that laziness has played for Haskell is making sure that it stays a pure language. At the time, we didn’t really know how to make pure languages, and a strict language would have likely eventually just added effects. You simply can’t do that with a lazy language, so Haskell was forced to be creative. But this is a thing of the past: we now know how to make pure languages (thanks to Haskell!), we don’t really need laziness anymore.

I honestly think that, nowadays, laziness is a hindrance. Whether you prefer laziness or strictness, I find it difficult to argue that the benefits of laziness are large enough to justify Haskell being the only lazy language out there. So much of the standard tooling assumes strictness (it’s a legitimate question to ask what it would mean to step through a Haskell program with gdb, but there is no doubt about what it means for Ocaml). So if you’re making a new language, I think that you should make it strict.


  1. As it happens, the faulty implementation of length can be found in base. It’s not exposed to the user but it’s used internally. It’s saved by the fact that it uses Int rather than Integer, and the strictness analysis automatically makes the go function strict. I honestly have no idea whether the author was conscious of the fact that they were leveraging the strictness analysis this way, or whether it’s another piece of evidence that it’s very easy to get wrong.

May 12, 2022 12:00 AM

May 11, 2022

Well-Typed.Com

Hasura and Well-Typed collaborate on Haskell tooling

editorial note: This is a cross-post of a post originally published on the Hasura blog.

Well-Typed and Hasura have been working together since 2020 to improve Haskell tooling for commercial Haskell users, taking advantage of Well-Typed’s expertise maintaining the Glasgow Haskell Compiler and Hasura’s experience using Haskell in production at scale. Over the last two years we have continued our productive relationship working on a wide variety of projects, in particular related to the profiling and debugging capabilities of the compiler, many of which have had a reinvention or spruce-up. In this post we’ll look at back at the progress we have made together.

Memory profiling and heap analysis

ghc-debug

One of the first big projects we worked on was ghc-debug, a new heap analysis tool that can gather detailed information about the heap of a running process or analyse a snapshot. This tool gives precise results so it can be used to reliably investigate memory usage issues, and we have used it numerous times to fix bugs in the GHC code base. Within Hasura we have used it to investigate fragmentation issues more closely and also to diagnose a critical memory leak regression before a release.

Since GHC 9.2, ghc-debug is supported natively in GHC. All the libraries and executables are on Hackage so it can be installed and used like any normal Haskell library.

Info table profiling

Also in GHC 9.2 we introduced “info table profiling” (or -hi profiling), a new heap profiling mode that analyses memory usage over time and relates it to source code locations. Crucially, it does not require introducing cost centres and recompiling with profiling enabled (which may distort performance). It works by storing a map from info tables to meta-information such as where it originated, what type it is and so on. The resulting profile can be viewed using eventlog2html to give a detailed table about the memory behaviour of each closure type over the course of a program.

We have used info table profiling extensively on GHC itself to find and resolve memory issues, meaning that GHC 9.2 and 9.4 bring significant reductions in compile-time memory usage.

Understanding memory fragmentation

Our early work with Hasura investigated why there was a large discrepency between the memory usage reported by the operating system and the Haskell runtime. The initial hypothesis was that, due to the extensive use of pinned bytestrings in Hasura’s code base, we were losing memory due to heap fragmentation.

We developed an understanding of how exactly fragmentation could occur on a Haskell heap, tooling to analyse the extent of fragmentation and ultimately some fixes to GHC’s memory allocation strategy to reduce fragmentation caused by short-lived bytestring allocations.

This investigation also led to a much deeper understanding of the memory retention behaviour of the GHC runtime and led to some additional improvements in how much memory the runtime will optimistically retain. For long-lived server applications the amount of memory used should return to a steady baseline after being idle for a long period.

This work also highlighted how other compilers trigger idle garbage collections. In particular, we may want to investigate triggering idle collections by allocation rate rather than simple idleness, as applications may continue to still do a small amount of work in their idle periods.

Runtime performance profiling and monitoring

Late cost centre profiling

Code centre profiling, the normal tool recommended for GHC users profiling their Haskell programs, allows recording both time/allocation and heap profiles. It requires compiling the project in profiling mode, which inserts cost centres to the compiled code. Traditionally, the issue with cost centre profiling has been that adding cost centres severly affects how your program is optimised. This means that the existing strategies for automatically inserting cost centres (such as -fprof-auto) can lead to major skew if they are inserted in an inopportune place.

We have implemented a new cost centre insertion mode in GHC 9.4, -fprof-late, which inserts cost centres after the optimiser has finished running. Therefore the cost centres will not affect how your code is optimised and the profile gives a more accurate view of how your unprofiled program would perform. The trade-off is that the names of the cost centres contain internal names, but they are nearly always easily understandable.

The utility of this mode can not be understated, you now get a very fine-grained profile that accurately reflects the actual runtime behaviour of your program. It’s made me start using the cost-centre profiler again!

We also developed a plugin which can be used to approximate this mode if you are using GHC 9.0 or 9.2.

Ticky-ticky profiling

Hasura have a suite of benchmarks that track different runtime metrics, such as bytes allocated.1 Investigating regressions in these benchmarks requires a profiling tool geared towards profiling allocations. GHC has long had support for ticky profiling, which gives a low level view about which functions are allocating. However, in the past ticky profiling has been used almost exclusively by GHC developers, not users, and profiles were only consumable in a rudimentary text-based format.

We added support to emit ticky samples via the eventlog in GHC 9.4, and support for rendering the information in the profile to an interactive HTML table using eventlog2html. In addition, we integrated the new info table mapping (as used by ghc-debug and -hi profiling) to give precise locations for each ticky counter, making it easier to interpret the profile.

Live profiling and monitoring via the eventlog

For a long time we have been interested in unifying GHC’s various profiling mechanisms via the eventlog, and making them easier to monitor. We developed a prototype live monitoring setup for Hasura, eventlog-live, that could attach to the eventlog and read events whilst the program was running. This prototype was subsequently extended thanks to funding from IOG.

Native Stack Pointer register

GHC-compiled programs use separate registers for the C stack and Haskell stack. One consequence of this is that native Linux debugging and statistical profiling tools (such as perf) see only the C stack pointer, and hence provide a very limited window into the behaviour of Haskell programs.

Hasura commissioned some experimental investigatory work to see whether it would be possible to use the native stack pointer register for the Haskell stack, and hence get more useful output from off-the-shelf debugging tools. Unfortunately we ran into issues getting perf to understand the debugging information generated by GHC, and there are challenges related to maintaining LLVM compatibility, but we remain interested in exploring this further.

Haskell Language Server

Lately we have started to support maintenance of the Haskell Language Server (HLS). The language server is now a key part of many developers’ workflows, so it is a priority to make sure it is kept up-to-date and works reliably, and sponsorship from companies like Hasura is crucial to enabling this.

Recently our work on HLS has included:

  • Supporting the GHC 9.2 release series, as Hasura were keen to upgrade and have access to all the improved functionality we discussed in this post.

  • Diagnosing and resolving difficult-to-reproduce segfaults experienced by HLS users. It turned out that the version compatability checks were not strict enough, and HLS could load incompatible object files when running Template Haskell. In particular, you must build haskell-language-server with exactly the same version of GHC with which you compiled your package dependencies, so that object files for dependencies have the correct ABI.

  • Starting to take advantage of the recently completed support for Multiple Home Units in GHC to make HLS work more robustly for projects consisting of multiple components.

Conclusion

Well-Typed are grateful to Hasura for funding this work, as it will benefit the whole Haskell community. With their help we have made significant progress in the last two years improving debugging and profiling capabilities of the compiler, and improving the developer experience using HLS. We look forward to continuing our productive collaboration in the future.

As well as experimenting with all these tools on Hasura’s code base, we have also been using them to great effect on GHC’s code base, in order to reduce memory usage and increase performance of the compiler itself (e.g. by profiling GHC compiling Hasura’s graphql-engine). The new profiling tools have been useful in finding places to optimise: ghc-debug and -hi profiling made eliminating memory leaks straightforward, the late cost centre patch gives a great overview of where GHC spends time, and ticky profiling gives a low level overview of the allocations. They have also been very helpful for our work on improving HLS performance.

Well-Typed are actively looking for funding to continue maintaining and enhancing GHC and HLS. If your company relies on robust Haskell tooling, and you could support this work, or would like help improving the developer experience for your Haskell engineers, please get in touch with us via info@well-typed.com!


  1. The number of bytes allocated acts as a proxy for the amount of computation performed, since Haskell programs tend to allocate frequently, and allocations are more consistent than CPU or wall clock time.↩︎

by matthew at May 11, 2022 12:00 AM

May 09, 2022

Magnus Therning

Comments and org-static-blog

I'm using org-static-blog to generate the contents of this site. So far I'm very happy with it, but I've gotten a few emails from readers who've wanted to comment on something I've written and they always point out that it's not easy to do. It's actually not a coincidence that it's a bit difficult!

Yesterday I came up with a way that might make is slightly easier without involving JavaScript from a 3rd party. By making use of the built-in support for adding HTML code for comments. One slight limitation is that it's a single variable holding the code, and I'd really like to allow for both

  • using a link to a discussion site, e.g. reddit, as well as
  • my email address

As the comment support in org-static-blog comes in the form of a single variable this seems a bit difficult to accomplish. However, it isn't difficult at all to do in elisp due to the power of advice-add.

By using the following advice on org-static-blog-publish-file

(advice-add 'org-static-blog-publish-file :around
            (lambda (orig-fn filename &rest args)
              (let*  ((comments-url (with-temp-buffer
                                      (insert-file-contents filename)
                                      (or (cadar (org-collect-keywords '("commentsurl")))
                                          my-blog-default-comments-url)))
                      (org-static-blog-post-comments (concat "Comment <a href=" comments-url ">here</a>.")))
                (apply orig-fn filename args))))

and defining my-blog-default-comments-url to a mailto:... URL I get a link to use for commenting by either

  1. set commentsurl to point to discussion about the post on reddit, or
  2. not set commentsurl at all and get the mailto:... URL.

If you look at my previous post you see the result of the former, and if you look below you see the result of the latter.

May 09, 2022 08:10 PM

May 08, 2022

Magnus Therning

A little Haskell: epoch timestamp

A need of getting the current UNIX time is something that comes up every now and then. Just this week I needed it in order to add a k8s liveness probe1.

While it's often rather straight forward to get the Unix time as an integer in other languages2, in Haskell there's a bit of type tetris involved.

  1. getPOSIXTime gives me a POSIXTime, which is an alias for NominalDiffTime.
  2. NominalDiffTime implements RealFrac and can thus be converted to anything implementing Integral (I wanted it as Int64).
  3. NominalDiffTime also implements Num, so if the timestamp needs better precision than seconds it's easy to do (I needed milliseconds).

The combination of the above is something like

truncate <$> getPOSIXTime

In my case the full function of writing the timestamp to a file looks like this

writeTimestampFile :: MonadIO m => Path Abs File -> m ()
writeTimestampFile afn = liftIO $ do
    truncate @_ @Int64 . (* 1000) <$> getPOSIXTime >>= writeFile (fromAbsFile afn) . show

Footnotes:

1

Over the last few days I've looked into k8s probes. Since we're using Istio TCP probes are of very limited use, and as the service in question doesn't offer an HTTP API I decided to use a liveness command that checks that the contents of a file is a sufficiently recent epoch timestamp.

2

Rust's Chrono package has Utc.timestamp(t). Python has time.time(). Golang has Time.Unix.

May 08, 2022 05:51 AM

May 04, 2022

Neil Mitchell

Working on build systems full-time at Meta

Summary: I joined Meta 2.5 years ago to work on build systems. I’m enjoying it.

I joined Meta over two years ago when an opportunity arose to work on build systems full time. I started the Shake build system at Standard Chartered over 10 years ago, and then wrote an open source version a few years later. Since then, I’ve always been dabbling in build systems, at both moderate and small scale. I really enjoyed writing the Build Systems a la Carte paper, and as a result, started to appreciate some of the Bazel and Buck design decisions. I was involved in the Bazel work at Digital Asset, and after that decided that there was still lots of work to be done on build systems. I did some work on Cloud Shake, but the fact that I wasn’t working on it every day, and that I wasn’t personally using it, made it hard to productionize. A former colleague now at Meta reached out and invited me for breakfast — one thing led to another, and I ended up at Meta working on build systems full time.

What I’ve learnt about build systems

The biggest challenge at Meta is the scale. When I joined they already used the Buck build system, which had been developed at Meta. Looking at the first milliseconds after a user starts an incremental build is illustrative:

  • With Shake, it starts the process, loads the database into memory, walks the entire graph calling stat on each input and runs any build actions.
  • With Buck, it connects to a running daemon, talks to a running file watcher (Watchman in the case of Buck) and uses reverse dependencies to jump to the running actions.

For Shake, on repos with 100K files, that process might take ~0.5s, but it is O(n). If you increase to 10M files, it takes 50s, and your users will revolt. With Buck, the overhead is proportional to the number of changed files, which is usually a handful.

While Shake is clearly infeasible at the scale of Meta, Buck was also starting to show its age, and I’ve been working with others to significantly improve Buck, borrowing lessons from everywhere, including Shake. Buck also addresses problems that Shake doesn’t, such as how to cope with multi-configuration builds (e.g. building for x86 and ARM simultaneously), having a separate file and target namespace and effective use of remote execution and caching.

We expect that the new version of Buck will be released open source soon, at which point I’ll definitely be talking more about the design and engineering trade-offs behind it.

What's different moving from finance to tech

My career to date has been in finance, so working at Meta is a very different world. Below are a few things that stand out (I believe most of these are common to other big tech companies too, but Meta is my first one).

Engineering career ladder: In finance the promotion path for a good engineer is to become a manager of engineers, then a manager of managers, and so on up. In my previous few roles I was indeed managing teams, which included setting technical direction and doing coding. At Meta, managers look after people, and help set the team direction. Engineers look after code and services, and set the technical direction. But importantly, you can be promoted as an engineer, without gaining direct reports, and the opportunities and compensation are equivalent to that for managers. There are definitely aspects of management that I like (e.g. mentoring, career growth, starting collaborations), and happily all of these are things engineers can still engage in.

Programmer centric culture: In finance the company is often built around traders and sales people. In tech, the company is built around programmers, which is visible in the culture. There are hardware vending machines, free food, free ice cream, minimal approvals. They’ve done a very good job of providing a relaxing and welcoming environment (with open plan offices, but I don’t mind that aspect). The one complaint I had was that Meta used to have a pretty poor work from home policy, but that’s now been completely rewritten and is now very good.

Reduced hierarchy: I think this may be more true of Meta than other tech, but there is very minimal hierarchy. Programmers are all just programmers, not “senior” or “junior”. I don’t have the power to tell anyone what to do, but in a slightly odd way, my manager doesn’t have that power either. If I want someone to tackle a bug, I have to justify that it is a worthwhile thing to do. One consequence of that is that the ability to form relationships and influence people is much more important. Another consequence that I didn’t foresee is that working with people in different teams is very similar to working with people in your team, since exactly the same skills apply. I can message any engineer at Meta, about random ideas and possible collaborations, and everyone is happy to talk.

Migration is harder: In previous places I worked, if we needed 100 files moved to a new version of a library, someone got told to do it, and they went away and spent a lot of time doing it. At Meta that’s a lot harder — firstly, it’s probably 100K files due to the larger scale, and secondly, telling someone they must do something is a lot less effective. That means there is a greater focus on automation (automatically editing the files), compatibility (doesn’t require editing the files) and benefits (ensuring that moving to the new version of the library will make your life better). All those are definitely better ways to tackle the problem, but sometimes, work must be done that is tedious and time consuming, and that is harder to make happen.

Open source: The process for open sourcing an internal library or tool in the developer infrastructure space is very smooth. The team I work in has open sourced the Starlark programming language (taking over maintenance from Google), the Gazebo Rust utility library and a Rust linter, plus we have a few more projects in the pipeline. As I write code in the internal Meta monorepo, it gets sync’d to GitHub a few minutes later. It’s also easy to contribute to open source projects, e.g. Meta engineers have contributed to my projects such as Hoogle (before I even considered joining Meta).

Hiring: Meta hires a lot of engineers (e.g. 1,000 additional people in London). That means that interviews are more like a production line, with a desire to have a repeatable process, where candidates are assigned teams after the interviews, rather than interviewing with a team. There are upsides and downsides to that—if I interview a strong candidate, it’s really sad to know that I probably won’t get to work closely with them. It also means that the interview process is determined centrally, so I can’t follow my preferences. But it does mean that if a friend is looking for a job there’s often something available for them (you can find details on compilers and programming here and a full list of jobs here), and a repeatable process is good for fairness.

Overall I’m certainly very happy to be working on build systems. The build system is the thing that stands between a user and trying out their changes, so anything I can do to make that process better benefits all developers. I’m very excited to share what I’ve been working on more widely in the near future!

(Disclosure: This blog post had to go through Meta internal review, because it’s an employee talking about Meta, but other than typos, came out unchanged.)

by Neil Mitchell (noreply@blogger.com) at May 04, 2022 03:16 PM

May 03, 2022

Matt Parsons

Moving the Programming Blog

I’m moving the programming stuff over to https://overcoming.software.

May 03, 2022 12:00 AM

Donnacha Oisín Kidney

Depth Comonads

Posted on May 3, 2022
Tags: Agda

I haven’t written much on this blog recently: since starting a PhD all of my writing output has gone towards paper drafts and similar things. Recently, though, I’ve been thinking about streams, monoids, and comonads and I haven’t manage to wrangle those thoughts into something coherent enough for a paper. This blog post is a collection of those (pretty disorganised) thoughts. The hope is that writing them down will force me to clarify things, but consider this a warning that the rest of this post may well be muddled and confusing.

Streams

The first thing I want to talk about is streams.

record Stream (A : Type) : Type where
  coinductive
  field head : A
        tail : Stream A

This representation is coinductive: the type above contains infinite values. Agda, unlike Haskell, treats inductive and coinductive types differently (this is why we need the coinductive keyword in the definition). One of the differences is that it doesn’t check termination for construction of these values:

alternating :: [Bool]
alternating = True : False : alternating

We have the equivalent in Haskell on the right. We’re also using some fancy syntax for the Agda code: copatterns (Abel and Pientka 2013).

Note that this type is only definable in a language with some notion of laziness. If we tried to define a value like alternating above in OCaml we would loop. Haskell has no problem, and Agda—through its coinduction mechanism—can handle it as well.

Update 4-5-22: thanks to Arnaud Spiwack (@aspiwack) for correcting me on this, it turns out the definition of alternating above can be written in Ocaml, even without laziness. Apparently Ocaml has a facility for strict cyclic data structures. Also, I should be a little more precise with what I’m saying above: even without the extra facility for strict cycles, you can of course write a lazy list with some kind of lazy wrapper type.

There is, however, an isomorphic type that can be defined without coinduction:

(notice that, in this form, the function ℕ-alternating is the same function as even : ℕ → Bool)

In fact, we can convert from the coinductive representation to the inductive one. This conversion function is more familiarly recognisable as the indexing function:

_[_] : Stream A → ℕ-Stream A
xs [ zero  ] = xs .head
xs [ suc n ] = xs .tail [ n ]

I’m not just handwaving when I say the two representations are isomorphic: we can prove this isomorphism, and, in Cubical Agda, we can use this to transport programs on one representation to the other.

Proof of isomorphism

tabulate : ℕ-Stream A → Stream A
tabulate xs .head = xs zero
tabulate xs .tail = tabulate (xs ∘ suc)

stream-rinv : (xs : Stream A) → tabulate (xs [_]) ≡ xs
stream-rinv xs i .head = xs .head
stream-rinv xs i .tail = stream-rinv (xs .tail) i

stream-linv : (xs : ℕ-Stream A) (n : ℕ) → tabulate xs [ n ] ≡ xs n
stream-linv xs zero    = refl
stream-linv xs (suc n) = stream-linv (xs ∘ suc) n

stream-reps : ℕ-Stream A ⇔ Stream A
stream-reps .fun = tabulate
stream-reps .inv = _[_]
stream-reps .rightInv = stream-rinv
stream-reps .leftInv xs = funExt (stream-linv xs)

One final observation about streams: another way to define a stream is as the cofree comonad of the identity functor.

record Cofree (F : Type → Type) (A : Type) : Type where
  coinductive
  field root : A
        step : F (Cofree F A)

�-Stream : Type → Type
�-Stream = Cofree id

Concretely, the Cofree F A type is a possibly infinite tree, with branches shaped like F, and internal nodes labelled with A. It has the following characteristic function:

{-# NON_TERMINATING #-}
trace : ⦃ _ : Functor � ⦄ → (A → B) → (A → � A) → A → Cofree � B
trace ϕ � x .root = ϕ x
trace ϕ � x .step = map (trace ϕ �) (� x)

Like how the free monad turns any functor into a monad, the cofree comonad turns any functor into a comonad. Comonads are less popular and widely-used than monads, as there are less well-known examples of them. I have found it helpful to think about comonads through spatial analogies. A lot of comonads can represent a kind of walk through some space: the extract operation tells you “what is immediately here�, and the duplicate operation tells you “what can I see from each point�. For the stream, these two operations are inhabited by head and the following:

duplicate : Stream A → Stream (Stream A)
duplicate xs .head = xs
duplicate xs .tail = duplicate (xs .tail)

Generalising Streams

There were three key observations in the last section:

  1. Streams are coinductive. This requires a different termination checker in Agda, and a different evaluation model in strict languages.
  2. They have an isomorphic representation based on indexing. This isomorphic representation doesn’t need coinduction or laziness.
  3. They are a special case of the cofree comonad.

Going forward, we’re going to look at generalisations of streams, and we’re going to see what these observations mean in the contexts of the new generalisations.

The thing we’ll be generalising is the index of the stream. Currently, streams are basically structures that assign a value to every ℕ: what does a stream of—for instance—rational numbers look like? To drive the intuition for this generalisation let’s first look at the comonad instance on the ℕ-Stream type:

ℕ-extract : ℕ-Stream A → A
â„•-extract xs = xs zero

ℕ-duplicate : ℕ-Stream A → ℕ-Stream (ℕ-Stream A)
â„•-duplicate xs zero    = xs
ℕ-duplicate xs (suc n) = ℕ-duplicate (xs ∘ suc) n

This is the same instance as is on the Stream type, transported along the isomorphism between the two types (we could have transported the instance automatically, using subst or transport; I have written it out here manually in full for illustration purposes).

The â„•-duplicate method here can changed a little to reveal something interesting:

ℕ-duplicate₂ : ℕ-Stream A → ℕ-Stream (ℕ-Stream A)
â„•-duplicateâ‚‚ xs zero    m = xs m
ℕ-duplicate₂ xs (suc n) m = ℕ-duplicate₂ (xs ∘ suc) n m

ℕ-duplicate₃ : ℕ-Stream A → ℕ-Stream (ℕ-Stream A)
ℕ-duplicate₃ xs n m = xs (go n m)
  where
  go : ℕ → ℕ → ℕ
  go zero    m = m
  go (suc n) m = suc (go n m)

ℕ-duplicate₄ : ℕ-Stream A → ℕ-Stream (ℕ-Stream A)
â„•-duplicateâ‚„ xs n m = xs (n + m)

In other words, duplicate basically adds indices.

There is something distinctly monoidal about what’s going on here: taking the (ℕ, +, 0) monoid as focus, the extract method above corresponds to the monoidal empty element, and the duplicate method corresponds to the binary operator on monoids. In actual fact, there is a comonad for any function from a monoid, often called the Traced comonad.

Traced : Type → Type → Type
Traced E A = E → A

extractᵀ : ⦃ _ : Monoid E ⦄ → Traced E A → A
extractᵀ xs = xs ε

duplicateᵀ : ⦃ _ : Monoid E ⦄ → Traced E A → Traced E (Traced E A)
duplicateᵀ xs e� e₂ = xs (e� ∙ e₂)

Reifying Traced

The second observation we made about streams was that they had an isomorphic representation which didn’t need coinduction. What we can see above, with Traced, is a representation that also doesn’t need coinduction. So what is the corresponding coinductive representation? What does a generalised reified stream look like?

So the first approach to reifying a function to a data structure is to simply represent the function as a list of pairs.

C-Traced : Type → Type → Type
C-Traced E A = Stream (E × A)

This representation obviously isn’t ideal: it isn’t possible to construct an isomorphism between C-Traced and Traced. We can—kind of—go in one direction, but even that function isn’t terminating:

{-# NON_TERMINATING #-}
lookup-env : ⦃ _ : IsDiscrete E ⦄ → C-Traced E A → Traced E A
lookup-env xs x = if does (x ≟ xs .head .fst)
                     then xs .head .snd
                     else lookup-env (xs .tail) x

I’m not too concerned with being fast and loose with termination and isomorphisms for the time being, though. At the moment, I’m just interested in exploring the relationship between streams and the indexing functions.

As a result, let’s try and push on this representation a little and see if it’s possible to get something interesting and almost isomorphic.

Segmented Streams

To get a slightly nicer representation we can exploit the monoid a little bit. We can do this by storing offsets instead of the absolute indices for each entry. The data structure I have in mind here looks a little like this:

�����������┳������┳������┉
┃x         ┃y     ┃z     ┉
┡����������╇������╇������┉
╵⇤a╌╌╌╌╌╌╌⇥╵⇤b╌╌╌⇥╵⇤c╌╌╌╌┈

Above is a stream containing the values x, y, and z. Instead of each value corresponding to a single entry in the stream, however, they each correspond to a segment. The value x, for instance, labels the first segment in the stream, which has a length given by a. y labels the second segment, with length b, z with length c, and so on.

The Traced version of the above structure might be something like this:

str :: Traced m a
str i | i < a         = x
      | i < a + b     = y
      | i < a + b + c = z
      | ...

So the index-value mapping is also segmented. The stream, in this way, is kind of like a ruler, where different values mark out different quantities along the ruler, and the index function takes in a quantity and tells you which entry in the ruler that quantity corresponds to.

In code, we might represent the above data structure with the following type:

record Segments (E : Type) (A : Type) : Type where
  field
    length : E
    label  : A
    next   : Segments E A

open Segments

The question is, then, how do we convert this structure to an Traced representation?

Monuses

We need some extra operations on the monoid in the segments in order to enable this conversion to the Traced representation. The extra operations are encapsulated by the monus algebra: I wrote about this in the paper I submitted with Nicolas Wu to ICFP last year (2021). It’s a simple algebra on monoids which basically encapsulates monoids which are ordered in a sensible way.

The basic idea is that we construct an order on monoids which says “x is smaller than y if there is some z that we can add to x to get to y�.

_≼_ : ⦃ _ : Monoid A ⦄ → A → A → Type _
x ≼ y = ∃ z × (y ≡ x ∙ z)

A monus is a monoid where we can extract that z, when it exists. On the monoid (â„•, +, 0), for instance, this order corresponds to the normal ordering on â„•.

Extracting the z above corresponds to a kind of difference operator:

_∸_ : ℕ → ℕ → ℕ
x     ∸ zero  = x
suc x ∸ suc y = x ∸ y
_     ∸ _     = zero

This operator is sometimes called the monus. It is a kind of partial, or truncating, subtraction:

_ : 5 ∸ 2 ≡ 3
_ = refl

_ : 2 ∸ 5 ≡ 0
_ = refl

And, indeed, this operator “extracts� the z, when it exists.

∸‿is-monus : ∀ x y → (x≼y : x ≼ y) → y ∸ x ≡ fst x≼y
∸‿is-monus zero    _       (z , y≡0+z) = y≡0+z
∸‿is-monus (suc x) (suc y) (z , y≡x+z) = ∸‿is-monus x y (z , suc-inj y≡x+z)
∸‿is-monus (suc x) zero    (z , 0≡x+z) = ⊥-elim (zero≢suc 0≡x+z)

Our definition of a monus is simple: a monus is anything where the order ≼, sometimes called the “algebraic preorder�, is total and antisymmetric. This is precisely what lets us write a function which takes the Segments type and converts it back to the Traced type.

{-# NON_TERMINATING #-}
Segments→Traced : ⦃ _ : Monus E ⦄ → Segments E A → Traced E A
Segments→Traced xs i with xs .length ≤? i
... | yes (j , i≡xsₗ∙j) = Segments→Traced (xs .next) j
... | no  _             = xs .label

This function takes an index, and checks if that length is greater than or equal to the first segment in the stream of segments. If it is, then it continues searching through the rest of the segments with the index reduced by the size of that first segment. If not, then it returns the label of the first segment.

Taking the old example, we are basically converting to ∸ from +:

str :: Traced m a
str i | i         < a = x
      | i ∸ a     < b = y
      | i ∸ a ∸ b < c = z
      | ...

The first issue here is that this definition is not terminating. That might seem an insurmountable problem at first—we are searching through an infinite stream, after all—but notice that there is one paremeter which is decreasing on each recursive call: the index. Well, it only decreases if the segment is non-zero: this can be enforced by changing the definition of the segments type:

record ℱ-Segments (E : Type) ⦃ _ : Monus E ⦄ (A : Type) : Type where
  coinductive
  field
    label    : A
    length   : E
    length≢ε : length ≢ ε
    next     : ℱ-Segments E A

open ℱ-Segments

This type allows us to write the following definition:

module _ ⦃ _ : Monus E ⦄ (wf : WellFounded _≺_) where
  wf-index : ℱ-Segments E A → (i : E) → Acc _≺_ i → A
  wf-index xs i a with xs .length ≤? i
  ... | no _ = xs .label
  wf-index xs i (acc wf) | yes (j , i≡xsₗ∙j) =
    wf-index (xs .next) j (wf j (xs .length , i≡xsₗ∙j ; comm _ _ , xs .length≢ε))

  ℱ-Segments→Traced : ℱ-Segments E A → Traced E A
  ℱ-Segments→Traced xs i = wf-index xs i (wf i)

Trying to build an isomorphism

So the ℱ-Segments type is interesting, but it only really gives one side of the isomorphism. There is no way to write a function Traced E A → ℱ-Segments E A.

The problem is that there’s no way to get the “next� segment from a function E → A. We can find the label of the first segment, by applying the function to ε, but there’s no real way to figure out the size of this segment. We can change Traced little to provide this size, though.

Ind : ∀ E → ⦃ _ : Monus E ⦄ → Type → Type
Ind E A = E → A × Σ[ length ⦂ E ] × (length ≢ ε)

This new type will return a tuple consisting of the value indicated by the supplied index, along with the distance to the next segment. For instance, on the example stream given in the diagram earlier, supplying an index i that is bigger than a but smaller than a + b, this function should return y along with some j such that i + j ≡ a + b. Diagrammatically:

╷⇤i╌╌╌╌╌╌╌╌⇥╷⇤j╌╌⇥╷
┢��������┳��┷�����╈������┉
┃x       ┃y       ┃z     ┉
┡��������╇��������╇������┉
╵⇤a╌╌╌╌╌⇥╵⇤b╌╌╌╌╌⇥╵⇤c╌╌╌╌┈

This can be implemented in code like so:

module _ ⦃ _ : Monus E ⦄ where
  wf-ind : ℱ-Segments E A → (i : E) → Acc _≺_ i → A × ∃ length × (length ≢ ε)
  wf-ind xs i _ with xs .length ≤? i
  ... | no xsₗ≰i =
    let j , _ , j≢ε = <⇒≺ i (xs .length) xsₗ≰i
    in xs .label , j , j≢ε
  wf-ind xs i (acc wf) | yes (j , i≡xsₗ∙j) =
    wf-ind (xs .next) j (wf j (xs .length , i≡xsₗ∙j ; comm _ _ , xs .length≢ε))

  ℱ-Segments→Ind : WellFounded _≺_ → ℱ-Segments E A → Ind E A
  ℱ-Segments→Ind wf xs i = wf-ind xs i (wf i)

Again, if the monus has finite descending chains, this function is terminating. And the nice thing about this is that it’s possible to write a function in the other direction:

Ind→ℱ-Segments : ⦃ _ : Monus E ⦄ → Ind E A → ℱ-Segments E A
Ind→ℱ-Segments ind =
  let x , s , s≢ε = ind ε
  in λ where .label    → x
             .length   → s
             .length≢ε → s≢ε
             .next     → Ind→ℱ-Segments (ind ∘ (s ∙_))

The problem here is that this isomorphism is only half correct. We can prove that converting to Ind and back is the identity, but not the other direction. There are too many functions in Ind.

Nonetheless, it’s still interesting!

State Comonad

There is a comonad on state (Waern 2018; Kmett 2018) that is different from store. Notice that above the Ind type has the same type (almost) as State E A.

This is interesting in two ways: first, it gives some concrete, spatial intuition for what’s going on with the state comonad.

Second, it gives a kind of interesting monad instance on the stream. If we apply the Ind→ℱ-Segments function to the implementation of join on state, we should get a join on ℱ-Segments. And we do!

First, we need to redefine Ind to the following:

�-Ind : ∀ E → ⦃ _ : Monus E ⦄ → Type → Type
�-Ind E A = (i : E) → A × Σ[ length ⦂ E ] × (i ≺ length)

This is actually isomorphic to the previous definition, but we return the absolute value of the next segment, rather than the distance to the next segment.

�-iso : ⦃ _ : Monus E ⦄ → �-Ind E A ⇔ Ind E A
�-iso .fun xs i =
  let x , s , k , s≡i∙k , k≢ε = xs i
  in  x , k , k≢ε
�-iso .inv xs i =
  let x , s , s≢ε = xs i
  in  x , i ∙ s , s , refl , s≢ε
�-iso .rightInv _ = refl
�-iso .leftInv  xs p i = 
  let x , s           , k , s≡i∙k                   , k≢ε = xs i
  in  x , s≡i∙k (~ p) , k , (λ q → s≡i∙k (~ p ∨ q)) , k≢ε

The implemention of join on this type is the following:

�-join : ⦃ _ : Monus E ⦄ → �-Ind E (�-Ind E A) → �-Ind E A
�-join xs i =
  let x , j , i<j = xs i
      y , k , k<j = x j
  in  y , k , ≺-trans i<j k<j

This is the same definition of join as for State, modulo the < fiddling.

On a stream, this operation corresponds to taking a stream of streams and collapsing it to a single stream. It does this by taking a prefix of each internal stream equal in size to the segment of the outer entry. Diagrammatically:

�����������┳������┳������┉
┃xs        ┃ys    ┃zs    ┉
┡����������╇������╇������┉
╵⇤a╌╌╌╌╌╌╌⇥╵⇤b╌╌╌⇥╵⇤c╌╌╌╌┈
          ╱        ╲
         ╱          ╲
        ╱            ╲
       ╱              ╲
      ╱                ╲
     ╷⇤b╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌⇥╷
     ┢�������┳������┳���┷┉
ys = ┃xʸ     ┃yʸ    ┃zʸ  ┉
     ┡�������╇������╇����┉
     ╵⇤aʸ╌╌╌⇥╵⇤bʸ╌╌⇥╵⇤cʸ╌┈

Here we start with a stream consisting of the streams xs, ys, and zs, followed by some other streams. Zooming in on ys, we see that it is in a segment of length b, and consists of three values xʸ, yʸ, and zʸ, with segment lengths aʸ, bʸ, and cʸ, respectively.

Calling join on this stream will give us the following stream:

��┉�┳����┳����┳�����┳����┉
┃ ┉ ┃xʸ  ┃yʸ  ┃zʸ   ┃    ┉
┡�┉�╇����╇����╇�����╇����┉
│   │⇤aʸ⇥╵⇤bʸ⇥╵⇤╌╌┈⇥│
╵⇤a⇥╵⇤b╌╌╌╌╌╌╌╌╌╌╌╌⇥╵⇤c╌╌┈

Again, we’re focusing on the ys section here, which occupies the segment from a to a ∙ b. After join, this segment is occupied by three elements, xʸ, yʸ, and zʸ.

Notice that this isn’t quite the normal join on streams. That join takes a stream of streams, and turns the ith entry into the ith entry in the underlying stream. It’s a diagonalisation, in other words.

This one is kind of similar, but it takes chunks of the outer stream.

Theory

All of this so far is very hand-wavy. We have an almost isomorphism (a split surjection, to be precise), but not much in the way of concrete theoretical insights, just some vague gesturing towards spatial metaphors and so on.

Thankfully, there are two seperate areas of more serious research that seem related to the stuff I’ve talked about here. The first is update monads and directed containers, and the second is graded comonads. I think I understand graded comonads and the related work better out of the two, but update monads and directed containers seems more closely related to what I’m doing here.

Update Monads and Directed Containers

There are a few papers on this topic: Ahman, Chapman, and Uustalu (2012), Ahman and Uustalu (2013; Ahman and Uustalu 2014; Ahman and Uustalu 2016).

The first of these, “When Is a Container a Comonad?� constructs, as the title suggests, a class for containers which are comonads in a standard way.

Here’s the definition of a container:

Container : Type�
Container = Σ[ Shape ⦂ Type ] × (Shape → Type)

⟦_⟧ : Container → Type → Type
⟦ S , P ⟧ X = Σ[ s ⦂ S ] × (P s → X)

Containers are a generic way to describe a class of well-behaved functors. Any container is a pair of a shape and position. Lists, for instance, are containers, where their shape is described by the natural numbers (the shape here is the length of the list). The positions in such a list are the numbers smaller than the length, in dependently-typed programming we usually use the Fin type for this:

Fin : ℕ → Type
Fin n = ∃ m × (m <ℕ n)

The container version of lists, then, is the following:

ℒ��� : Type → Type
ℒ��� = ⟦ ℕ , Fin ⟧

Here’s the same list represented in the standard way, and as a container:

The benefit of using containers is that it gives a standard, generic, and composable way to construct functors that have some nice properties (like strict positivity). They’re pretty annoying to use in practice, though, which is a shame.

Directed containers are container that have three extra operations.

  • A tail-like operation, where a position can be converted into the shape of containers that the suffic from that position.
  • A head-like operation, where you can always return the root position.
  • A +-like operation, where you take a position on some tail and translate it into a position on the original container, by adding it.

As the paper observes, these are very similar to a “dependently-typed� version of the monoid methods. This seems to me to be very similar to the indexing stuff we were doing earlier on.

The real interesting part is in the paper “Updated Monads: Cointerpreting Directed Containers� (Ahman and Uustalu 2014). This paper presents a variant on state monads, called “update monads�.

These are monads that use a monoid action:

record RightAction (� : Type) (� : Type) : Type where
  infixl 5 _↓_
  field
    ⦃ monoid⟨�⟩ ⦄ : Monoid �
    _↓_ : � → � → �
    ↓-assoc : ∀ x y z → (x ↓ y) ↓ z ≡ x ↓ (y ∙ z)
    ↓-ε : ∀ x → x ↓ ε ≡ x

A (right) monoid action is a monoid along with a function ↓ that “acts� on some other set, in a way that coheres with the monoid methods. The definition is given above. One way to think about it is that if a monoid � has an action on � it means that elements of � can kind of be transformed into elements of � → �.

This can be used to construct a monad that looks suspiciously like the state monad:
Upd : (� � : Type) ⦃ _ : RightAction � � ⦄ → Type → Type
Upd � � X = � → � × X

η : ⦃ _ : RightAction � � ⦄ → A → Upd � � A
η x s = ε , x

μ : ⦃ _ : RightAction � � ⦄ → Upd � � (Upd � � A) → Upd � � A
μ xs s = let p , x = xs s
             q , y = x (s ↓ p)
         in  (p ∙ q , y)

It turns out that the dependently-typed version of this gives directed containers.

Grading and the Cofree Comonad

I’m still in the early stages of understanding all of this material, but at the moment graded comonads and transformers are concepts that I’m much more familiar and comfortable with.

The idea behind graded monads and comonads is similar to the idea behind any indexed monad: we’re adding an extra type parameter to the monad or type, which can constrain the operations involved. The graded monads and comonads use a monoid as that index. This works particularly nicely, in my opinion: just allowing any index at all sometimes feels a little unstructured. The grading construction seems to constrain things to the right degree: the use of the monoid, as well, works really well with comonads.

That preamble out of the way, here’s the definition of a graded comonad:

record GradedComonad (� : Type) ⦃ _ : Monoid � ⦄ (� : � → Type → Type) : Type� where
  field
    extract : � ε A → A
    extend  : (� y A → B) → � (x ∙ y) A → � x B
This also has a few laws, which are expressed cleaner using cokleisli composition:
  _=<=_ : (� x B → C) → (� y A → B) → � (x ∙ y) A → C
  (g =<= f) x = g (extend f x)

  field
    idˡ : (f : � x A₀ → B₀) → PathP (λ i → � (ε∙ x i) A₀ → B₀) (extract =<= f) f
    idʳ : (f : � x A₀ → B₀) → PathP (λ i → � (∙ε x i) A₀ → B₀) (f =<= extract) f
    c-assoc : (f : � x C₀ → D₀) (g : � y B₀ → C₀) (h : � z A₀ → B₀) →
          PathP (λ i → � (assoc x y z i) A₀ → D₀) ((f =<= g) =<= h) (f =<= (g =<= h))

This seems to clearly be related to the stream constructions. Grading is all about the monoidal information about a comonad: the streams above are a comonad which indexes its entries with a monoid.

There are now two constructions I want to show that suggest a link betweent the stream constructions and graded comonads. First of these is the Cofree degrading comonad:

record G-CofreeF (� : Type → Type) (� : � → Type → Type) (A : Type) : Type where
  coinductive; constructor _â—ƒ_
  field here : A
        step : � (∃ w × � w (G-CofreeF � � A))
open G-CofreeF

G-Cofree : ⦃ _ : Monoid � ⦄ → (Type → Type) → (� → Type → Type) → Type → Type
G-Cofree � � A = � ε (G-CofreeF � � A)

This construction is similar to the cofree comonad transformer: it is based on the cofree comonad, but with an extra (graded) comonad wrapped around each level. For any functor � and graded comonad �, G-Cofree � � is a comonad. The implementation of extract is simple:

extract′ : ⦃ _ : Monoid � ⦄ ⦃ _ : GradedComonad � � ⦄ → G-Cofree � � A → A
extract′ = here ∘ extract

extend is more complex. First, we need a version of extend which takes a proof that the grade is of the right form:

module _ { � : � → Type → Type } where
  extend[_] : ⦃ _ : Monoid � ⦄ ⦃ _ : GradedComonad � � ⦄ →
              x ∙ y ≡ z → (� y A → B) → � z A → � x B
  extend[ p ] k = subst (λ z → � z _ → _) p (extend k)

Then we can implement the characteristic function on the free comonad: traceT. On graded comonads it has the following form:

module Trace ⦃ _ : Monoid � ⦄ ⦃ _ : GradedComonad � � ⦄ ⦃ _ : Functor � ⦄ where
  module _ {A B} where
    {-# NON_TERMINATING #-}
    traceT : (� ε A → B) → (� ε A → � (∃ w × � w A)) → � ε A → G-Cofree � � B
    traceT ϕ � = ψ
      where
      ψ : � x A → � x (G-CofreeF � � B)
      ψ = extend[ ∙ε _ ] λ x → ϕ x ◃ map (map₂ ψ) (� x)

This function is basically the unfold for the free degrading comonad. If G-Cofree is a internally-labelled tree, then ϕ above is the labelling function, and � is the “next� function, returning the children for some root.

Using this, we can implement extend:

  extend′ : (G-Cofree � � A → B) → G-Cofree � � A → G-Cofree � � B
  extend′ f = traceT f (step ∘ extract)

The relation between this and the stream is that the stream can be defined in terms of this: Stream W = G-Cofree id (GC-Id W).

Finally, the last construction I want to introduce is the following:

module _ ⦃ _ : Monus � ⦄ where
  data Prefix-F⊙ (� : Type → Type) (� : � → Type → Type) (i j : �) (A : Type) : Type where 
    prefix : ((i≤j : i ≤ j) → A × � (∃ k × � k (Prefix-F⊙ � � k (fst i≤j) A))) → Prefix-F⊙ � � i j A

  Prefix⊙ : (� : Type → Type) (� : � → Type → Type) (j : �) (A : Type) → Type
  Prefix⊙ � � j A = � ε (Prefix-F⊙ � � ε j A)

  Prefix : (� : Type → Type) (� : � → Type → Type) (A : Type) → Type
  Prefix � � A = ∀ {i} → Prefix⊙ � � i A

This type is designed to mimic sized type definitions. It has an implicit parameter which can be set, by the user of the type, to some arbitrary depth. Basically the parameter means “explore to this depth�; by using the ∀ we say that it is defined up to any arbitrary depth.

When the ≺ relation on the monus is well founded it is possible to implement traceT:

  module _ ⦃ _ : GradedComonad � � ⦄ ⦃ _ : Functor � ⦄ (wf : WellFounded _≺_) {A B : Type} where
    traceT : (� ε A → B) → (� ε A → � (∃ w × (w ≢ ε) × � w A)) → � ε A → Prefix � � B
    traceT ϕ � xs = extend[ ∙ε _ ] (λ xs′ → prefix λ _ → ϕ xs′ ,  map (map₂ (ψ (wf _))) (� xs)) xs
      where
      ψ : Acc _≺_ y → (x ≢ ε) × � x A → � x (Prefix-F⊙ � � x y B)
      ψ (acc wf) (x≢ε , xs) =
        extend[ ∙ε _ ]
          (λ x → prefix
            λ { (k , y≡x∙k) →
              Ï• x , map
                (λ { (w , w≢ε , xs) →
                  w , ψ (wf k (_ , y≡x∙k ; comm _ _ , x≢ε)) (w≢ε , xs)}) (� x)})
          xs

Conclusion

Comonads are much less widely used than monads in Haskell and similar languages. Part of the reason, I think, is that they’re too powerful in a non-linear language. Monads are often used to model sublanguages where it’s possible to introduce “special� variables which interact with the monadic context.

pyth = do
  x <- [1..10]
  y <- [1..10]
  z <- [1..10]
  guard (x*x + y*y == z*z)
  return (x,y,z)

The x variable here semantically spans over the range [1..10]. In the following two examples we see the semantics of state and maybe:

sum :: [Int] -> Int
sum xs = flip evalState 0 $ do
  put 0
  for_ xs $ \x -> do
    n <- get
    put (n + x)
  m <- get
  return m
data E = Lit Int | E :+: E | E :/: E

eval :: E -> Maybe Int
eval (Lit n) = n
eval (xs :+: ys) = do x <- eval xs
                      y <- eval ys
                      return (x + y)
eval (xs :/: ys) = do x <- eval xs
                      y <- eval ys
                      guard (y /= 0)
                      return (x / y)

The variables n and m introduced in the state example are “special� because their values depend on the computations that came before. In the maybe example the variables introduced could be Nothing.

You can’t do the same thing with comonads because you’re always able to extract the “special� variable with extract :: m a -> a. Instead of having special variable introduction, comonads let you have special variable elimination. But, since Haskell isn’t linear, you can always just discard a variable so this isn’t much use.

Looking at the maybe example, we have a function eval :: E -> Maybe Int that introduces an Int variable with a “catch�: it is wrapped in a Maybe. We want to use the eval function as if it were a normal function E -> Int, with all of the bookkeeping managed for us: that’s what monads and do notation (kind of) allow us to do.

An analagous example with comonads might be having a function consume :: m V -> String. This “handles� a V value, but the “catch� is that it needs an m context to do so. If we want to treat the consume function as if it were a normal function V -> String then comonads (and codo notation Orchard and Mycroft 2013) would be a perfect fit.

The reason that this analagous case doesn’t arise very often is that we don’t have many handlers that look like m V -> String in Haskell. Why? Because if we want to “handle� a V we can just discard it: as a non-linear language, you do not need to perform any ceremony to discard a variable in Haskell.

Graded comonads, though, seem to be much more useful than normal comonads. I think it is becuase they basically get rid of the m a -> a function, changing it into a much more restricted form. In this way, they give a kind of small linear language, but just for the monoidal type parameter.

And there are a lot of uses for the graded comonads. Above we’ve used them for termination checking. A recursive function might have the form a -> b, where a is the thing being recursed on. If we’re using well-founded recursion to show that it’s terminating, though, we add an extra parameter, an Acc _<_ proof, turning this function into Acc _<_ w × a -> b. The Acc _<_ here is the graded comonad, and this recursive function is precisely the “handler�.

Other examples might be provacy or permissions: a function might be able to work on some value, but only if it has particular permission regarding that value. The permission here is the monoid.

There are other examples I’m sure, those are just the couple that I have been thinking about.

References

Abel, Andreas, and Brigitte Pientka. 2013. “Wellfounded Recursion with Copatterns� (0) (June): 25. http://www2.tcs.ifi.lmu.de/%7Eabel/icfp13-long.pdf.

Ahman, Danel, James Chapman, and Tarmo Uustalu. 2012. “When Is a Container a Comonad?� In Foundations of Software Science and Computational Structures, 74–88. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg. doi:10.1007/978-3-642-28729-9_5.

Ahman, Danel, and Tarmo Uustalu. 2014. “Update Monads: Cointerpreting Directed Containers�: 23 pages. doi:10.4230/LIPICS.TYPES.2013.1.

———. 2013. “Distributive laws of directed containers.� Progress in Informatics (10) (March): 3. doi:10.2201/NiiPi.2013.10.2.

———. 2016. “Directed Containers as Categories� (April). doi:10.4204/EPTCS.207.5.

Kidney, Donnacha Oisín, and Nicolas Wu. 2021. “Algebras for Weighted Search.� Proceedings of the ACM on Programming Languages 5 (ICFP) (August): 72:1–72:30. doi:10.1145/3473577.

Kmett, Edward. 2018. “The State Comonad.� Blog. The Comonad.Reader. http://comonad.com/reader/2018/the-state-comonad/.

Orchard, Dominic, and Alan Mycroft. 2013. “A Notation for Comonads.� In Implementation and Application of Functional Languages, ed by. Ralf Hinze, 1–17. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer. doi:10.1007/978-3-642-41582-1_1.

Waern, Love. 2018. “I made a monad that I haven’t seen before, and I have a few questions about it.� Reddit Post. reddit.com/r/haskell. https://www.reddit.com/r/haskell/comments/7oav51/i_made_a_monad_that_i_havent_seen_before_and_i/.

by Donnacha Oisín Kidney at May 03, 2022 12:00 AM

May 02, 2022

ERDI Gergo

Cheap and cheerful microcode compression

This post is about an optimization to the Intel 8080-compatible CPU that I describe in detail in my book Retrocomputing in Clash. It didn't really fit anywhere in the book, and it isn't as closely related to the FPGA design focus of the book, so I thought writing it as a blog post would be a good idea.

Retrocomputing with Clash

Just like the real 8080 from 1974, my Clash implementation is microcoded: the semantics of each machine code instruction of the Intel 8080 is described as a sequence of steps, each step being the machine code instruction of an even simpler, internal micro-CPU. Each of these micro-instruction steps are then executed in exactly one clock cycle each.

My 8080 doesn't faithfully replicate the hardware 8080's micro-CPU; in fact, it doesn't replicate it at all. It is a from-scratch design based on a black box understanding of the 8080's instruction set, and the main goal was to make it easy to understand, instead of making it efficient in terms of FPGA resource usage. Of course, since my micro-CPU is different, the micro-instructions have no one to one correspondence with the orignal Intel 8080, and so the microcode is completely different as well.

In the CPU, after fetching the machine code instruction byte, we look up the microcode for that byte, and then execute it cycle by cycle until we're done. This post is about how to store that microcode efficiently.

An illustrative example

To avoid dealing with the low-level details of what exactly goes on in our microcode, for the rest of this blog post let's use a dictionary of a small handful of English words as our running example. Suppose that we want to store the following table:

0.  shape
1.  shaping
2.  shift
3.  shapeshifting
4.  ape
5.  aping
6.  ship
7.  shipping
8.  grape
9.  elope
10. shard
11. sharding
12. shared
13. geared
    

There's a lot of redundancy between these words, and we will see how to exploit that. But does this make it a poor example that won't generalize to our real use case of storing microcode? Not at all. There are lots of 8080 instructions that are just minimal variations of each other, such as doing the exact same operation but on different general purpose registers; thus, their microcode is also going to be very similar, doing the same setup/teardown around a different kernel.

Fixed length vectors

Since our eventual goal is designing hardware, everything ultimately needs a fixed size. The most straightforward representation of our dictionary, then, is as a vector that is sized to fit the longest single word:

      type Dictionary = Vec 14 (Vec 13 Char)
    

The longest word "shapeshifting" is 13 characters. For all 14 possible inputs, we store 13 characters, using a special "early termination" marker like '.' in the middle for those words that are shorter:

0.  shape........
1.  shaping......
2.  shift........
3.  shapeshifting
4.  ape..........
5.  aping........
6.  ship.........
7.  shipping.....
8.  grape........
9.  elope........
10. shard........
11. sharding.....
12. shared.......
13. geared.......
    

We can then use this table very easily in a hardware implementation: after fetching the "instruction", i.e. the dictionary key, we look up the corresponding Vec 13 Char in the dictionary ROM, and keep a 4-bit counter of type Index 13 to process it cycle by cycle.

This is the equivalent of the microcode representation that we use in Retrocomputing in Clash, but it is easy to see that it is very wasteful. In our illustrative example, we store a total of 14 ⨯ 13 = 182 characters, whereas the total length of all strings is only 85, so we waste about 55% of our storage.

On our 8080-compatible CPU we get similar (slightly worse) numbers: the longest instruction, XTHL, takes 18 cycles. We don't need to store microcode for the first cycle, since that always corresponds to just fetching the instruction byte itself. This leaves us with 17 micro-operations. For all 256 possible machine code instruction bytes, we end up storing a total of 256 ⨯ 17 = 4352 micro-operations, but if we look at the cycle count of each individual 8080 instruction, the useful part is only 1493 micro-operations. That's a waste of about 65%.

Using terminators

No, wait, not this guy.

The obvious way to cut down on some of that fat is to store each word only up to its end. We can use a terminated representation for this, by keeping some end-of-word marker ('.' in the examples below), and concatenating all items:

0.  shape.
6.  shaping.
14. shift.
20. shapeshifting.
34. ape.
38. aping.
44. ship.
49. shipping.
58. grape.
64. elope.
70. shard.
76. sharding.
85. shared.
92. geared.
    

Instead of storing 182 characters, we now only store 99. While this is still more than 85, because we also have to store all those word-separating '.' markers, it is still a big improvement.

However, there's a bit of cheating going in here, because with the above table as given, we'd have no way of looking up words by their original index. For example, word #7 is shipping, but if we started at entry number 7 in this representation, we'd get haping. We need to also store a table of contents that gives us the starting address of each dictionary entry:

0.  0
1.  6
2.  14
3.  20
4.  34
5.  38
6.  44
7.  49
8.  58
9.  64
10. 70
11. 76
12. 85
13. 92
    

If we want to calculate the contribution of the table of contents to the total size, we have to get a bit more precise. Previously, we characterized ROM footprint in units of characters, but now we need to store 7-bit indices as well. To be able to add the two together, we need to also fix the bit width of each character. For now, let's just use 8 bits per character.

The total size in bits, for storing the table of contents and the dictionary in terminated form, comes out to 14 ⨯ 7 + 99 ⨯ 8 = 890. We can compare this to the 14 ⨯ 13 ⨯ 8 = 1456 bits of the fixed-length representation to see that it's a huge improvement.

Linked lists

Not this guy either.

As we've seen, the table of contents takes up 98 bits, or about 11% of our total footprint in the terminated representation. Can we get rid of it?

One way of doing this is to change the starting address of each word to its key. This is already the case for our first word, shape, since its key is 0 and it starts at address 0. However, the next word, shaping, can't start at address 1, since that is where the second letter of the first word resides.

If we store the next character's address instead of making the assumption that it's going to be the next address, we can start each word at the address corresponding to its key, and then leave subsequent letters to addresses beyond the largest key:

0.  s → @14
1.  s → @18
2.  s → @24
3.  s → @28
4.  a → @40
5.  a → @42
6.  s → @46
7.  s → @49
8.  g → @56
9.  e → @60 
10. s → @64
11. s → @68
12. s → @75 
13. g → @80
14. h → a → p → e → @85
18. h → a → p → i → n → g → @85
24. h → i → f → t → @85
28. h → a → p → e → s → h → i → f → t → i → n → g → @85
40. p → e → @85
42. p → i → n → g → @85
46. h → i → p → @85
49. h → i → p → p → i → n → g → @85
56. r → a → p → e → @85
60. l → o → p → e → @85
64. h → a → r → d → @85
68. h → a → r → d → i → n → g → @85
75. h → a → r → e → d → @85
80. e → a → r → e → d → @85
    

Not only did we get rid of the table of contents, we can also store the terminators more implicitly, by using a special value for the next pointer. In this example, we can use @85 for that purpose, pointing beyond the last cell. This leaves us with just 85 cells compared to the 99 cells with terminators.

However, each cell now contains both a character and a pointer. Since we need to address 85 cells, the latter takes up 7 bits, for a total of 85 ⨯ (8 + 7) = 1275 bits.

This is a step back from the the terminated representation's 890 bits. We can make a note, though, that if each character was at least 35 bits wide instead of 8, the linked representation would come out ahead. But the real reason we are interested in the linked-list form is that it suggests a further optimization that finally exploits the redundancy between the words in our dictionary.

Common suffix elimination

This is where we get to the actual meat of this post. Let's focus on the following subset of our linked list representation:

12. s → @75 
13. g → @80
75. h → a → r → e → d → @85
80. e → a → r → e → d → @85
    

We are storing the shared suffix ared twice, when instead, we could redirect the second one to the first occurrence, saving 4 cells:

12. s → @75 
13. g → @80
75. h → @76
76. a → r → e → d → @85
80. e → @76
    

If we apply the same idea to all words, we arrive at the following linked representation. Note that we still start every word at the index corresponding to its key, avoiding the need for a table of contents:

0.  s → @14
1.  s → @15
2.  s → @16
3.  s → @20
4.  a → @32
5.  a → @34
6.  s → @35
7.  s → @38
8.  g → @41
9.  e → @42
10. s → @44
11. s → @48
12. s → @52
13. g → @56
14. h → @4
15. h → @5
16. h → i → f → t → @57
20. h → a → p → e → s → h → i → f → t → @29
29. i → n → g → @57
32. p → e → @57
34. p → @29
35. h → i → p → @57
38. h → i → p → @34
41. r → @4
42. l → o → @32
44. h → a → r → @47
47. d → @57
48. h → a → r → d → @29
52. h → @53
53. a → r → e → @47
56. e → @53
    

It's hard to see what exactly is going on here from this textual format, but things become much cleaner if we display it as a graph:

We can compute the size of this representation along the same lines as the linked-list one, except now we only have 57 cells. This also means that the pointers can be 6 bits instead of 7, for a total size of 57 ⨯ (8 + 6) = 798 bits. A 10% save compared to the 890 bits of the terminated representation!

Going back to our real-world use case of 8080 microcode, each micro-instruction is 15 bits wide. We have already computed that the fixed-length representation uses 4352 ⨯ 15 = 65,280 bits; if we do the same calculation for the other representations, we get 28,286 bits in the terminated representation, 37,492 bits with linked lists, and a mere 13,675 bits, that is, just 547 cells, with the common suffixes shared!

So how do we compute this shared-suffix representation? Luckily, it turns out we can do that in just a handful of lines of code.

Reverse trie representation

To come up with the basic idea, let's start by thinking about why we are aiming to share common suffixes instead of common prefixes, such as the prefix between shape and shaping. The answer, of course, is that we need to address each word separately. If we try to unify shape and shaping, starting with the key 0 or 1 and making it as far as shap doesn't tell us on its own if we should continue with e or ing for the given key.

On the other hand, once we start a given word, it doesn't matter where subsequent letters are, including even the beginning of other words (as is the case between shaping and aping). So fan-out is bad (it would mean having to make a decision), but fan-in is a-OK.

There's an obvious data structure for exploiting common prefixes: we can put all our words in a trie. We can make one in Haskell by using a finite map from the next key element to the stored data (for terminal nodes) and the rest of the trie:

import Data.List.NonEmpty (NonEmpty(..))
import qualified Data.List.NonEmpty as NE     
import qualified Data.Map as M

newtype Trie k a = MkTrie{ childrenMap :: M.Map k (Maybe a, Trie k a) }

children :: Trie k a -> [(k, Maybe a, Trie k a)]
children t = [(k, x, t') | (k, (x, t')) <- M.toList $ childrenMap t]
    

Note that a terminal node doesn't necessarily mean no children, since one full key may be a proper prefix of another key. For example, if we build a trie that stores "FOO" ↦ 1 and "FOOBAR" ↦ 2, then the node at 'F' → 'O' → 'O' will contain value Just 1 and also the child trie for 'B' → 'A' → 'R'.

The main operation on a Trie that we will need is building one from a list of (key, value) pairs via repeated insertion:

empty :: Trie k a
empty = MkTrie M.empty

insert :: (Ord k) => NonEmpty k -> a -> Trie k a -> Trie k a
insert ks x = insertOrUpdate (const x) ks

insertOrUpdate :: (Ord k) => (Maybe a -> a) -> NonEmpty k -> Trie k a -> Trie k a
insertOrUpdate f = go
  where
    go (k :| ks) (MkTrie ts) = MkTrie $ M.alter (Just . update . fromMaybe (Nothing, empty)) k ts
      where
        update (x, t) = case NE.nonEmpty ks of
            Nothing  -> (Just $ f x, t)
            Just ks' -> (x, go ks' t)

fromList :: (Ord k) => [(NonEmpty k, a)] -> Trie k a
fromList = foldr (uncurry insert) empty
    

Ignoring the actual indices for a moment, looking at a small subset of our example consisting of only shape, shaping, and aping, the trie we'd build from it looks like this:

But we want to find common suffixes, not prefixes, so how does all this help us with that? Well, a suffix is just a prefix of the reversed sequence, so watch what happens when we build a trie after reversing each word, and lay it out left-to-right:

In this representation, terminal nodes correspond to starting letters of each word, so we can store the dictionary index as the value associated with them:

At this point, it should be clear how we are going to build our nicely compressed representation: we build a suffix trie, and then flatten it by traversing it bottom up. Before we do that, though, let's take care of one more subtlety: what if we have the same word at multiple indices in our dictionary? This is not an invalid case, and does come up in practice for the multiple NOP instructions of the 8080, all mapping to the exact same microcode. The solution is to simply allow a NonEmpty list of dictionary keys on terminal nodes in the resulting trie:

fromListMany :: (Ord k) => [(NonEmpty k, a)] -> Trie k (NonEmpty a)
fromListMany = foldr (\(ks, x) -> insertOrUpdate ((x :|) . maybe [] NE.toList) ks) empty

suffixTree :: (KnownNat n, Ord a) => Vec n (NonEmpty a) -> Trie a (NonEmpty (Index n))
suffixTree = fromListMany . toList . imap (\i word -> (NE.reverse word, i))
    

Flattening

The pipeline going from a suffix tree to flat ROM payload containing the linked-list representation has three steps:

  1. Allocate an address to each cell, and fill in the links. We can make our lives much easier by using Either (Index n) Int for the addresses: Left i is an index from the original dictionary, and Right ptr means it corresponds to the middle of a word.
  2. Renumber the addresses by reserving the first n addresses to the original indices (remember, this is how we avoid the need for a table of contents), and using the rest for the internal ones.
  3. After step two we have a single continuous address block of cells. All that remains to be done is reordering the elements, sorting each one into its own position.
compress :: forall n a. (KnownNat n, Ord a) => Vec n (NonEmpty a) -> [(a, Maybe Int)]
compress = reorder . renumber . links . suffixTree
  where
    reorder = map snd . sortBy (comparing fst)

    renumber xs = [ (flatten addr, (x, flatten <$> next)) | (addr, x, next) <- xs ]
      where
        offset = snatToNum (SNat @n)

        flatten (Left k) = fromIntegral k
        flatten (Right idx) = idx + offset
    

Since we don't know the full size of the resulting ROM upfront, we have to use Int as the final unified address type; this is not a problem in practice since for our real use case, all this microcode compression code runs at compile time via Template Haskell, so we can dynamically compute the smallest pointer type and just fromIntegral the link pointers into that.

We conclude the implementation with links, the function that computes the next pointers. The cell emitted for each trie node should link to the cell for its parent node, so we pass the parent down as we traverse the trie (this is the next parameter below). The new cell itself is either put in the next empty cell if it is not a terminal node, i.e. if it doesn't correspond to a first letter in our original dictionary; or, it and all its aliases are emitted at their corresponding Left addresses.

links :: Trie k (NonEmpty a) -> [(Either a Int, k, Maybe (Either a Int))]
links = execWriter . flip runStateT 0 . go Nothing
  where
    go next = mapM_ (node next) . children

    node next (k, mx, t') = do
        this <- case mx of
            Nothing -> Right <$> alloc
            Just (x:|xs) -> do
                tell [(Left x', k, next) | x' <- xs]
                return $ Left x
        tell [(this, k, next)]
        go (Just this) t'

    alloc = get <* modify succ
    

If you want to play around with this, you can find the full code on GitHub; in particular, there's Data.Trie for the no-frills trie implementation, and Hardware.Intel8080.Microcode.Compress implementing the microcode compression scheme described in this post.

And finally, just for the fun of it, this is what the 8080 microcode — that prompted all this — looks like in its suffix tree form, clearling showing the clusters of instructions that share the same epilogue:

May 02, 2022 07:05 PM

May 01, 2022

Philip Wadler

Object-Oriented Programming — The Trillion Dollar Disaster

 


Elixir engineer Ilya Suzdalnitski explains from the perspective of an engineer who has used both why he prefers functional programming to object-oriented programming systems. OOPS!

by Philip Wadler (noreply@blogger.com) at May 01, 2022 03:12 PM

GHC Developer Blog

GHC 9.4.1-alpha1 released

GHC 9.4.1-alpha1 released

bgamari - 2022-05-01

The GHC developers are happy to announce the availability of the first alpha release of the GHC 9.4 series. Binary distributions, source distributions, and documentation are available at downloads.haskell.org.

This major release will include:

  • A new profiling mode, -fprof-late, which adds automatic cost-center annotations to all top-level functions after Core optimisation has run. This incurs significantly less performance cost while still providing informative profiles.

  • A variety of plugin improvements including the introduction of a new plugin type, defaulting plugins, and the ability for typechecking plugins to rewrite type-families.

  • An improved constructed product result analysis, allowing unboxing of nested structures, and a new boxity analysis, leading to less reboxing.

  • Introduction of a tag-check elision optimisation, bringing significant performance improvements in strict programs.

  • Generalisation of a variety of primitive types to be levity polymorphic. Consequently, the ArrayArray# type can at long last be retired, replaced by standard Array#.

  • Introduction of the \cases syntax from GHC proposal 0302

  • A complete overhaul of GHC’s Windows support. This includes a migration to a fully Clang-based C toolchain, a deep refactoring of the linker, and many fixes in WinIO.

  • Support for multiple home packages, significantly improving support in IDEs and other tools for multi-package projects.

  • A refactoring of GHC’s error message infrastructure, allowing GHC to provide diagnostic information to downstream consumers as structured data, greatly easing IDE support.

  • Significant compile-time improvements to runtime and memory consumption.

  • … and much more

We would like to thank Microsoft Azure, GitHub, IOG, the Zw3rk stake pool, Tweag I/O, Serokell, Equinix, SimSpace, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

Happy testing,

  • Ben

by ghc-devs at May 01, 2022 12:00 AM

April 30, 2022

Ken T Takusagawa

[ljxgdqve] makeRegexOpts example

we demonstrate how to use the functions makeRegex and makeRegexOpts using the regex-tdfa Haskell regular expression package.

the key point is, you cannot use =~ if you want to use these functions.  if you do, for example:

bad :: String -> Bool;
bad s = s =~ (makeRegex "[[:digit:]]");

you will get inscrutable error messages:

* Ambiguous type variable `source0' arising from a use of `=~' prevents the constraint `(RegexMaker Regex CompOption ExecOption source0)' from being solved.

* Ambiguous type variables `source0', `compOpt0', `execOpt0' arising from a use of `makeRegex' prevents the constraint `(RegexMaker source0 compOpt0 execOpt0 [Char])' from being solved.

instead, you have to use matchTest or similar functions described in Text.Regex.Base.RegexLike in regex-base.  the functions are reexported by but not documented in Text.Regex.TDFA .

https://gabebw.com/blog/2015/10/11/regular-expressions-in-haskell is a good explanation.

below is an example program that searches case-insensitively for input lines that contain the substring "gold", equivalent to "grep -i gold".  we need to use makeRegexOpts to disable case sensitivity.

module Main where {
import qualified Text.Regex.TDFA as Regex;

main :: IO();
main = getContents >>= ( mapM_ putStrLn . filter myregex . lines);

myregex :: String -> Bool;
myregex s = Regex.matchTest r s where {
  r :: Regex.Regex;
  r = Regex.makeRegexOpts mycompoptions myexecoptions "gold" ;
  mycompoptions :: Regex.CompOption;
  mycompoptions = Regex.defaultCompOpt {Regex.caseSensitive = False}; -- record syntax
  myexecoptions :: Regex.ExecOption;
  myexecoptions = Regex.defaultExecOpt;
};
}

here is documentation about all the available ExecOption and CompOption for this TDFA regex implementation.

previously, on the lack of substitution in Haskell regexes.

by Unknown (noreply@blogger.com) at April 30, 2022 07:06 PM

April 27, 2022

Philip Wadler

How to Speak

A master class from Patrick Winston of MIT on how to present ideas clearly. Chockfull of useful advice, much of which I've not seen elsewhere. Recommended.

by Philip Wadler (noreply@blogger.com) at April 27, 2022 04:46 PM

FP Complete

The Hidden Dangers of Haskell's Ratio Type

Here's a new Haskell WAT?!

Haskell has a type Rational for working with precisely-valued fractional numbers, and it models the mathematical concept of a rational number. Although it's relatively slow compared with Double, it doesn't suffer from the rounding that's intrinsic to floating-point arithmetic. It's very useful when writing tests because an exact result can be predicted ahead of time. For example, a computation that should produce zero will produce exactly zero rather than a small value within some range that would have to be determined.

Rational is actually a (monomorphic) specialization of the more general (polymorphic) type Ratio (from Data.Ratio). Ratio allows you to specify the underlying type used for the numerator and denominator. For example, to work with rational numbers using Int as the underlying type you can use Ratio Int. For the common case of using Integer as the underlying type, the type synonym Rational is provided:

type Rational = Ratio Integer

It's tempting to use Ratio with a fixed-width type like Int because Int is much faster than Integer. However, let's see what can happen if you do this:

λ> import Data.Int
λ> import Data.Ratio
λ> let r = 1 % 12 :: Rational   in r - r == 0
True
λ> let r = 1 % 12 :: Ratio Int8 in r - r == 0
False

WAT?!

Let's see what those subtracted values evaluate to:

λ> let r = 1 % 12 :: Rational   in r - r
0 % 1
λ> let r = 1 % 12 :: Ratio Int8 in r - r
0 % (-1)

Hmmm, let's see if that Ratio Int8 value is considered equal to 0:

λ> let r = 0 % (-1) :: Ratio Int8 in r == 0
True

WAT?!

Let's see what those manually-entered values are:

λ> 0 % (-1) :: Ratio Int8
0 % 1
λ> 0 :: Ratio Int8
0 % 1

OK, so these values really are equal, but why are the values in the subtraction different? The explanation is two-fold.

First, 0 % (-1) is a denormalized state for Ratio and shouldn't occur. (As you've probably suspected, it arises from integer overflow. More on that in a minute.) It's not too surprising, then, that it isn't equal to 0.

But why is it equal to 0 when we enter it directly? It's because % is a function not a constructor, and it normalizes the signs of the numerator and denominator before constructing the value:

x % y = reduce (x * signum y) (abs y)

The underlying assumption (the invariant) is that denominators will always be positive.

reduce is a function that reduces the numerator and denominator to their lowest terms, by dividing by the greatest common divisor:

reduce x y = (x `quot` d) :% (y `quot` d)
  where d = gcd x y

Here you can see the constructor that actually creates the values from their components, which is :%. It's not exported from Data.Ratio and the "smart constructor" % is used instead, to ensure that new Ratio values always satisfy the invariant.

Second, addition and subtraction are implemented without trying to minimize the possibility of integer overflow. For example:

(x :% y) - (x' :% y') = reduce (x * y' - x' * y) (y * y')

If y * y' overflows to a negative value, reduce will not normalize the signs. The result of gcd is always non-negative so the signs don't change and denormalized values are never renormalized. That happens only in % when constructing Ratio values.

Let's look at what happens in our example:

λ> x = 1; y = 12; x' = 1; y' = 12
λ> x * y' - x' * y :: Int8
0
λ> y * y' :: Int8
-112
λ> gcd 0 (-112)
112
λ> 0 `quot` 112
0
λ> (-112) `quot` 112
-1

The reduced result of 1 % 12 - 1 % 12 is therefore the denormalized value 0 :% (-1) which isn't considered equal to the normalized value 0 % 1.

Even though 12 is much less than maxBound :: Int8, when squared it results in integer overflow. The implementation of Num for Ratio is not designed to avoid overflows and they can happen very easily with numerators and denominators that are much less than the maxBound for the type.

The implementation could have used a slightly different approach:

(x :% y) - (x' :% y') = reduce (x * z' - x' * z) (y * z')
  where z = y `quot` d
        z' = y' `quot` d
        d = gcd y y'

However, the use of reduce is still necessary (consider 3 % 10 - 2 % 15) so this requires two more divisions and a gcd compared with the actual implementation.

Using a type as small as Int8 might seem a little unrealistic, but the problem can occur with any fixed-width integral type and I used Int8 for the illustration because it's easier to understand the problem when working with small values. I originally encountered it when using Ratio Int even though Int has a very large maxBound. I was writing property tests using QuickCheck for some polymorphic arithmetic code that was supposed to produce a zero sum as a result. The test succeeded with Rational and failed with Ratio Int and I couldn't understand why because the random values being generated by the test framework had numerators and denominators far less than maxBound :: Int. However, they were greater than its square root.

The documentation for Ratio says:

Note that Ratio's instances inherit the deficiencies from the type parameter's. For example, Ratio Natural's Num instance has similar problems to Natural's.

However, that doesn't really prepare you for what might happen with other type parameters! The moral of this story is that Ratio isn't much use on its own and you should always use Rational unless you really understand what you're getting into.

Further reading

Like that blog post? Check out the Haskell section of our site with tutorials and other blog posts. You can also check out all Haskell tagged blog posts.

We're hiring. Interested in working with our team on solving these kinds of WAT issues? Check out our jobs page for more information.

April 27, 2022 12:00 AM

April 24, 2022

Gil Mizrahi

Building a bulletin board using twain and friends

This is a port of my previous scotty tutorial for the twain web (micro) framework.

We are going to build a very simple bulletin board website using twain and friends. It'll be so simple that we won't even use a database, but hopefully it'll provide enough information on twain that you can continue it yourselves if you'd like.

But first, we are going to cover some of the basics of web programming, what are WAI and warp, and how to use twain.

Web programming and twain

Twain is a (tiny) server-side web framework, which means it provides a high-level API for describing web apps.

Twain is built on top of WAI, which is a lower level Web Application Interface. Warp is a popular web server implementation that runs WAI apps (also called a WAI handler).

A web server is a network application that receives requests from clients, processes them, and returns responses. The communication between the web client and web server follows the HTTP protocol. The HTTP protocol defines what kind of requests a user can make, such as "I want to GET this file", and what kind of responses the server can return, such as "404 Not Found".

wai provides a slightly low level mechanism of talking about requests and responses, and twain provides a bit more convenient mechanism than WAI for defining WAI apps. Warp takes descriptions of web programs that are written using WAI and provides the actual networking functionality, including the concurrent processing.

If you are interested in working with wai directly, Michael Snoyman's video workshop Your First Web App with WAI and Warp is a good place to learn more about it.

How to Run

Twain (and more specifically WAI) apps have the type Application, which can be considered as a specification of a web application. Application is a type alias:

type Application
  = Request -> (Response -> IO ResponseReceived) -> IO ResponseReceived

This type means that a WAI application is a function that takes a user's HTTP Request, and is expected to produce an HTTP Response to the user which it will pass to the function it got as a second argument (which is responsible for actually delivering the response, continuation passing style).

In order to run this application, we need to pass this function to a WAI handler which will do all of the networking heavy lifting of actually:

  • Opening sockets on a certain port
  • Receiving messages from the socket
  • Handling threading
  • Sending responses through the socket

...and so on. This is where warp comes in.

Once we have a WAI app, we can run it using the on a certain port using the run function from the warp package. This function will handle the heavy lifting and will call our web app when a Request from a user comes in, and will ask for a Respone from us.

Building a WAI Application with Twain

To build an Application with wai, one needs to take a Request and produce a Response. While this is fairly straightforward in principle, parsing a request and composing a response can be a bit tedious and repetative. For responses we need to branch over the HTTP method type, parse the route, extract the variable parts and branch on it, and so on. For requests, we need to set the HTTP status, the HTTP headers, the response body, and so on.

Twain provides us with a slightly more convenient API to describe web apps. It provides us with API for declaring methods and routes, extract variable information, compose routes, and create responses with less boilerplate.

A twain app is generally constructed by listing several HTTP methods + routes to be tried in order, and a matching responders for each method+route.

Let's explore these steps one by one, starting with declaring methods and routes, then defining responders, and finally gluing them all together.

Routing

Two of the most important details that can be found in an HTTP request is which component does the user want to access and in what way. The first is described using a Path, and the second using a Method.

For example, if a user would like to view the bulletin board post number 13, they will send the HTTP request GET /post/13. The first part is the method, and the second is the path.

To construct a route in twain, we need to specify the method using one of the method functions (such as get), and apply it the route and an action to generate a response.

Twain provides a textual interface for describing routes using the GHC extension OverloadedStrings. For example, we can describe static paths such as /static/css/style.css by writing the string "/static/css/style.css".

When writing routes, we often want to describe more than just a static path, sometimes we want part of the path to vary. We can give a name to a variable part of the path by prefixing the name with a colon (:).

For example, "/post/:id" will match with /post/17, /post/123, /post/hello and so on, and later, when we construct a response, we will be able to extract to this variable part with the function param by passing it the name "id".

For our bulletin board we want to create several routes:

get "/" -- Our main page, which will display all of the bulletins
get "/post/:id" -- A page for a specific post
get "/new" -- A page for creating a new post
post "/new" -- A request to submit a new page
post "/post/:id/delete" -- A request to delete a specific post

Next, we'll define what to do if we match on each of these routes.

Responding

Once we match an HTTP method and route, we can decide what to do with it. This action is represented by the type ResponderM a.

ResponderM implements the monadic interface, so we can chain such action in the same way we are used to from types like IO, this will run one action after the other.

In ResponderM context, we can find out more details about the request, do IO, decide how to respond to the user, and more.

Querying the Request

The request the user sent often has more information than just the HTTP method and route. It can hold request headers such as which type of content the user is expecting to get or the "user-agent" it uses, in case of the HTTP methods such as POST and PUT it can include a body which includes additional content, and more.

Twain provides a few utility functions to query a few of the more common parts of a request, with functions such as body, header and files. Or the entire request if needed.

It also provides easy access to the varying parts of the route and body with param and params.

For our case this will come into play when we want to know which post to refer to (what is the :id in the /post/:id route), and what is the content of the post (in the /new route).

Responding to the user

There are several ways to respond to the user, the most common ones is to return some kind of data. This can be text, HTML, JSON, a file or more.

In HTTP, in addition to sending the data, we also need to describe what kind of data we are sending and even that the request was successful at all.

Twain handles all that for the common cases by providing utility functions such as text, html, and json.

These functions take the relevant data we want to send to the user and create a WAI Response, which we can then send to the user.

For example, if we want to send a simple html page on the route /hello, we'll write the following

get "/hello" $
  send $
    html "<html><head><link rel=\"stylesheet\" type=\"text/css\" href=\"/style.css\"></head><body>Hello!</body></html>"

The HTTP Reponse we created with html will automatically set the status code 200, and the Content-Type which is appropriate for HTML pages, This is also something that we can set ourselves without help if we like using the raw function by applying it with the status, headers and body directly, instead of calling html. For example:

get "/hello" $
  send $
    raw
      status200
      [("Content-Type", "text/html; charset=utf-8")]
      "<html><head><link rel=\"stylesheet\" type=\"text/css\" href=\"/style.css\"></head><body>Hello!</body></html>"
IO

It is possible to use IO operations in a ResponderM context using the function liftIO. For example:

get "/hello" $ do
  liftIO (putStrLn "They said hello!")
  send $ text "Hello back!"

This way we can write to console, change a song in our music player, or query a database in the middle of processing a request! Fair warning though: Warp runs request processing concurrently, so make sure you avoid race conditions in your code!

Gluing routes together

Each example we have seen above has the type Middleware, which is defined like this:

type Middleware = Application -> Application

As a reminder, Application is also a type alias:

type Application
  = Request -> (Response -> IO ResponseReceived) -> IO ResponseReceived

In essence, a Middleware is a function that takes a WAI Application and can add additional processing to it - it can process the user Request before passing it to the Application it received, and it can do extra processing to the Response the Application generates before calling the Response -> IO ResponseReceived it received.

To illustrate, here's a very simple Middleware that prints some data from a Request before passing it to the web app, And prints some data from the web app's Response before sending it to the user:

mylogger :: Twain.Middleware
mylogger app request respond = do
  print (Twain.requestMethod request)
  app request $ \response ->
    print (Twain.responseStatus response)
    respond response

Twain uses this to mechanism to compose route handlers as well: a route handler is essentially a function that checks the request first and decides whether it wants to handle it (if the route matches) or pass it to the next route handler. So we can compose route handlers using regular function composition!

All that's missing is that final route handler of type Application that will definitely handle all requests that were not processed by previous handlers request. We can use twain's notFound function to send the user a failure message in no other route handler was able to handle their request.

Here's an example of a simple WAI Application with several route handlers:

{-# language OverloadedStrings #-}

import Web.Twain
import Network.Wai.Handler.Warp (run)

main :: IO ()
main = do
  putStrLn "Server running at http://localhost:3000 (ctrl-c to quit)"
  run 3000 app

app :: Application
app =
  ( get "/" (send (text "hello"))
  . get "/echo/hi" (send (text "hi there"))
  . get "/echo/:str" (param "str" >>= \str -> send (text str))
  )
  (notFound (send (text "Error: not found.")))

Note that the order of the routes matters - we try to match the /echo/hi route before the /echo/:str route and provide a custom handler to a specific case, all other cases will be caught by the more general route handler.

And as an aside, I personally don't like to use that many parenthesis and find using $ a bit more aesthetically pleasing, but . has presedence over $ so it's not going to work so well here. Fortunately we can place the routes in a list and then fold over the list to compose them instead:

app :: Application
app =
  foldr ($)
    (notFound $ send $ text "Error: not found.")
    [ get "/" $
      send $ text "hello"

    , get "/echo/hi" $
      send $ text "hi there"

    , get "/echo/:str" $ do
      str <- param "str"
      send $ text str
    ]

I like this style a bit more!

Alright, enough chitchat - let's get to work

We now have the basic building blocks with which we can build our bulletin board! There are a few more things we can cover that will make our lives easier, but we'll pick them up as we go.

At the time of writing the most recent version of twain is 2.1.0.0.

Some simple structure

Here's the simple initial structure which we will iterate on to build our bulletin board app:

{-# language OverloadedStrings #-}

-- | A bulletin board app built with twain.
module Bulletin where

import qualified Web.Twain as Twain
import Network.Wai.Handler.Warp (run, Port)

-- | Entry point. Starts a bulletin-board server at port 3000.
main :: IO ()
main = runServer 3000

-- | Run a bulletin-board server at at specific port.
runServer :: Port -> IO ()
runServer port = do
  putStrLn $ unwords
    [ "Running bulletin board app at"
    , "http://localhost:" <> show port
    , "(ctrl-c to quit)"
    ]
  run port mkApp

-- | Bulletin board application description.
mkApp :: Twain.Application
mkApp =
  foldr ($)
    (Twain.notFound $ Twain.send $ Twain.text "Error: not found.")
    routes

-- | Bulletin board routing.
routes :: [Twain.Middleware]
routes =
  -- Our main page, which will display all of the bulletins
  [ Twain.get "/" $
    Twain.send $ Twain.text "not yet implemented"

  -- A page for a specific post
  , Twain.get "/post/:id" $
    Twain.send $ Twain.text "not yet implemented"

  -- A page for creating a new post
  , Twain.get "/new" $
    Twain.send $ Twain.text "not yet implemented"

  -- A request to submit a new page
  , Twain.post "/new" $
    Twain.send $ Twain.text "not yet implemented"

  -- A request to delete a specific post
  , Twain.post "/post/:id/delete" $
    Twain.send $ Twain.text "not yet implemented"
  ]

We'll start with a very simple routing skeleton. For the sake of simplicity, I'm going to put this code in main.hs and run it using:

stack runghc --package twain-2.1.0.0 --package warp main.hs

Eventually the program will greet us with the following output:

Running bulletin board app at http://localhost:3000 (ctrl-c to quit)

Which means that we can now open firefox a go to http://localhost:3000 and be greeted by our twain application.

  • I've also create a complete cabal project if you'd prefer to use that instead: see the commit

Displaying posts

Next, we are going to need figure out how to represent our bulletin data and how to keep state around.

We are going to add a few new packages to use for our data representation: text, time, and containers.

Add above:

import qualified Data.Text as T
import qualified Data.Time.Clock as C
import qualified Data.Map as M

And we'll represent a post in the following way:

-- | A description of a bulletin board post.
data Post
  = Post
    { pTime :: C.UTCTime
    , pAuthor :: T.Text
    , pTitle :: T.Text
    , pContent :: T.Text
    }

And we'll use a Map to represent all of the posts:

-- | A mapping from a post id to a post.
type Posts = M.Map Integer Post

Once we have these types, we can thread a value of type Posts to routes, so they will be available to all requests and response handlers. We'll change runServer and app a bit and add some dummy data.

-- | Run a bulletin-board server at at specific port.
runServer :: Port -> IO ()
runServer port = do
  app <- mkApp
  putStrLn $ unwords
    [ "Running bulletin board app at"
    , "http://localhost:" <> show port
    , "(ctrl-c to quit)"
    ]
  run port app

-- ** Application and routing

-- | Bulletin board application description.
mkApp :: IO Twain.Application
mkApp = do
  dummyPosts <- makeDummyPosts
  pure $ foldr ($)
    (Twain.notFound $ Twain.send $ Twain.text "Error: not found.")
    (routes dummyPosts)

-- | Bulletin board routing.
routes :: Posts -> [Twain.Middleware]
routes posts =
  -- Our main page, which will display all of the bulletins
  [ Twain.get "/" $
    Twain.send (displayAllPosts posts)

  -- A page for a specific post
  , Twain.get "/post/:id" $ do
    pid <- Twain.param "id"
    Twain.send (displayPost pid posts)

  -- A page for creating a new post
  , Twain.get "/new" $
    Twain.send $ Twain.text "not yet implemented"

  -- A request to submit a new page
  , Twain.post "/new" $
    Twain.send $ Twain.text "not yet implemented"

  -- A request to delete a specific post
  , Twain.post "/post/:id/delete" $
    Twain.send $ Twain.text "not yet implemented"
  ]

And add some additional business logic to display posts as simple text for now:

-- ** Business logic

-- | Respond with a list of all posts
displayAllPosts :: Posts -> Twain.Response
displayAllPosts =
  Twain.text . T.unlines . map ppPost . M.elems

-- | Respond with a specific post or return 404
displayPost :: Integer -> Posts -> Twain.Response
displayPost pid posts =
  case M.lookup pid posts of
    Just post ->
      Twain.text (ppPost post)

    Nothing ->
      Twain.raw
        Twain.status404
        [("Content-Type", "text/plain; charset=utf-8")]
        "404 Not found."

And add code that define the types, creates a dummy posts list, and implements ppPost which converts a Post to text:

-- ** Posts

-- | A mapping from a post id to a post.
type Posts = M.Map Integer Post

-- | A description of a bulletin board post.
data Post
  = Post
    { pTime :: C.UTCTime
    , pAuthor :: T.Text
    , pTitle :: T.Text
    , pContent :: T.Text
    }

-- | Create an initial posts Map with a dummy post
makeDummyPosts :: IO Posts
makeDummyPosts = do
  time <- C.getCurrentTime
  pure $
    M.singleton
      0
      ( Post
        { pTime = time
        , pTitle = "Dummy title"
        , pAuthor = "Dummy author"
        , pContent = "bla bla bla..."
        }
      )

-- | Prettyprint a post to text
ppPost :: Post -> T.Text
ppPost post =
  let
    header =
      T.unwords
        [ "[" <> T.pack (show (pTime post)) <> "]"
        , pTitle post
        , "by"
        , pAuthor post
        ]
    seperator =
      T.replicate (T.length header) "-"
  in
    T.unlines
      [ seperator
      , header
      , seperator
      , pContent post
      , seperator
      ]

Now, when running our program with:

stack runghc --package twain-2.1.0.0 --package warp --package text --package containers main.hs

We should be able to see a post when going to http://localhost:3000, see the same post when going to http://localhost:3000/post/0, and see a not found message when trying to go to a post with a different id such as http://localhost:3000/post/17

We can also create HTTP requests and see the results from the command-line using curl:

To see all posts:

curl -X GET http://localhost:3000

To see the post with id 0:

curl -X GET http://localhost:3000/post/0

Managing mutable state

Now this is a good start but we are still missing a few important parts:

  • Adding new posts
  • Generating new distinct post ids on post creation
  • Making sure all threads access the same state without stepping on each other's toes

While we could use a mutable variable like IORef or MVar, writing code that can run a sequence of commands that use mutable data can be tricky.

For example one thing we want to do is, when creating a new post:

  1. Get the current id
  2. Increment it, use that id to create a new post
  3. update the mutable variable to point to the new Map

However, if, for example, two threads manage to get the same id before incrementing the id, we'll get two posts with the same id. Or if two threads create the new Map and ask the mutable variable to point at their new Map, one post will be not actually be added and will be lost forever.

To combat that, we'll use shared memory using Software Transactional Memory (in short, STM). The stm packages provides us with mutable variables that can be shared and updated concurrently in an atomic way. Meaning that we can describe a sequence of operations on shared memory that are guaranteed to run atomically as one transaction without other operations on the same mutable variables getting mixed in between.

I recommend reading the chapter of STM in PCPH to get a more in-depth overview of stm.

Now - we can create a state data type the with contain the posts currently existing in the system as well as a updating new id for the next post added to the system:

-- | Application state.
data AppState
  = AppState
    { asNextId :: Integer -- ^ The id for the next post
    , asPosts :: Posts -- ^ All posts
    }

And then wrap it up in a transaction mutable variable: STM.TVar AppState.

We can create a new TVar in an IO context and pass it to routes so that the twain web app is a closure containing the mutable variable, and that way any thread handling requests and responses will have access to it!

We'll add a new import:

import qualified Control.Concurrent.STM as STM

And we'll edit mkApp to create the TVar and pass it to routes:

mkApp :: IO Application
mkApp = do
  dummyPosts <- makeDummyPosts
  appstateVar <- STM.newTVarIO AppState{asNextId = 1, asPosts = dummyPosts}
  pure $ foldr ($)
    (Twain.notFound $ Twain.send $ Twain.text "Error: not found.")
    (routes appstateVar)

routes :: STM.TVar AppState -> [Middleware]
routes appstateVar = do
  ...

The three most interesting functions we have (for now) to operate on our mutable transactional variable appstateVar are:

readTVar   :: TVar a -> STM a
writeTVar  :: TVar a -> a -> STM ()

atomically :: STM a -> IO a

the STM type we see here is similar to IO, it is a description of a transactional program - a sequence of steps that must run atomically. And the atomically function is one that converts that program into something that the Haskell runtime system can run in IO context.

So now, creating a new post and adding it to the current state of the system looks like this:

-- | Add a new post to our store.
newPost :: Post -> STM.TVar AppState -> IO Integer
newPost post appstateVar =
  STM.atomically $ do
    appstate <- STM.readTVar appstateVar
    STM.writeTVar
      appstateVar
      ( appstate
        { asNextId = asNextId appstate + 1
        , asPosts = M.insert (asNextId appstate) post (asPosts appstate)
        }
      )
    pure (asNextId appstate)

And these operations are guaranteed to run atomically. (We can also use STM.modifyTVar :: TVar a -> (a -> a) -> STM () for a slightly more convenient code.)

Let's add another import so we can run IO actions inside ResponderM:

import Control.Monad.IO.Class (liftIO)

and change the code of routes to handle viewing posts from our store:

-- | Bulletin board routing.
routes :: STM.TVar AppState -> [Twain.Middleware]
routes appstateVar =
  -- Our main page, which will display all of the bulletins
  [ Twain.get "/" $ do
    posts <- liftIO $ asPosts <$> STM.readTVarIO appstateVar
    Twain.send (displayAllPosts posts)

  -- A page for a specific post
  , Twain.get "/post/:id" $ do
    pid <- Twain.param "id"
    posts <- liftIO $ asPosts <$> STM.readTVarIO appstateVar
    Twain.send (displayPost pid posts)

  -- A page for creating a new post
  , Twain.get "/new" $
    Twain.send $ Twain.text "not yet implemented"

  -- A request to submit a new page
  , Twain.post "/new" $
    Twain.send $ Twain.text "not yet implemented"

  -- A request to delete a specific post
  , Twain.post "/post/:id/delete" $
    Twain.send $ Twain.text "not yet implemented"
  ]

Note how we can run IO operations inside a ResponderM context using liftIO.

Let's also add the ability to delete posts:

routes :: STM.TVar AppState -> [Twain.Middleware]
routes appstateVar =
  [ ...

  -- A request to delete a specific post
  , Twain.post "/post/:id/delete" $ do
    pid <- Twain.param "id"
    response <- liftIO $ handleDeletePost pid appstateVar
    Twain.send response
  ]

-- | Delete a post and respond to the user.
handleDeletePost :: Integer -> STM.TVar AppState -> IO Twain.Response
handleDeletePost pid appstateVar = do
  found <- deletePost pid appstateVar
  pure $
    if found
      then
        Twain.redirect302 "/"

      else
        Twain.raw
          Twain.status404
          [("Content-Type", "text/html; charset=utf-8")]
          "404 Not Found."

-- | Delete a post from the store.
deletePost :: Integer -> STM.TVar AppState -> IO Bool
deletePost pid appstateVar =
  STM.atomically $ do
    appstate <- STM.readTVar appstateVar
    case M.lookup pid (asPosts appstate) of
      Just{} -> do
        STM.writeTVar
          appstateVar
          ( appstate
            { asPosts = M.delete pid (asPosts appstate)
            }
          )
        pure True

      Nothing ->
        pure False

We can also test POST requests from the command-line using curl:

To delete the post with id 0:

curl -X POST http://localhost:3000/post/0/delete

HTML and forms

We're going to start writing some HTML to display our data and add a form for adding a new post.

We're going to use lucid. If you are interested in more possible choices for html libraries vrom911's article about html libraries is a good place to start.

Lucid provides a monadic EDSL for writing html pages. The functions are all suffixed with underscore (_) and represent the relevant html tags.

We'll add this import at the top:

import qualified Lucid as H

And the following type for convenience:

type Html = H.Html ()

And first, we'll create a template boilerplate which into we'll inject our content later:

-- | HTML boilerplate template
template :: T.Text -> Html -> Html
template title content =
  H.doctypehtml_ $ do
    H.head_ $ do
      H.meta_ [ H.charset_ "utf-8" ]
      H.title_ (H.toHtml title)
      H.link_ [ H.rel_ "stylesheet", H.type_ "text/css", H.href_ "/style.css"  ]
    H.body_ $ do
      H.div_ [ H.class_ "main" ] $ do
        H.h1_ [ H.class_ "logo" ] $
          H.a_ [H.href_ "/"] "Bulletin Board"
        content

Notice how the lists represent the attributes of a tag, how tags are sequenced using the monadic interface, and how tags are nested by passing them as input to other tags.

Let's create pages for posts:

-- | All posts page.
allPostsHtml :: Posts -> Html
allPostsHtml posts = do
  H.p_ [ H.class_ "new-button" ] $
    H.a_ [H.href_ "/new"] "New Post"
  mapM_ (uncurry postHtml) $ reverse $ M.toList posts

postHtml :: Integer -> Post -> Html
postHtml pid post = do
  H.div_ [ H.class_ "post" ] $ do
    H.div_ [ H.class_ "post-header" ] $ do
      H.h2_ [ H.class_ "post-title" ] $
        H.a_
          [H.href_ ("/post/" <> T.pack (show pid))]
          (H.toHtml $ pTitle post)

      H.span_ $ do
        H.p_ [ H.class_ "post-time" ] $ H.toHtml (T.pack (show (pTime post)))
        H.p_ [ H.class_ "post-author" ] $ H.toHtml (pAuthor post)

    H.div_ [H.class_ "post-content"] $ do
      H.toHtml (pContent post)

And change our web handlers to use html instead of text:

 -- | Respond with a list of all posts
 displayAllPosts :: Posts -> Twain.Response
 displayAllPosts =
-  Twain.text . T.unlines . map ppPost . M.elems
+  Twain.html . H.renderBS . template "Bulletin board - posts" . allPostsHtml

 -- | Respond with a specific post or return 404
 displayPost :: Integer -> Posts -> Twain.Response
 displayPost pid posts =
   case M.lookup pid posts of
     Just post ->
-      Twain.text (ppPost post)
+      Twain.html $
+        H.renderBS $
+          template "Bulletin board - posts" $
+            postHtml pid post

     Nothing ->
       Twain.raw
         Twain.status404
         [("Content-Type", "text/plain; charset=utf-8")]
         "404 Not found."

In order to delete a post, we need to make a POST command to the URL /post/<post-id>/delete. We can do that using HTML by creating a form, defining its URL and method, and create an input HTML element of type submit.

    -- delete button
    H.form_
      [ H.method_ "post"
      , H.action_ ("/post/" <> T.pack (show pid) <> "/delete")
      , H.onsubmit_ "return confirm('Are you sure?')"
      , H.class_ "delete-post"
      ]
      ( do
        H.input_ [H.type_ "submit", H.value_ "Delete", H.class_ "deletebtn"]
      )

You can stick this wherever you want in postHtml, I placed it at the end. Now, if you run the program using:

stack runghc --package twain --package text --package containers --package stm --package lucid main.hs

and go to the website (http://localhost:3000), you'll be greeted with beautiful (well, not beautiful, but functional) posts and a delete button for each post.

Submitting data via forms and processing it

Next we are going to add a post. To do that we need to create a new HTML page which will contain another HTML form. This time we will want to capture some input which will then be part of the body of the POST request.

-- | A new post form.
newPostHtml :: Html
newPostHtml = do
  H.form_
    [ H.method_ "post"
    , H.action_ "/new"
    , H.class_ "new-post"
    ]
    ( do
      H.p_ $ H.input_ [H.type_ "text", H.name_ "title", H.placeholder_ "Title..."]
      H.p_ $ H.input_ [H.type_ "text", H.name_ "author", H.placeholder_ "Author..."]
      H.p_ $ H.textarea_ [H.name_ "content", H.placeholder_ "Content..."] ""
      H.p_ $ H.input_ [H.type_ "submit", H.value_ "Submit", H.class_ "submit-button"]
    )

And we need to be able to access the following from the request on the server. We can do that using param. So let's implement the relevant parts in routes:

  -- A page for creating a new post
  , Twain.get "/new" $
    Twain.send handleGetNewPost

  -- A request to submit a new page
  , Twain.post "/new" $ do
    title <- Twain.param "title"
    author <- Twain.param "author"
    content <- Twain.param "content"
    time <- liftIO C.getCurrentTime

    response <-
      liftIO $ handlePostNewPost
        ( Post
          { pTitle = title
          , pAuthor = author
          , pContent = content
          , pTime = time
          }
        )
        appstateVar

    Twain.send response

and the handlers:

-- | Respond with the new post page.
handleGetNewPost :: Twain.Response
handleGetNewPost =
  Twain.html $
    H.renderBS $
      template "Bulletin board - posts" $
        newPostHtml

-- | Respond with the new post page.
handlePostNewPost :: Post -> STM.TVar AppState -> IO Twain.Response
handlePostNewPost post appstateVar = do
  pid <- newPost post appstateVar
  pure $ Twain.redirect302 ("/post/" <> T.pack (show pid))

And now we have a fairly functional little bulletin board! Hooray!

Styling

This post is already pretty long, so I will not cover styling in depth.

There are multiple way to use styling:

The first is using the EDSL approach like we did with lucid using a library like clay, the second is to write the css text inline in a Haskell module using something like the raw-strings-qq library, another is to write it in an external file and embed to context at compile time using template haskell and the file-embed library, another is to ship the css file along with the executable and use responseFile from the wai package to send it as a file.

For each of these - don't forget to set the content type header to "text/css; charset=utf-8"!

We can send a very rudimentary CSS as a string with the css function by adding this to the end of the routes list:

  -- css styling
  , Twain.get "/style.css" $
    Twain.send $ Twain.css ".main { width: 900px; margin: auto; }"

Logging, Sessions, Cookies, Authentication, etc.

The wai ecosystem has a wide variety of features that can be composed together. These features are usually encapsulated as "middlewares".

Remember, a middleware is a function that takes an Application and returns an Application. Middlewares can add functionality before the request passes to our twain app or after the response.

The wai-extra packages contains a bunch of middlewares we can use. Like logging, gzip compression of responses, forcing ssl usage, or simple http authentication.

For example, let's add some logging from wai-extra to our bulletin-app. We import a request logger from wai-extra:

import qualified Network.Wai.Middleware.RequestLogger as Logger

And then we can apply our twain app to a function such as logStdoutDev to add request logging to our twain app:

 -- | Run a bulletin-board server at at specific port.
 runServer :: Port -> IO ()
 runServer port = do
   app <- mkApp
   putStrLn $ unwords
     [ "Running bulletin board app at"
     , "http://localhost:" <> show port
     , "(ctrl-c to quit)"
     ]
-  run port app
+  run port (Logger.logStdoutDev app)

Testing

Testing WAI apps can be relatively straightforward with packages such as hspec-wai. Check out this twain test module for example usage.

Deploying

I usually create a static executable using ghc-musl and docker so I can deploy my executable on other linux servers.

In a stack project, add the following sections:

Add this to the stack.yaml:

docker:
  enable: true
  image: utdemir/ghc-musl:v24-ghc922

and this to the .cabal file under the executable section:

  ghc-options: -static -optl-static -optl-pthread -fPIC -threaded -rtsopts -with-rtsopts=-N

Check the ghc-musl repo for more instructions.

That's it

I hope you found this tutorial useful. If there's something you feel I did not explain well or you'd like me to cover, let me know via email, twitter.

The whole program including the stack and cabal files can be found on Github.

April 24, 2022 12:00 AM

April 12, 2022

Well-Typed.Com

GHC activities report: February-March 2022

This is the eleventh edition of our GHC activities report, which describes the work on GHC and related projects that we are doing at Well-Typed. The current edition covers roughly the months of February and March 2022.

You can find the previous editions collected under the ghc-activities-report tag.

A bit of background: One aspect of our work at Well-Typed is to support GHC and the Haskell core infrastructure. Several companies, including IOHK, Meta, and GitHub via the Haskell Foundation, are providing us with funding to do this work. We are also working with Hasura on better debugging tools. We are very grateful on behalf of the whole Haskell community for the support these companies provide.

If you are interested in also contributing funding to ensure we can continue or even scale up this kind of work, please get in touch.

Of course, GHC is a large community effort, and Well-Typed’s contributions are just a small part of this. This report does not aim to give an exhaustive picture of all GHC work that is ongoing, and there are many fantastic features currently being worked on that are omitted here simply because none of us are currently involved in them in any way. Furthermore, the aspects we do mention are still the work of many people. In many cases, we have just been helping with the last few steps of integration. We are immensely grateful to everyone contributing to GHC. Please keep doing so (or start)!

Team

The current GHC team consists of Ben Gamari, Andreas Klebinger, Matthew Pickering, Zubin Duggal and Sam Derbyshire.

Many others within Well-Typed, including Adam Gundry, Alfredo Di Napoli, Alp Mestanogullari, Douglas Wilson and Oleg Grenrus, are contributing to GHC more occasionally.

Releases

  • Ben finished backports to GHC 9.2.2 and cut the release.

  • Matt worked on preparing the 9.4 release.

  • Zubin has started preparing the 9.2.3 release.

Typechecker

  • Sam has been implementing syntactic unification, which allows two types to be checked for equality syntactically. This is useful in several places in the typechecker when we don’t want to emit an equality constraint to be processed by the constraint solver (thus giving less work to the constraint solver). This work will allow us to progress towards fixing #13105 (allowing rewriting in RuntimeReps). (!7812)

  • Sam fixed the implementation of isLiftedType_maybe, an internal function in GHC used to determine whether something is definitely lifted (e.g. Int :: Type), definitely unlifted (e.g. Int# :: TYPE IntRep), or unknown (e.g. a :: TYPE r for a type variable r). This function did not correctly account for type families or levity variables, as noted in #20837. This was hiding several bugs, e.g. in strictness analysis and in pattern matching inhabitation tests.

  • Sam allowed HasField constraints to appear in quantified constraints (#20989).

  • Sam added a check preventing users to derive KnownNat instances, which could be used to cause segfaults as shown in #21087.

  • Sam made the output of GHCi’s :type command more user-friendly, by improving instantiation of types involving out-of-order inferred type variables (#21088) and skipping normalisation for types that aren’t fully instantiated (#20974).

Code generation

  • Ben reworked the x86-64 native code generator to produce more position-independent code where possible. This enables use of GHC with new Windows toolchains, which enable address-space layout randomization by default (#16780).

  • Ben fixed a slew of bugs in GHC’s code generation for unaligned array accesses, revealed by recent work on the bytestring package (#20987, #21015).

  • Ben characterised and fixed a rather tricky bug in the generation of static reference tables for programs containing cyclic binding groups containing CAFs, static functions, and static data constructor applications (#20959).

  • Ben debugged fixed a recently introduced regression where GHC miscompiled code involving jump tables with sub-word discriminants (#21186).

  • Ben debugged a tricky non-deterministic crash due to a set of missing GC roots (#21141).

  • Spurred by insights from #21141, Ben started investigating how we can reduce the impact of error paths on SRT sizes. Sadly, there are some tricky challenges in this area which will require further work (#21169, #21183)

  • Ben migrated GHC’s Windows distribution towards a fully Clang/LLVM-based toolchain, eliminating a good number of bugs attributable to the previous GNU toolchain. This was a major undertaking involving changes in code generation (!7449), linking (!7774, !7528), the RTS (!7511, !7512, !7446), Cabal (Cabal #8062), the driver (!7448), and packaging and should significantly improve GHC’s maintainability and reliability on Windows platforms. See !7448 for a full overview of all of the moving parts involved in this migration.

  • Sam fixed a bug with code generation of keepAlive#, which was incorrectly being eta reduced even though it is supposed to always be kept eta-expanded (#21090).

Core

  • Sam added a check that prevents unboxed float literals from occurring in patterns, to avoid case expressions needing to implement complicated floating-point equality rules. This didn’t affect any packages on head.hackage.

Runtime system

  • Ben refactored the handling of adjustor thunks, an implementation detail of GHC’s foreign function interface implementation. The new representation significantly reduces their size and contribution to address-space fragmentation, fixing eliminating a known memory leak in GHCi (#20349) and fixing a source of testsuite fragility on Windows and i386 (#21132).

  • With help from Matt, Ben at long last finished and merged his refactoring of GHC’s eventlog initialization logic, eliminating a measurable source of RTS startup overhead and removing the last barrier to enabling eventlog support by default (!4477).

  • Ben rewrote the RTS linker used on Windows platforms, greatly improving link robustness by extending and employing the RTS’s m32 allocator for mapping object code (!7447).

  • Ben identified a GC bug, revealed by recent improvements in pointer tagging consistency, where the GC failed to untag function closures referenced from PAPs (#21254).

  • Matt identified and fixed a discrepency in the runtime stats calculations which would lead to incorrect CPU time calculations when using multiple GC threads. (!7890)

Error messages

  • Sam migrated more error messages to use the diagnostic infrastructure, such as “Missing signature” errors, and illegal wildcard errors (!7033).

  • Sam improved the treatment of promotion ticks on symbolic operators, which means that GHC now correctly report unticked promoted symbolic constructors when compiling with -Wunticked-promoted-constructors (#19984).

  • Zubin added warnings that get triggered when file header pragmas like LANGUAGE pragmas are found in the body of the module where they would usually be ignored (#20385).

Parser

  • Sam allowed COMPLETE pragmas involving qualified constructor names to be parsed correctly (!7645).

Driver

  • Ben rebased and extended work by Tamar Christina to improve GHC’s support for linking against C++ libraries, addressing pains revealed by the text libraries recent addition of a dependency on simdutf (#20010).

  • Ben introduced response file support into GHC’s command-line parser, making it possible to circumvent the restrictive limits on command-line-length imposed by some platforms (#16476).

  • Zubin and Matt added more fine grained recompilation checking for modules using Template Haskell, so that they are only recompiled when a dependency actually used in a splice is changed (#20605, !7353, blog post).

  • Matt fixed some more bugs in the dependency calculations in the driver. In particular some situations involving redundant hs-boot files are now handled correctly.

  • Matt modified the driver to store a cached transitive dependency calculation which can share work of computing a transitive dependency across modules. This is used when computing what instances are in scope for example.

  • Matt once again improved the output of the -Wunused-packages warning to now display some more information about the redundant package imports (!7883).

  • Matt fixed a long-standing bug where in one-shot mode, the compiler would look for interface files in the -i dirs even if the -hidir was set (!7851).

API features

  • Zubin fixed a few bugs with HIE file support, one where certain variable scopes weren’t being calculated properly (#18425) and another one where relationships between derived typeclass instances weren’t being recorded (#20341).

  • Matt and Zubin finished the hi-haddock patch which makes GHC lex and rename Haddock documentation strings. These are then stored in interface files so downstream tools such as HLS and GHCi can directly read this information without having to process the doc strings themselves.

Template Haskell

  • Sam fixed a compiler panic triggered by illegal occurrences of type wildcards resulting from splicing in a Template Haskell type (#15433).

  • Zubin fixed a few issues with the pretty printing of TH syntax (#20868, #20842).

  • Zubin added support for quoting patterns containing negative numeric literals (#20711).

Profiling

  • Andreas added a new profiling mode: -fprof-late. This mode will add cost centres only after the simplifier had a chance to run resulting in profiling performance more in line with regular builds as profiling in this mode will allow most optimizations to fire even with profiling enabled. The user guide has more information and we encourage people to try it out!

  • Andreas and Matt made various changes to the ticky-profiling infrastructure which allow it to be used with the eventlog, and furthermore to use it with eventlog2html. Thanks to Hasura for funding this work!

  • Matt made a few improvements to Source Notes, which are used to give source locations to expressions. This should result in more accurate location information in more situations (!7536).

Libraries

  • Ben fixed a regression in the process library due to the recently-introduced support for posix_spawnp (process #224).

  • Andreas opened up a proposal to export MutableByteArray from Data.Array.Byte. This makes the treatment of MutableByteArray consistent with the treatment of ByteArray in regards to being exported by base. The proposal has been implemented, accepted and will be in ghc-9.4.

Compiler performance

  • Sam continued work on directed coercions, which avoids large coercions being produced when rewriting type families (#8095). Unfortunately, benchmarking revealed some severe regressions, such as when compiling the singletons package. This is due to coercion optimisation being less effective than it was previously. Sam implemented a workaround (coercion zapping in the coercion optimiser), but due to the complexity of the patch, it was decided it would be better to investigate zapping without directed coercions, as the implementation of the directed coercions patch suggested a way forward that could avoid previous pitfalls. The directed coercions patch has been put on hold for the time being, until a better approach can be found for coercion optimisation. Sam wrote up an overview of the difficulties encountered during the implementation on the GHC wiki here, which should be useful to future implementors.

Packaging

  • All the release bindists are now produced by Hadrian. Starting from the 9.4 release, all the bindists that we distribute will be built in this manner.

  • Zubin and Matt finished the reinstallable GHC patch, which allows GHC and all its libraries to be built using normal cabal-install commands. In the future it’s hoped that this will allow ghc to be rebuilt in cabal build plans but we still have some issues to work out to do with Template Haskell (#20742).

Runtime performance

  • After a long time tag inference has finally landed in !5614 which implements #16970.

    This is a new optimization which can allow the compiler to omit branches checking for the presence of a pointer tag if we can infer from the context that a tag must always be present.

    For the nofib benchmark suite benchmarks the best result was obtained when -fworker-wrapper-cbv was enabled resulting in a 4.01% decrease in instructions executed, with a similar benefit in runtime. Sadly because of #20364 -fworker-wrapper-cbv can prevent RULEs from firing when INLINE[ABLE] pragmas are not used correctly. Which turned out to be a fairly common problem!

    For this reason -fworker-wrapper-cbv is off by default at which point the improvement was “only” by a 1.53% reduction in instructions executed. But in general -fworker-wrapper-cbv can be safely enabled for all modules except these which define RULE relevant functions which are currently not subject to a W/W split. The user guide has some guidance about when -fworker-wrapper-cbv can be safely enabled.

    The main goal of this optimization was to improve tight loops like the ones performed by the lookup operations in containers. There a significant amount of performance was lost to redundant checks for pointer tags and we saw an improvement of runtime by up to 15% for some of the lookup operations.

Infrastructure

  • Ben and Matt introduced a lint to verify references between GHC’s long-form comments (so-called “Notes”). This helps eliminate a long-standing problem where note references grow stale across refactorings of the compiler (!7482).

  • Sam migrated the linting infrastructure to allow all linting steps to be run locally, so that developers can be confident their merge requests won’t fail during the linting stage in CI (!7578).

  • Matt created a script which generates the main CI pipelines for all supported build combinations. The script is a simple Haskell file which is easier to understand and modify than the old gitlab yaml file which led to various inconsistencies between the build configurations and bugs such as missing build artifacts (!7753).

by ben, andreask, matthew, zubin, sam, adam at April 12, 2022 12:00 AM

April 07, 2022

Philip Wadler

Vote!


It's time once again! The below is copied from https://www.gov.uk/register-to-vote. The site is easy to use and registration takes less than five minutes. I'll be voting for the Scottish Green Party.

Deadline for registering to vote in the 5 May 2022 elections

Register by 11:59pm on 14 April to vote in the following elections on 5 May:

  • local government, combined authority mayoral, mayoral and parish council elections in England
  • local government and community council elections in Wales
  • Northern Ireland Assembly election

Register by 11:59pm on 18 April to vote in the local government elections in Scotland on 5 May.

Who can register

You must be aged 16 or over (or 14 or over in Scotland and Wales).

You must also be one of the following:

  • a British citizen
  • an Irish or EU citizen living in the UK
  • a Commonwealth citizen who has permission to enter or stay in the UK, or who does not need permission
  • a citizen of another country living in Scotland or Wales who has permission to enter or stay in the UK, or who does not need permission

Check which elections you’re eligible to vote in.

You can vote when you’re 18 or over. If you live in Scotland or Wales, you can vote in some elections when you’re 16 or over.

You normally only need to register once - not for every election. You’ll need to register again if you’ve changed your name, address or nationality. 

Register online

It usually takes about 5 minutes.

Start now

by Philip Wadler (noreply@blogger.com) at April 07, 2022 05:47 PM

Ken T Takusagawa

[xcruhlyr] first-class pattern guards in Haskell

patterns are not first class objects in Haskell.  they cannot be assigned to variables nor passed around.

in the function f1 below, the patterns Apple and Banana are hardcoded.

data Fruit = Apple | Banana | Orange ;

f1 :: Fruit -> String;
f1 Apple = "got first choice fruit";
f1 Banana = "got second choice fruit";
f1 _ = "did not get what we want";

however, unlike patterns, pattern guards can be first-class objects.  using them, we can accomplish anything a first-class pattern could do.  in the example below, we pass patterns as boolean predicates to f2 and call them in the pattern guards (to the right of the vertical bar).  applep, bananap, and orangep are patterns turned into boolean functions.

f2 :: (Fruit -> Bool) -> (Fruit -> Bool) -> Fruit -> String;
f2 pattern1 pattern2 fruit
| pattern1 fruit = "got first choice fruit" -- note: no semicolon here
| pattern2 fruit = "got second choice fruit";
f2 _ _ _ = "did not get what we want";

applep :: Fruit -> Bool;
applep Apple = True;
applep _ = False;

bananap :: Fruit -> Bool;
bananap Banana = True;
bananap _ = False;

orangep :: Fruit -> Bool;
orangep Orange = True;
orangep _ = False;

examplef2 :: Fruit -> IO();
examplef2 fruit = do {
putStrLn $ f2 applep bananap fruit;
putStrLn $ f2 orangep applep fruit;
};

we can also do pattern matching, encapsulating extraction of a value from a pattern into a (first-class) function returning Maybe.  below, the function superhero calls its supplied pattern and attempts to match against Just.  if the pattern returns Nothing, the guard fails, and we fall through to "muggle".

data Health = Mind Int | Body Int;

superhero :: (Health -> Maybe Int) -> Health -> String;
superhero pattern health | Just power <- pattern health = if power > 9000
  then "is superhero"
  else "not strong enough";
superhero _ _ = "muggle";

getmind :: Health -> Maybe Int;
getmind (Mind i) = Just i;
getmind _ = Nothing;

getbody :: Health -> Maybe Int;
getbody (Body i) = Just i;
getbody _ = Nothing;

powermeter :: Health -> IO();
powermeter health = do {
putStrLn $ superhero getmind health;
putStrLn $ superhero getbody health;
};

in general, because what we pass is a function, we can build up a pattern guard in the myriad of ways we can build a function in a functional programming language.  however, I have not explored this very far.

a pattern guard can have several components, separated by commas.  a comma acts like the boolean AND operator.  each component can be a boolean expression, a pattern match (its boolean value is whether the match succeeds), or an assignment of a local variable with "let" (always evaluates to True).

note well: pattern matching in a guard is different from pattern matching in a let in a guard.  unlike the definition above, the following always succeeds, never falling through to "muggle".  if pattern returns Nothing, then a run-time error "Non-exhaustive patterns" occurs.

superhero pattern health | let { Just power = pattern health } = if power ...

by Unknown (noreply@blogger.com) at April 07, 2022 05:36 AM

Lysxia's blog

The pro-PER meaning of "proper"

A convenient proof tactic is to rewrite expressions using a relation other than equality. Some setup is required to ensure that such a proof step is allowed. One important obligation is to prove Proper theorems for the various functions in our library. For example, a theorem like

Instance Proper_f : Proper ((==) ==> (==)) f.

unfolds to forall x y, x == y -> f x == f y, meaning that f preserves some relation (==), so that we can “rewrite x into y under f�. Such a theorem must be registered as an instance so that the rewrite tactic can find it via type class search.

Where does the word “proper� come from? How does Proper ((==) ==> (==)) f unfold to forall x y, x == y -> f x == f y?

You can certainly unfold the Coq definitions of Proper and ==> and voilà, but it’s probably more fun to tell a proper story.

It’s a story in two parts:

  • Partial equivalence relations
  • Respectfulness

Some of the theorems discussed in this post are formalized in this snippet of Coq.

Partial equivalence relations (PERs)

Partial equivalence relations are equivalence relations that are partial. 🤔

In an equivalence relation, every element is at least related to itself by reflexivity. In a partial equivalence relation, some elements are not related to any element, not even themselves. Formally, we simply drop the reflexivity property: a partial equivalence relation (aka. PER) is a symmetric and transitive relation.

Class PER (R : A -> A -> Prop) :=
  { PER_symmetry : forall x y, R x y -> R y x
  ; PER_transitivity : forall x y z, R x y -> R y z -> R x z }.

We may remark that an equivalence relation is technically a “total� partial equivalence relation.

An equivalent way to think about an equivalence relation on a set is as a partition of that set into equivalence classes, such that elements in the same class are related to each other while elements of different classes are unrelated. Similarly, a PER can be thought of as equivalence classes that only partially cover a set: some elements may belong to no equivalence class.

On the left, a partition of a set of points, representing an equivalence relation.
On the right, a partial partition representing a PER.
On the left, a set of points grouped in three classes. On the right, a set of points grouped in two classes, with some points leftover.

Exercise: define the equivalence classes of a PER; show that they are disjoint.

Solution

The equivalence classes of a PER R : A -> A -> Prop are sets of the form C x = { y ∈ A | R x y }.

Given two equivalence classes C x and C x', we show that these sets are either equal or disjoint. By excluded middle:

  • Either R x x', then R x y -> R x' y by symmetry and transitivity, so y ∈ C x -> y ∈ C x', and the converse by the same argument. Therefore C x = C x'.

  • Or ~ R x x', then we show that ~ (R x y /\ R x' y):

    • assume R x y and R x' y,
    • then R x x' by symmetry and transitivity,
    • by ~ R x x', contradiction.

    Hence, ~ (y ∈ C x /\ y ∈ C x'), therefore C x and C x' are disjoint.

(I wouldn’t recommend trying to formalize this in Coq, because equivalence classes are squarely a set-theoretic concept. We just learn to talk about things differently in type theory.)

A setoid is a set equipped with an equivalence relation. A partial setoid is a set equipped with a PER.

PERs are useful when we have to work in a set that is “too big�. A common example is the set of functions on some setoid. For instance, consider the smallest equivalence relation (≈) on three elements {X, X', Y} such that X ≈ X'. Intuitively, we want to think of X and X' as “the same�, so that the set morally looks like a two-element set.

How many functions {X, X', Y} -> {X, X', Y} are there? If we ignore the equivalence relation, then there are 33 functions. But if we think of {X, X', Y} as a two-element set by identifying X and X', there should be 22 functions. The actual set of functions {X, X', Y} -> {X, X', Y} is “too big�:

  1. it contains some “bad� functions which break the illusion that X and X' are the same, for example by mapping X to X and X' to Y;

    (* A bad function *)
    bad X = X
    bad X' = Y
    bad Y = Y
  2. it contains some “duplicate� functions, for example the constant functions const X and const X' should be considered the same since X ≈ X'.

To tame that set of functions, we equip it with the PER R where R f g if forall x y, x ≈ y -> f x ≈ g y.

Definition R f g : Prop := forall x y, x ≈ y -> f x ≈ g y.

That relation R has the following nice features:

  1. Bad functions are not related to anything: forall f, not (R bad f).

  2. Duplicate functions are related to each other: R (const X) (const X').

Having defined a suitable PER, we now know to ignore the “bad� unrelated elements and to identify elements related to each other. Those remaining “good� elements are called the proper elements.

A proper element x of a relation R is one that is related to itself: R x x.

This is how the Proper class is defined in Coq:

(* In the standard library: From Coq Require Import Morphisms *)
Class Proper {A} (R : A -> A -> Prop) (x : A) : Prop :=
  proper_prf : R x x.

Note that properness is a notion defined for any relation, not only PERs. This story could probably be told more generally. But I think PERs make the motivation more concrete, illustrating how relations let us not only relate elements together, but also weed out badly behaved elements via the notion of properness.

The restriction of a relation R to its proper elements is reflexive. Hence, if R is a PER, its restriction is an equivalence relation. In other words, a PER is really an equivalence relation with an oversized carrier.

Exercise: check that there are only 4 functions {X, X', Y} -> {X, X', Y} if we ignore the non-proper functions and we equate functions related to each other by R.

Solution

The equivalence classes are listed in the following table, one per row, with each sub-row giving the mappings of one function for X, X', Y. There are 4 equivalence classes spanning 15 functions, and 12 “bad� functions that don’t belong to any equivalence classes.

      X  X' Y
------------------
1     X  X  X    1
      X  X  X'   2
      X  X' X    3
      X  X' X'   4
      X' X  X    5
      X' X  X'   6
      X' X' X    7
      X' X' X'   8
------------------
2     X  X  Y    9
      X  X' Y   10
      X' X  Y   11
      X' X' Y   12
------------------
3     Y  Y  X   13
      Y  Y  X'  14
------------------
4     Y  Y  Y   15
------------------
Bad   X  Y  X   16
      X  Y  X'  17
      X' Y  X   18
      X' Y  X'  19
      X  Y  Y   20
      X' Y  Y   21
      Y  X  X   22
      Y  X  X'  23
      Y  X' X   24
      Y  X' X'  25
      Y  X  Y   26
      Y  X' Y   27

Exercise: given a PER R, prove that an element is related to itself by R if and only if it is related to some element.

Theorem Prim_and_Proper {A} (R : A -> A -> Prop) :
  PER R ->
  forall x, (R x x <-> exists y, R x y).

(Solution)

Respectfulness

The relation R defined above for functions {X, X', Y} -> {X, X', Y} is an instance of a general construction. Given two sets D and C, equipped with relations RD : D -> D -> Prop and RC : C -> C -> Prop (not necessarily equivalences or PERs), two functions f, g : D -> C are respectful if they map related elements to related elements. Thus, respectfulness is a relation on functions, D -> C, parameterized by relations on their domain D and codomain C:

(* In the standard library: From Coq Require Import Morphisms *)
Definition respectful {D} (RD : D -> D -> Prop)
                      {C} (RC : C -> C -> Prop)
    (f g : D -> C) : Prop :=
  forall x y, RD x y -> RC (f x) (g y).

(Source)

The respectfulness relation is also cutely denoted using (==>), viewing it as a binary operator on relations.

Notation "f ==> g" := (respectful f g) (right associativity, at level 55)
  : signature_scope.

(Source)

For example, this lets us concisely equip a set of curried functions E -> D -> C with the relation RE ==> RD ==> RC. Respectfulness provides a point-free notation to construct relations on functions.

(RE ==> RD ==> RC) f g
<->
forall s t x y, RE s t -> RD x y -> RC (f s x) (g t y)

Respectfulness on D -> C can be defined for any relations on D and C. Two special cases are notable:

  • If RD and RC are PERs, then RD ==> RC is a PER on D -> C (proof), so this provides a concise definition of extensional equality on functions (This was the case in the example above.)

  • If RD and RC are preorders (reflexive, transitive), then the proper elements of RD ==> RC are exactly the monotone functions.

Proper respectful functions and rewriting

Now consider the proper elements of a respectfulness relation. Recalling the earlier definition of properness, it transforms a (binary) relation into a (unary) predicate:

Proper : (A -> A -> Prop) -> (A -> Prop)

While we defined respectfulness as a binary relation above, we shall also say that a single function f is respectful when it maps related elements to related elements. The following formulations are equivalent; in fact, they are all the same proposition by definition:

forall x y, RD x y -> RC (f x) (f y)
=
respectful RD RC f f
=
(RD ==> RC) f f
=
Proper (RD ==> RC) f

The properness of a function f with respect to the respectfulness relation RD ==> RC is exactly what we need for rewriting. We can view f as a “context� under which we are allowed to rewrite its arguments along the domain’s relation RD, provided that f itself is surrounded by a context that allows rewriting along the codomain’s relation RC. In a proof, the goal may be some proposition in which f x occurs, P (f x), then we may rewrite that goal into P (f y) using an assumption RD x y, provided that Proper (RD ==> RC) f and Proper (RC ==> iff) P, where iff is logical equivalence, with the infix notation <->.

Definition iff (P Q : Prop) : Prop := (P -> Q) /\ (Q -> P).
Notation "P <-> Q" := (iff P Q).

Respectful functions compose:

Proper (RD ==> iff) (fun x => P (f x))
=
forall x y, RD x y -> P (f x) <-> P (f y)

And that, my friends, is the story of how the concept of “properness� relates to the proof technique of generalized rewriting.


Appendix: Pointwise relation

Another general construction of relations on functions is the “pointwise relation�. It only assumes a relation on the codomain RC : C -> C -> Prop. Two functions f, g : D -> C are related pointwise by RC if they map each element to related elements.

(* In the standard library: From Coq Require Import Morphisms *)
(* The domain D is not implicit in the standard library. *)
Definition pointwise_relation {D C} (RC : C -> C -> Prop)
    (f g : D -> C) : Prop :=
  forall x, RC (f x) (g x).

(* Abbreviation (not in the stdlib) *)
Notation pr := pointwise_relation.

(Source)

This is certainly a simpler definition: pointwise_relation RC is equivalent to eq ==> RC, where eq is the standard intensional equality relation.

One useful property is that pointwise_relation RC is an equivalence relation if RC is an equivalence relation. In comparison, we can at most say that RD ==> RC is a PER if RD and RC are equivalence relations. It is not reflexive as soon as RD is bigger than eq (the smallest equivalence relation) and RC is smaller than the total relation fun _ _ => True.

In Coq, the pointwise_relation is also used for rewriting under lambda abstractions. Given a higher-order function f : (E -> F) -> D, we may want to rewrite f (fun z => M z) to f (fun z => N z), using a relation forall z, RF (M z) (N z), where the function bodies M and/or N depend on z so the universal quantification is necessary to bind z in the relation. This can be done using the setoid_rewrite tactic, after having proved a Proper theorem featuring pointwise_relation:

Instance Proper_f : Proper (pointwise_relation RF ==> RD) f.

One disadvantage of pointwise_relation is that it is not compositional. For instance, it is not preserved by function composition:

Definition compose {E D C} (f : D -> C) (g : E -> D) : E -> C :=
  fun x => f (g x).

Theorem not_Proper_compose :
  not
   (forall {E D C}
           (RD : D -> D -> Prop) (RC : C -> C -> Prop),
    Proper (pr RC ==> pr RD ==> pr RC)
           (compose (E := E))).

Instead, at least the first domain of compose should be quotiented by RD ==> RC instead:

Instance Proper_compose {E D C}
    (RD : D -> D -> Prop) (RC : C -> C -> Prop) :
    Proper ((RD ==> RC) ==> pr RD ==> pr RC)
           (compose (E := E)).

We can even use ==> everywhere for a nicer-looking theorem:

Instance Proper_compose' {E D C} (RE : E -> E -> Prop)
    (RD : D -> D -> Prop) (RC : C -> C -> Prop) :
    Proper ((RD ==> RC) ==> (RE ==> RD) ==> (RE ==> RC))
           compose.

Exercise: under what assumptions on relations RD and RC do pointwise_relation RD and RC ==> RD coincide on the set of proper elements of RC ==> RD?

Solution
Theorem pointwise_respectful {D C} (RD : D -> D -> Prop) (RC : C -> C -> Prop)
  : Reflexive RD -> Transitive RC ->
    forall f g, Proper (RD ==> RC) f -> Proper (RD ==> RC) g ->
    pointwise_relation RC f g <-> (RD ==> RC) f g.
(Link to proof)

This table summarizes the above comparison:

pointwise_relation respectful (==>)
is an equivalence yes no
allows rewriting under binders yes no
respected by function composition no yes

Appendix: Parametricity

Respectfulness lets us describe relations RD ==> RC on functions using a notation that imitates the underlying type D -> C. More than a cute coincidence, this turns out to be a key component of Reynolds’s interpretation of types as relations: ==> is the relational interpretation of the function type constructor ->. Building upon that interpretation, we obtain free theorems to harness the power of parametric polymorphism.

Free theorems provide useful properties for all polymorphic functions of a given type, regardless of their implementation. The canonical example is the polymorphic identity type ID := forall A, A -> A. A literal reading of that type is that, well, for every type A we get a function A -> A. But this type tells us something more: A is abstract to the function, it cannot inspect A, so the only possible implementation is really the identity function fun A (x : A) => x. Free theorems formalize that intuition.

The type ID := forall A, A -> A is interpreted as the following relation RID:

Definition RID (f g : forall A, A -> A) : Prop :=
  forall A (RA : A -> A -> Prop), (RA ==> RA) (f A) (g A).

where we translated forall A, to forall A RA, and A -> A to RA ==> RA.

The parametricity theorem says that every typed term t : T denotes a proper element of the corresponding relation RT : T -> T -> Prop, i.e., RT t t holds. “For all t : T, RT t t� is the “free theorem� for the type T.

The free theorem for ID says that any function f : ID satisfies RID f f. Unfold definitions:

RID f f
=
forall A (RA : A -> A -> Prop) x y, RA x y -> RA (f A x) (f A y)

Now let z : A be an arbitrary element of an arbitrary type, and let RA := fun x _ => x = z. Then the free theorem instantiates to

x = z -> f A x = z

Equivalently,

f A z = z

that says exactly that f is extensionally equal to the identity function.


More reading

by Lysxia at April 07, 2022 12:00 AM

April 06, 2022

Well-Typed.Com

large-anon: Practical scalable anonymous records for Haskell

The large-anon library provides support for anonymous records; that is, records that do not have to be declared up-front. For example, used as a plugin along with the record-dot-preprocessor plugin, it makes it possible to write code such as this:

magenta :: Record [ "red" := Double, "green" := Double, "blue" := Double ]
magenta = ANON { red = 1, green = 0, blue = 1 }

reduceRed :: RowHasField "red" r Double => Record r -> Record r
reduceRed c = c{red = c.red * 0.9}

The type signatures are not necessary; type inference works as aspected for these records. If you prefer to use lenses1, that is also possible:

reduceBlue :: RowHasField "blue" r Double => Record r -> Record r
reduceBlue = over #blue (* 0.9)

The library offers a small but very expressive API, and it scales to large records (with 100 fields and beyond), with excellent compilation time performance and good runtime performance. In this blog post we will first present the library from a user’s perspective, then give an overview of the internals with an aim to better to understand the library’s runtime characteristics, and finally show some benchmarks. The library is available from Hackage and is currently compatible with ghc 8.8, 8.10 and 9.0 (extending this to 9.2 should not be too hard).

If you want to follow along, the full source code for all the examples in this blog post can be found in Test.Sanity.BlogPost in the large-anon test suite.

The simple interface

The library offers two interfaces, “simple” and “advanced.” We will present the simple interface first, then explore the advanced interface below.

The simple interface can be summarized as follows:

Data.Record.Anon.Simple

data Record (r :: Row Type)
  deriving (Eq, Ord, Show, Large.Generic, ToJSON, FromJSON)

data Pair a b = a := b
type Row k = [Pair Symbol k]

instance (RowHasField n r a, ..) => HasField n (Record r) a

empty   :: Record '[]
insert  :: Field n -> a -> Record r -> Record (n := a : r)
get     :: RowHasField n r a => Field n -> Record r -> a
set     :: RowHasField n r a => Field n -> a -> Record r -> Record r
project :: SubRow r r' => Record r -> Record r'
inject  :: SubRow r r' => Record r' -> Record r -> Record r
merge   :: Record r -> Record r' -> Record (Merge r r')
where Large.Generic comes from the large-generics package.

In the remainder of this section we will introduce this API by means of examples. When there is possibility for confusion, we will use the prefix S. to refer to the simple interface (and A. for the advanced interface).

Record construction and field access

In the introduction we used some syntactic sugar: the ANON record constructor makes it possible to use regular record syntax for anonymous records. This syntax is available as soon as you use the large-anon plugin. ANON desugars to calls to empty and insert; it does not depend on any kind of internal or unsafe API) and there is no need to use it if you prefer not to (though see Applying pending changes):

purple :: Record [ "red" := Double, "green" := Double, "blue" := Double ]
purple =
     S.insert #red   0.5
   $ S.insert #green 0
   $ S.insert #blue  0.5
   $ S.empty

Similarly, the example in the introduction used RecordDotSyntax as provided by record-dot-preprocessor, but we can also use get and set:

reduceGreen :: RowHasField "green" r Double => Record r -> Record r
reduceGreen c = S.set #green (S.get #green c * 0.9) c

Constraints

The summary of the simple interface showed that Record has a Show instance. Let’s take a closer look at its precise signature:

instance (KnownFields r, AllFields r Show) => Show (Record r)

The KnownFields constraint says that the field names of r must be known, and the AllFields r Show constraint says that all fields of r must in turn satisfy Show; the show instance uses this to output records like this:

> magenta
ANON {red = 1.0, green = 0.0, blue = 1.0}

In fact, Show for Record simply uses gshow from large-generics.

The order of the fields is preserved in the output: large-anon regards records with rows that differ only in their order as different types; isomorphic, but different. The project function can be used to translate between records with different field order; we shall see an example when we discuss sequenceA.

The RowHasField, KnownFields, AllFields and SubRow constraints (for project) are solved by the large-anon typechecker plugin, so you will need to add

{-# OPTIONS_GHC -fplugin=Data.Record.Anon.Plugin #-}

at the top of your Haskell file. We will see later how to manually prove such constraints when the plugin cannot.

Project and inject

In the previous section we saw that project can be used to reorder fields, but is actually more general than that. In addition to reordering fields, we can also omit fields: a SubRow r r' constraint is satisfied whenever the fields of r' are a subset of the fields of r. Moreover, when SubRow r r' holds we can also update the larger record from the smaller one: project and inject together form a lens.

Let’s consider an example. Suppose we have some kind of renderer with a bunch of configuration options:

type Config = [
      "margin"   := Double
    , "fontSize" := Int
    , "header"   := String
    , ...
    ]

defaultConfig :: Record Config
defaultConfig = ANON {
      margin   = 1
    , fontSize = 18
    , header   = ""
    , ...
    }