The Pathway to Information Security Management and Certification
Information security is a complex area to handle well. The possible risks to information assets and reputation, including computer systems and countless filing cabinets full of valuable proprietary information, are difficult to determine and bring under control. Plus, this needs to be done in ways that don't unduly interfere with the legitimate use of information by authorized users.
The most practical and cost-effective way to handle information security and governance obligations, and to be seen to be doing so, is to adopt an Information Security Management System (ISMS) that complies with the international standard such as SOC-2 or ISO 27001. An ISMS is a framework of policies, processes and controls used to manage information security in a structured, systematic manner.
Why implement an ISMS and pursue an Information Security Certification?
Improve policies and procedures by addressing critical security related processes and controls
Minimizes the actual and perceived impact of data breaches
Objective verification that there are controls on the security risks related to Information Assets
At a high level, the ISMS will help minimize the costs of security incidents and enhance your brand. In more detail, the ISMS will be used to:
systematically assess the organization's information risks in order to establish and prioritize its security requirements, primarily in terms of the need to protect the confidentiality, integrity and availability of information
design a suite of security controls, both technical and non-technical in nature, to address any risks deemed unacceptable by management
ensure that security controls satisfy compliance obligations under applicable laws, regulations and contracts (such as privacy laws, PCI and HIPAA)
operate, manage and maintain the security controls
monitor and continuously improve the protection of valuable information assets, for example updating the controls when the risks change (e.g. responding to novel hacker attacks or frauds, ideally in advance thereby preventing us from suffering actual incidents!).
Information Security Focus Areas
What is the proper scope for the organization?
What are applicable areas and controls?
Are the proper policies & procedures documented?
Is the organization living these values?
What are the Outcomes
Improved InfoSec policies and procedures
Confirmation of the implementation of Incident and Risk Management
Completion of Asset and Risk register
Implementation of an Information Security Management System (ISMS) for your scope
Prepared for independent certification auditor
Gain trust from customers and partners.
Information Security Certification Preparation Project
Key Project Activities
Define Certification Scope
Perform Gap Assessment against the relevant standard (SOC-2, ISO 27001)
Identify Documentation Requirements
Identify Evidence Requirements
Develop New Documents required for certification
Perform Impact Assessment
Maintain Data Flow diagrams
Maintain Risk Register
Prepare for Pre-Certification Audit
Remediate findings from Pre-Cert Audit
Prepare for Stage 1 and Stage 2
Obtain Standards Body Certification or audited Report
FP Complete has extensive experience in the preparation of SOC-2 and ISO 270001 certifications, as well as many other security certifications. Contact us if we can help your organization.
We're going to take a departure from the style of articles regularly written about the Kleisli category, because, firstly, there aren't articles regularly written about the Kleisli category.
That's a loss for the world. Why? I find the Kleisli category so useful that I'm normally programming in the category, and, conversely, I find most code in industry, unaware of this category, is doing a lot of the work of (unintentionally) setting up this category, only to tear it down before using it effectively.
So. What is the Kleisli Category? Before we can properly talk about this category, and its applications, we need to talk about monads.
Monads
A monad, as you can see by following the above link, is a domain with some specific, useful properties. If we have a monad, T, we know it comes with an unit function, η, and a join function, μ. What do these 'thingies' do?
T, the monadic domain, says that if you are the domain, then that's where you stay: T: T →T.
"So what?" you ask. The beauty of this is that once you've proved you're in a domain, then all computations from that point are guaranteed to be in that domain.
So, for example, if you have a monadic domain called Maybe, then you know that values in this domain are Just some defined value or Nothing. What about null? There is no null in the Maybe-domain, so you never have to prove your value is (or isn't) null, once you're in that monadic domain.
But how do you prove you're in that domain? η lifts an unit value (or a 'plain old' value) from the ('plain old') domain into the monadic domain. You've just proved you're in the monadic domain, simply by applying η.
η null→ fails
η some value → some value in T.
We're going to talk about the join-function, μ, after we define the Kleisli category.
The Kleisli Category
Okay. Monads. Great. So what does have to do with the Kleisli category? There's a problem with the above description of Monads that I didn't address. You can take any function, say:
f : A → B
and lift that into the monadic domain:
fT : T A → T B
... but what is fT? fT looks exactly like f, when you apply elimination of the monadic domain, T. How can we prove or 'know' that anywhere in our computation we indeed end up in the monadic domain, so that the next step in the computation we know we are in that monadic domain, and we don't have to go all the way back to the beginning of the proof to verify this, every time?
Simple: that's what the Kleisli category does for us.
What does the Kleisli category do for us? Kleisli defines the Kleisli arrow thusly:
fK : A → T B
That is to say, no matter where we start from in our (possibly interim) computation, we end our computation in the monadic domain. This is fantastic! Because now we no longer need to search back any farther than this function to see (that is: to prove) that we are in the monadic domain, and, with that guarantee, we can proceed in the safety of that domain, not having to frontload any nor every computation that follows with preconditions that would be required outside that monadic domain.
null-check simply vanishes in monadic domains that do not allow that (non-)value.
Incidentally, here's an oxymoron: 'NullPointerException' in a language that doesn't have pointers.
Here's another oxymoron: 'null value,' when 'null' means there is no value (of that specified type, or, any type, for that matter). null breaks type-safety, but I get ahead of myself.
Back on point.
So, okay: great. We, using the Kleisli arrows, know at every point in that computation that we are in the monadic domain, T, and can chain computations safely in that domain.
But, wait. There's a glaring issue here. Sure, fK gets us into the monadic domain, but let's chain the computation. Let's say we have:
gK : B → T C
... and we wish to chain, or, functionally: compose fK and gK? We get this:
gK∘ fK ⇒A → T B → TT C
Whoops! We're no longer in the monadic domain T, but at the end of the chained-computation, we're in the monadic domain TT, and what, even, is that? I'm not going to answer that question, because 1) who cares? because 2) where we really want to be is in the domain T, so the real question is: how do we get rid of that extra T in the monadic domain TT and get back into the less-cumbersome monadic domain we understand, T?
That's where the join-function, μ, comes in.
μ : TT A → T A
The join-function, μ, of a monad, T, states that when you join a monad of type T to a monad of that same type, the result, TT, when joined, simplifies to that (original) monad, T.
It's as simple as that.
But, then, it's even simpler than that, as the Kleisli arrow anticipates that you're starting in a monadic domain and wish to end up in that domain, so the Kleisli arrow entails the join function, μ, 'automagically.' With that understanding, we rewrite standard functional composition to composition under the Kleisli category:
gK∘KfK ⇒A → T B → T C
Which means we can now compose as many Kleisli arrows as we like, and anywhere we are in that compositional-chain, we know we are in our good ole' safe, guaranteed monadic domain.
Practical Application
We've heard of the Maybe monad, which entails semi-determinism, this monad is now part of the core Java language as the Optional-type, with practical application in functional mapping, filtering and finding, as well as interactions with data-stores (e.g.: databases).
Boy, I wish I had that back in the 1990's, developing the Sales Comparison Approach ingest process of mortgage appraisal forms for Fannie Mae. I didn't. I had Java 1.1. A glaring 'problem' we had to address with ingesting mortgage appraisals what that every single field is optional, and, as (sub-)sections depend on prior fields, entire subsections of the mortgage appraisal form were dependently-optional. The solution the engineering team at that time took was to store every field of every row of the 2000 fields of the mortgage appraisal form.
Fannie Mae deals with millions of mortgage appraisals
Each.
Year.
Having rows upon rows (upon rows upon rows) of data tables of null values quickly overloaded database management systems (at that time, and, I would argue, is tremendously wasteful, even today).
My solution was this, as each field is optional, I lifted that value into the monadic domain, that way, when a value was present, follow-on computations, for follow-on (sub-)sections would execute, and, as the process of lifting failed on a null-value, follow-on computations were automagically skipped for absent values. Only values present were stored. Only computations on present values were performed. The Sales Comparison Approach had over 600 fields, all of them optional, many of them dependently-optional, and only the values present in those 600 fields were stored. The savings in data storage was exponentially more efficient for the Sales Comparison Approach section as compared to the storage for the rest of the mortgage appraisal.
Although easy enough, and intuitive, to say, actually implementing Kleisli Arrows in Java 1.1 (the langua fraca for that project) required I first implement the concept of both Function and Functor, using inner classes, then I needed to implement the concept of Monad, then Maybe, then, with monad, I needed to implement the Kleisli Arrow monadic-bind function to be able to stitch together dozens and hundreds of computations together, without one single explicit null-check.
The data told me if they were present, or not, and, once lifted into the monadic domain, the Kleisli arrows stitched together the entire Sales Comparison Approach section, all 600 computations, into a seamless and concise workflow.
Summary
Kleisli arrows: so useful they are recognized as integral to the modern programming paradigm. What's fascinating about these constructs is that they are grounded in provable mathematical forms, from η to μ to Kleisli-composition. This article summarized the mathematics underpinning the Kleisli category and showed a practical application of this pure mathematical form that translated directly into space-efficient and computationally-concise software delivered into production that is still used to this day.
TIL that ZSH completions that come with Nix packages end up in
~/.nix-profile/share/zsh/vendor-completions/ and that folder is not added to
$FPATH by the init script that comes with Nix.
After modifying the bit in ~/.zshenv it now looks like this
The GHC team is very pleased to announce the availability of GHC 8.10.5. Source and binary distributions are available at the usual place.
This release adds native ARM/Darwin support, as well as bringing performance improvements and fixing numerous bugs of varying severity present in the 8.10 series:
First-class support for Apple M1 hardware using GHC’s LLVM ARM backend
Fix a bug resulting in segmentation faults where code may be unloaded prematurely when using the parallel garbage collector (#19417) along with other bugs in the GC and linker (#19147, #19287)
Improve code layout fixing certain performance regressions (#18053) and other code generation bug fixes (#19645)
Bug fixes for signal handling when using the pthread itimer implementation.
I’m a huge fan of Slay the Spire and I wanted to share a small probability trick I’ve learned that has helped me improve my game by calculating the odds of “greedy” plays succeeding. This trick will likely also benefit other games based on cards or probability, too, but the examples from this post will be specific to Slay the Spire.
The general problem I was trying to solve is:
I have a deck containing N desirable cards out of D total cards. If I draw H cards from that deck, what is the chance that I draw (exactly / at least / at most) M desirable cards?
This sort of question comes up often in card games, including complex ones (like Slay the Spire) or simpler ones (like poker). Here are some concrete examples of how this relates to Slay the Spire:
“Next turn I draw 5 cards. What is the likelihood that I draw at least 3 Strikes if there are 7 cards left in the deck, 4 of which are Strikes”.
(Answer: 5 / 7 ≈ 71% chance)
“One Neow bonus lets me lose 7 max HP to select from 1 of 3 random rare cards. Right now I’m only interested in 6 of the 17 possible rare cards, so what is the chance that I random at least 1 of those 6?”
(Answer: 103 / 136 ≈ 76% chance)
The nice thing about Slay the Spire is that it’s a turn-based single-player game, so (unlike poker) you can take all the time you want when deciding your turn and nobody will rush you to hurry up. This means that I can safely geek out on these sorts of probability calculations to increase my likelihood of winning.
In this post I’ll first present the solution to the above probability question (both as a mathematical formula and as code) and then explain why the formula works.
The formula
Before presenting the solution to the above problem, I’d first like to reframe the problem using more precise set notation to avoid ambiguity:
Define U to be the universal set of all cards in the deck and then define two (potentially overlapping) subsets of those cards:
Finally, we’ll use ∩ to denote set intersection, which we will use to define four non-overlapping (i.e. disjoint) sets:
¬B │ B ┌─────────┼─────────┐ ¬A │ ¬A ∩ ¬B │ ¬A ∩ B │ ────┼─────────┼─────────┤ A │ A ∩ ¬B │ A ∩ ¬B │ └─────────┴─────────┘
… where:
A ∩ B means the set of cards that are drawn and desirable
A ∩ ¬B means the set of cards that are drawn, but undesirable
¬A ∩ B means the set of cards that are desirable, but not drawn
¬A ∩ ¬B means the set of cards that are neither desirable nor drawn
Then the problem becomes:
If the size of the sets A and B are fixed, what is the probability that their overlap (i.e. the set A ∩ B) has a given size?
I use this presentation to highlight that:
The problem isn’t really specific to cards. It will work for arbitrary sets
The problem isn’t specific to drawing cards or desirable cards. It works for any two subsets of cards
The problem is symmetric with respect to A and B. If we swap the size of the two subsets of cards we expect to get the same answer
Using |S| to denote the size (i.e. cardinality) of a set S and using ! to denote factorial, then the probability of two subsets of fixed sizes overlapping by a given size is:
With this solution in hand, we can solve our original problem by just renaming the variables:
I have a deck containing |B| desirable cards out of |U| total cards. If I draw |A| cards from that deck, what is the chance that I draw (exactly / at least / at most) |A ∩ B| desirable cards?
We already have the solution for the case where we draw exactly |A ∩ B| desirable cards. From that we can compute the solutions for drawing at most or at least |A ∩ B| desirable cards.
The code
If you still don’t follow along, I translate the above solution into Haskell code in this section.
Note that even though the previous formula depends on the sizes of nine sets:
|U|
|A|
|¬A|
|B|
|¬B|
|A ∩ B|
|A ∩ ¬B|
|¬A ∩ B|
|¬A ∩ ¬B|
… there are really only four degrees of freedom, because the size of the following four disjoint sets:
|A ∩ B|
|A ∩ ¬B|
|¬A ∩ B|
|¬A ∩ ¬B|
… uniquely determine the sizes of the other sets, because:
… so each function in our API will only take four function arguments, corresponding to the size of those four disjoint sets. There are other ways we could define the API that use a different four degrees of freedom, but I find this interface to be the simplest and most symmetric.
moduleProbabilitywhere importNumeric.Natural (Natural) -- Use exact Rational arithmetic by default instead of floating point arithmetic default (Natural, Rational) -- Obligatory: http://www.willamette.edu/~fruehr/haskell/evolution.html? factorial :: (Enum n, Num n) => n -> n factorial n =product [1..n] {-| The probability that two sets of sizes @|A|@ and @|B|@ overlap by a set of exactly size @|A ∩ B|@ prop> exactly 1 0 0 a === 1 % (a + 1) prop> exactly a b 0 0 === 1 prop> exactly a 0 b 0 === 1 prop> exactly a b c d === exactly a c b d -} exactly :: (Enum n, Fractional n) =>Natural -- ^ The size of @|A ∩ B|@ ->Natural -- ^ The size of @|A ∩ ¬B|@ ->Natural -- ^ The size of @|¬A ∩ B|@ ->Natural -- ^ The size of @|¬A ∩ ¬B|@ -> n exactly _AB _AnotB _BnotA notAB = fromIntegralnumerator/fromIntegraldenominator where _A = _AB + _AnotB _B = _AB + _BnotA notB = _AnotB + notAB notA = _BnotA + notAB _U = _AB + _AnotB + _BnotA + notAB numerator=product (map factorial [ _A, notA, _B, notB ]) denominator=product (map factorial [ _AB, _AnotB, _BnotA, notAB, _U ]) {-| The probability that two sets of sizes @|A|@ and @|B|@ overlap by a set of at least size @|A ∩ B|@ prop> atLeast 0 a b c === 1 prop> atLeast a b c d === atMost a c b d -} atLeast :: (Enum n, Fractional n) =>Natural -- ^ The minimum size of @|A ∩ B|@ ->Natural -- ^ The maximum size of @|A ∩ ¬B|@ ->Natural -- ^ The maximum size of @|¬A ∩ B|@ ->Natural -- ^ The minimum size of @|¬A ∩ ¬B|@ -> n atLeast _AB _AnotB _BnotA notAB =sum (map overshootBy [0..(min _AnotB _BnotA)]) where overshootBy x = exactly (_AB + x) (_AnotB - x) (_BnotA - x) (notAB + x) {-| The probability that two sets of sizes @|A|@ and @|B|@ overlap by a set of at most size @|A ∩ B|@ prop> atMost 0 a b c === exactly 0 a b c prop> atMost a 0 b c === 1 prop> atMost a b 0 c === 1 prop> atMost a b c d === atMost a c b d prop> atMost a b c d + atLeast a b c d === 1 + exactly a b c d -} atMost :: (Enum n, Fractional n) =>Natural -- ^ The maximum size of @|A ∩ B|@ ->Natural -- ^ The minimum size of @|A ∩ ¬B|@ ->Natural -- ^ The minimum size of @|¬A ∩ B|@ ->Natural -- ^ The maximum size of @|¬A ∩ ¬B|@ -> n atMost _AB _AnotB _BnotA notAB =sum (map undershootBy [0..(min _AB notAB)]) where undershootBy x = exactly (_AB - x) (_AnotB + x) (_BnotA + x) (notAB - x)
Exercise: If you don’t use Haskell, try to port the above functions to your favorite programming language.
Examples
Let’s test drive the above utilities on the example scenarios from Slay the Spire.
“Next turn I draw 5 cards. What is the likelihood that I draw at least 3 Strikes if there are 7 cards left in the deck, 4 of which are Strikes”.
Here, we will be using the atLeast function with the following input sizes for each disjoint set:
| A ∩ B| = 3 - We wish to draw at least 3 Strike cards
| A ∩ ¬B| = 2 - We wish to draw at most 2 non-Strike cards
|¬A ∩ B| = 1 - We wish to leave at most 1 Strike card in the deck
|¬A ∩ ¬B| = 1 - We wish to leave at least 1 non-Strike card in the deck
… which gives us a 5 in 7 chance:
>>> atLeast 3211 5%7
Exercise: Use the atMost function to compute the chance of falling short by drawing at most 2 Strikes.
“One Neow bonus lets me lose 7 max HP to select from 1 of 3 random rare cards. Right now I’m only interested in 6 of the 17 possible rare cards, so what is the chance that I random at least 1 of those 6?”
This will also use the atLeast function with the following input sizes for each disjoint set:
| A ∩ B| = 1 - We wish to draw at least 1 desired rare card
| A ∩ ¬B| = 2 - We wish to draw at most 2 undesirable rare cards
|¬A ∩ B| = 5 - We wish to leave at most 5 desirable cards in the pool
|¬A ∩ ¬B| = 9 - We wish to leave at least 9 undesirable cards in the pool
… which gives us a 103 in 136 chance:
>>> atLeast 1259 103%136
Exercise: Use the exactly function to compute the chance of randoming 0 desirable cards.
Explanation
This section provides a semi-rigorous explanation of why the formula works, although probably not rigorous enough to be called a proof.
The probability of drawing a correct hand is:
{The number of correct hands} p = ────────────────────────────── {The number of possible hands}
The number of possible hands is the number of ways that we can draw a hand of size |A| (without replacement) from a pool of |U| cards (our deck):
{The number of possible hands} = |U| choose |A|
Similarly, the number of correct hands is the product of:
the number of ways to draw |A ∩ B| desirable cards from a pool of |B| desirable cards and
the number of ways to draw |A ∩ ¬B| undesirable cards from a pool of |¬B| undesirable cards
{The number of correct hands} = (|B| choose |A ∩ B|) × (|¬B| choose |A ∩ ¬B|)
Then we can use the “n choose k” formula for computing the number of ways to select n cards from a pool of k cards without replacement, which gives us:
Unfortunately, this formula is quite difficult to do in one’s head so I usually load these utility functions into a Haskell REPL to do the math on tricky turns. I haven’t yet figured out an easy heuristic for getting a quick and approximate answer for this sort of probability calculation.
However, if you’re willing to take the time to compute the odds this sort of calculation can quickly add up to improving your odds of winning a Slay the Spire run. I often use this trick to compute the expectation value of greedy plays which can save quite a bit of health or potions over the course of a run.
I'm happy to announce that over the next few months, FP Complete will be offering intermediate training courses on both Haskell and Rust. This is a follow up to our previous beginner courses on both languages as well. I'm excited to get to teach both of these courses.
More details below, but cutting to the chase: if you'd like to sign up, or just get more information on these courses, please email training@fpcomplete.com.
Overall structure
Each course consists of:
Four sessions, held on Sunday, 1500 UTC, 8am Pacific time, 5pm Central European
Each session is three hours, with a ten minute break
Slides, exercises, and recordings will be provided to all participants
Private Discord chat room is available to those interested to interact with other students and the teacher, kept open after the course finishes
Dates
We'll be holding these courses on the following dates
Haskell
June 13
June 20
July 11
July 25
Rust
August 8
August 15
August 22
August 29
Cost and signup
Each course costs $150 per participant. Please register and arrange payment (via PayPal or Venmo) by contacting training@fpcomplete.com.
Topics covered
Before the course begins, and throughout the course, I'll ask participants for feedback on additional topics to cover, and tune the course appropriately. Below is the basis of the course which we'll focus on:
The following post was meant to be posted at https://forum.dfinity.org/, but that discourse instance didn’t like it; maybe too much inline code, so I’m posting it here instead. To my regular blog audience, please excuse the lack of context. Please comment at the forum post. The text was later also posted on the DFINITY medium blog
You probably have used https://identity.ic0.app/ to log into various applications (the NNS UI, OpenChat etc.) before, and if you do that, you are trusting this service to take good care of your credentials. Furthermore, you might want to check that the Internet Identity is really not tracking you. So you want to know: Is this really running the code we claim it to run? Of course the following applies to other canisters as well, but I’ll stick to the Internet Identity in this case.
I’ll walk you through the steps of verifying that:
Find out what is running
A service on the Internet Computer, i.e. a canister, is a WebAssembly module. The Internet Computer does intentionally not allow you to just download the Wasm code of any canisters, because maybe some developer wants to keep their code private. But it does expose a hash of the Wasm module. The easiest way to get it is using dfx:
$ dfx canister --no-wallet --network ic info rdmx6-jaaaa-aaaaa-aaadq-cai
Controller: r7inp-6aaaa-aaaaa-aaabq-cai
Module hash: 0xd4af9277f3e8d26fd8cdc7874a9f47b6456587fbb2a64d61b6b6880d144d3c04
The “controller” here is the canister id of the governance canister. This tells you that the Internet Identity is controlled by the Network Nervous System (NNS), and its code can only be changed via proposals that are voted on. This is good; if the controller was just, say, me, I could just change the code of the Internet Identity and take over all your identities.
The “Module hash” is the SHA-256 hash of the .wasm that was deployed. So let’s follow that trace.
We have to find the latest proposal upgrading the Internet Identity. The folder unfortunately contains proposals for many canisters, and the file naming isn’t super helpful. I usually go through the list from bottom and look at the second column, which contains the title of the latest commit creating or modifying a file.
The file also says that the wasm hash is d4a…c04, which matches what we saw above. This is good: it seems we really found the youngest proposal upgrading the Internet Identity, and that the proposal actually went through.
WARNING: If you are paranoid, don’t trust this file. There is nothing preventing a proposal proposer to create a file pointing to one revision, but actually including other code in the proposal. That’s why the next steps are needed.
Getting the source
Now that we have the revision, we can get the source and check out revision bd51eab:
In the last line you see that the Internet Identity team has tagged that revision with a tag name that contains the proposal description file name. Very tidy!
Reproducing the build
The README.md has the following build instructions:
Official build
The official build should ideally be reproducible, so that independent parties can validate that we really deploy what we claim to deploy.
We try to achieve some level of reproducibility using a Dockerized build environment. The following steps should build the official Wasm image
You don’t believe me? Try it yourself (and let us know if you get a different hash, maybe I got hacked). This may fail if you have too little RAM configured for Docker, 8GB should be enough.
At this point you have a trust path from the code sitting in front of you to the Internet Identity running at https://identity.ic0.app, including the front-end code, and you can start auditing the source code.
What about the canister id?
If you payed close attention you might have noticed that we got the module has for canister rdmx6-jaaaa-aaaaa-aaadq-cai, but we are accessing a web application at https://identity.ic0.app. So where is this connection?
In the future, I expect some form of a DNS-like “nice host name registry” on the Internet Computer that stores a mapping from nice names to canister ids, and that you will be able to query that to for “which canister serves rdmx6-jaaaa-aaaaa-aaadq-cai” in a secure way (e.g. using certified variables). But since we don’t have that yet, but still want you to be able to use a nice name for the Internet Identity (and not have to change the name later, which would cause headaches), we have hard-coded this mapping for now.
The relevant code here is the “Certifying Service Worker” that your browser downloads when accessing any *.ic0.app URL. This piece of code will then intercept all requests to that domain, map it to an query call, and then use certified variables to validate the response. And indeed, the mapping is in the code there:
In principle, the same approach works for other canisters, whether it’s OpenChat, the NNS canisters etc. But the details will differ, as every canister developer might have their own way of
communicating the location and revision of the source for their canisters
building the canisters
In particular, without a reproducible way of building the canister, this will fail, and that’s why projects like https://reproducible-builds.org/ are so important in general.
Starting with ghc-9.2.1, parse trees contain “annotations” (these are, for example, comments and the locations of keywords). This represents a non-trivial upgrade of GHC parse trees. If you work with GHC ASTs in your project, there will be no avoiding getting to know about them. This note is a summary overview of annotations: the where and how of their representations.
In-tree annotations enable exact-printing of GHC ASTs. This feature and the reformulation of the GHC AST with in-tree annotations to support it was conceived of and implemented by Alan Zimmerman (@alan_zimm). The achievement is of truly epic scale.
Annotations on syntactic elements
An EpaLocation is a span giving the exact location of a keyword in parsed source.
A DotFieldOcc arises in expressions like (.e) (field-selector) or a.e (field-selection) when OverloadedRecordDot is enabled. A DotFieldOcc value in the parse phase is associated with an AnnFieldLabel in its extension field (annotations in ghc-9.2.1 lean heavily on the facilities afforded by TTG). The AnnFieldLabel contains the location of the ‘.’. AnnFieldLabel is an “annotation type”. You’ll recognize annotation types (there are many) by the convention that their names are prefixed Ann.
-- GHC.Hs.Expr dataAnnFieldLabel =AnnFieldLabel { afDot ::MaybeEpaLocation } typeinstanceXCDotFieldOcc (GhcPass _) =EpAnnAnnFieldLabel -- Language.Haskell.Syntax.Expr dataDotFieldOcc p =DotFieldOcc { dfoExt ::XCDotFieldOcc p , dfoLabel ::XRec p FieldLabelString } |XDotFieldOcc!(XXDotFieldOcc p)
(What XRec p FieldLabelString means will be explained in the next section.)
Note that the extension field dfoExt doesn’t contain a “raw” AnnFieldLabel, rather, it contains an EpAnn AnnFieldLabel.
EPAnn, envelopes an annotation. It associates a base location for the start of the syntactic element containing the annotation along with any comments enclosed in the source span of the element to which the EPAnn is attached. EpAnnUnsed is used when an annotation is required but there’s no annotation available to envelope (e.g one obvious case being in generated code).
dataEpAnn ann =EpAnn { entry ::Anchor , anns :: ann , comments ::EpAnnComments } |EpAnnNotUsed dataEpAnnComments=...
It’s the Anchor type where the base location is held.
Annotations don’t just get attached to syntactic elements, they frequently get attached to source spans too.
dataSrcSpanAnn' a =SrcSpanAnn { ann :: a, locA ::SrcSpan }
Usually SrcSpanAnn' is used with EpAnn and that combination is named a SrcAnn.
dataSrcAnn ann =SrcSpanAnn' (EpAnn ann)
There are many annotation types. The most ubiquitous are AnnListItem, NameAnn, AnnList, AnnPragma and AnnContext. Their use is common enough that names are given to their SrcAnn types (which you recall, wrap them in EpAnn and associate them with a SrcSpan).
These type synonyms are only for the most commonly used annoation types. The general case is LocatedAn an.
typeLocatedAn an =GenLocated (SrcAnn an)
To recap, a LocatedAn an is a GenLocated (SrcAnn an) which is a GenLocated (SrcSpanAnn' (EpAnn an)).
Abstracting over locations
Syntax definitions are generalized with respect to location information. That is, rather than hard-coding SrcSpan into syntax type definitions as we used to, type families are used in their place so that the structure of the syntax including locations can be described without fixing concrete types for the locations where you’d once have had a source span type.
It works like this. In Language.Haskell.Syntax.Extension there is this definition:
typefamilyXRec p a = r | r -> a
Locations are specified in terms of XRecs. For example in Language.Haskell.Syntax.Expr we have this:
typeLHsExpr p =XRec p (HsExpr p)
How XRec p (HsExpr p) is mapped onto a specific type in GHC is achieved in the following way. First in Language.Haskell.Syntax.Extension there is the following definition:
typefamilyAnno a = b
Then, in GHC.Hs.Extension this definition:
typeinstanceXRec (GhcPass p) a =GenLocated (Anno a) a
Specific choices for each syntatic element can then be made for GHC’s use of the parse tree and phase. For example, in GHC.Hs.Expr we have the following.
To see how this works, consider what that means for the located expression type LHsExpr GhcPs in GHC. We have LHsExpr GhcPs is XRec GhcPs (HsExpr GhcPs) which is GenLocated (Anno (HsExpr GhcPs)) (HsExpr GhcPs) or GenLocated SrcSpanAnnA (HsExpr GhcPs) (or, LocatedA (HsExpr GhcPs)) if you like).
Expanding further we have GenLocated SrcSpanAnnA (HsExpr GhcPs) is GenLocated (SrcAnn AnnListItem) (HsExpr GhcPs). So ultimately, LHsExpr GhcPs is GenLocated (SrcSpanAnn' (EpAnn AnnListItem)) (HsExpr GhcPs).
I have always loved finding those parts of mathematics where big ideas just pop out almost by accident, appearing fully formed in a surprising and delightful way. Who, for example, could fail to marvel at how topology manages to capture so many unique and striking meanings into words like continuous which can mean very different things just by choosing a different topology in which to interpret the same definition?
Something like this happens in categories by considering limits and colimits. One simply draws a simple picture, follows a simple set of rules, and important ideas pop out. Here, I’m going to present the technique, and then draw a particular picture and see how two big ideas in dynamical systems jump out and startle us with their unexpected appearance.
Don’t worry if you’re not up to speed on category theory. In fact, I won’t even be using real category theory at all here, because we don’t need that kind of abstraction. The only ideas we need are sets, functions, function composition, and identity functions. I’ll assume you know what these mean.
Sets without elements?
Here’s a game: How much can we say about sets without talking about their elements (that is, the things inside of them)? We can still talk about the functions from one set to another, including the identity function, and even composition of those functions. We just can’t talk about the members themselves.
At first, that might seem like a silly restriction, sort of like trying to go a whole day without using any words with the letter Q. That’s okay; I don’t need to convince you it’s not silly. If I were trying to convince you, though, I might mention that the more general notion of categories includes more than just sets, and those things might not have elements. I might also say that the whole idea of membership in a set is an ultimately undefinable primitive idea in conventional mathematics, and using that idea too informally has led to logical paradoxes in the past. So it makes some sense to ask whether, instead of just assuming that set membership makes sense, we can start with a different foundation (functions) and derive the idea of set membership in a logically sound way. But again, you don’t need to be convinced. Just humor me. It’s a game. Those are the rules.
Let’s practice:
Can I say “The set A has only one element” without saying anything about elements? … Surprisingly, yes! Instead of talking about the one element, I can say this: “Given any other set B, there is exactly one function f : B → A.”
If A had more than one element, then you could choose where to map an element of B. A having only one element means that everything in B has to map to that element; there is no choice. So the existence of exactly one such function means the same thing as saying A has one element.
Can you find a way to say “The set A is empty”? There is more than one answer. I’ll leave it as an exercise for the reader! (Hint: you can do it with a straight-forward application of constructions and diagrams in this post.)
Let’s do one that’s a little more complicated, just to get an idea of how things go in general.
Can you say “The set A has the same elements as the set B” without saying anything about elements? … Since we have no way to actually name or describe those elements (since we’re forbidden from even talking about them), we can not say this. But we can get close, by saying that A has the same number of elements as B. Here’s how to do it: “There is a function f : A → B, and another function g : B → A, and g ∘ f is the identity function on A, and f ∘ g is the identity function on B.”
In mathematics, this is known as a one-to-one correspondence, and it proves that these two sets have the same number of elements (formally: the same cardinality, since it might actually be infinite and therefore not a number, per se.)
This will be a sort of pattern. We can’t name the things in our sets, but we can get an idea of what’s in them by showing that they correspond to the things we want. So instead of a set with the same elements, we got a set, and a function that shows how whatever is in that set corresponds to the set we wanted. Another way of saying this is to say that the sets are the same “up to one-to-one-correspondence,” or more abstractly, “up to isomorphism.”
Getting the hang of it? Okay, one more. This one will come up later, so follow along. Take a couple of sets: call them A and B. The cartesian product of A and B, written A × B, is the set of ordered pairs (x, y), where x is an element of A, and y is an element of B.
Back to our game from earlier.
Can you define the cartesian product without talking about elements? … Well, just like the previous example, we can’t be sure what exactly the elements in our set are called, but we can say that a set has the right number of elements to be the cartesian product. And what’s more, we can even provide a pair of functions that give the components of the ordered pair, giving a way to interpret everything in that set as an ordered pair.
To look at how that’s done, let’s define a couple functions p and q, called projections, that each pick one of the components out of an ordered pair.
The green function is p : A × B → A, and the red function is q : A × B → B. They don’t really do much. They are almost as boring as the identity function! But just like the identity function was useful for saying how two sets have the same number of elements, these will be useful for defining the cartesian product.
We have a problem: the projections are defined in terms of what they do with elements, but we cannot talk about elements. (The rules of the game allow us to talk about the identity function as we did last time, but we have no such exception for projections.) So at first glance, the best we can do is just say that p and q are functions from A × B to A and B.
All that structure about ordered pairs and components, sadly, is lost. But we can recover it by saying something about how this relates to other ways to define functions to A and B. Here’s the final answer.
“A × B is a set where there are two functions p : A × B → A, and q : A × B → B, and for any other set S and functions f : S → A and g : S → B, there is auniqueh : S → A × B, such that f = p ∘ h, and g = q ∘ h.”
Well, that was a mouthful. Here’s what it means.
The functions f and g mean that every element of S has some associated pair of elements from A and B. Any particular S may not be associated to all of the possible pairs, but importantly, we can choose an S and f and g that associate any pair, if we like.
That h exists means, therefore, that for any choice of elements from A and B, there’s an element in A × B that corresponds to it. The projections p and q tell us which A and which B each element corresponds to.
That h is unique means that there is only one such element for each pair, since otherwise there would be a choice of which element h points at.
So, all together, this says A × B has exactly one element in it for each choice of a pair of elements from A and B, and p and q tell us which pair that is. Success!
Generalizing to the limit
That last answer has a common form. It says that among all the possible choices of a set and related functions (that is, S, f, and g), there is one particular choice (A × B, p, and q) that’s somehow universal, because any other choice you could make is nothing more than a mapping to that universal one. In category theory, this form of thing is called a limit. This is only distantly related to the limits you might have seen in calculus, so if that’s what jumped in your mind, go ahead and forget it again.
The cool thing about limits is that you can take any collection of sets and functions that you like, and ask what their limit is. Start from any little diagram of sets and functions, and an idea pops out. Often, it’s an important central idea about sets and functions, like cardinality and one-to-one correspondence (which come from the diagram with one set) or cartesian products (which come from the diagram with two or more sets, as we saw). The limit is always the universal set, along with functions to all of the sets you started with, so that any other set and functions are uniquely determined by a mapping from that set into the universal set. The idea then just pops out of which diagram you start with.
Let’s look at another example, though it’s a bit of an odd one. Let’s take an empty diagram, with no sets and functions at all. Then the limit of this diagram is just some set U, such that for any other set S, there is a unique function h : S → U.
Sound familiar? That was the answer to the first question in our game where we described sets with one element. That answer, as well, is just the limit, this time of the empty diagram!
So far, the diagrams we’ve looked at had only sets, and not functions. There’s another wrinkle in the picture if you want to construct the limit of a diagram with a function. Here’s a diagram with two sets A and B and a function f, as well as the universal object U and functions p and q that form the limit of this diagram.
The rule says that when you compose a function into the diagram (say, p) with a function in the diagram itself (say, f), you have to get the same thing as the function into the diagram that points to the same place. So f ∘ p = q. In category theory, this is sometimes expressed by saying it “commutes”, which is just a fancy word that means it doesn’t matter which path you take, because you get the same function in the end.
In this case, that means q is completely determined by p and f, so we don’t need to think about it at all! In fact, this ends up meaning we don’t need to think about B or f either, and the limit here is the same as a diagram with just A: any set with the same cardinality as A, where p is a specific one-to-one correspondence that witnesses this. So this wasn’t an interesting example. But there are interesting limits with functions, and we’re about to see one.
Dynamical systems appear!
If we’re thinking of simple diagrams to work out limits for, what about this one?
The diagram has a single object A, and a single function f from A to itself. One of the smallest diagrams we can think of. What could its limit be?
We can just follow the same process, and end up with this statement: “U is some set, together with a function g : U → A, where f ∘ g = g. Additionally, for any other set S and function k : S → A where f ∘ k = k, there is a unique function h : S → U so that g ∘ h = k.”
But what does that really mean? We can puzzle it out.
f ∘ g = g means that f doesn’t change anything in the result of g. In other words, for any x in the result of g, we must have f(x) = x. There’s a word for values that are unchanged by a function: fixed points (or sometimes fixpoints, or stationary points). So g can only point to the fixed points of f.
By the same logic, any choice of k can only point to the fixed points of f. But it’s possible to choose a k that points to any fixed point of A.
That h exists, then, means that g must map some element of U to every fixed point of f. (If it didn’t, then k might map something to that fixed point, and it couldn’t be obtained by composing with g.)
That h is unique means that g maps only one thing to each fixed point of f. (Otherwise, there would be a choice of where h should map some things, so h wouldn’t be unique.)
So U is in one-to-one correspondence with the fixed points of f, and g is that one-to-one correspondence. Essentially (up to isomorphism) the limit of this diagram is the fixed points of f.
Fixed points are a fundamental idea that pops up all over the place. I’ve written about their role in denotational semantics of programming languages before. So I find it delightful that they jump out of such a simple diagram. At the same time, though, it’s not surprising that they jump out of this diagram, which is the one that really expresses the essence of a discrete dynamical system: a set (the phase space), together with a function from that set to itself that defines a dynamics on that phase space.
Reversing course: colimits
One famous thing about category theory that everything has a dual, obtained by just reversing the directions of all the arrows. In this case, we can convert the definition of a limit into a different definition: the colimit. This looks just like the limit, but reverses the direction of the arrows.
As a warm-up, let’s consider the diagram with just two sets. Recall that the limit of this diagram was the cartesian product of those two sets. (More specifically, it was the cartesian product, along with the two projection functions that explained how to interpret each element as an ordered pair.)
The same diagram can be used with arrows reversed to find a colimit.
The formal definition is this: The colimit of the diagram is a set A + B, together with functions p : A → A + B and q : B → A + B, such that for every other set S and functions f : A → S and g : B → S, there is a unique function h : A + B → S so that f = h ∘ p and g = h ∘ q.
Let’s unpack what that means, now. First of all, everything in A maps to something in A + B, and likewise, everything in B maps to something in A + B. (We would usually call p and q injections now instead of projections.) But that’s not the whole picture, because we also have to look at h.
The existence of h means that no two elements of A or B map to the same element of A + B. If they did, then we could find an f and g that map them somewhere different, and then we couldn’t write that f and g as compositions with h.
The uniqueness of h means that there’s nothing else in A + B except these elements that are mapped from A and B. If there were, then there would be a choice of where h maps those extra elements.
If you’re familiar with basic ideas about sets, you might want to say that A + B is the union of A and B. But that’s not quite right, because if A and B have elements in common, A + B still needs to contain an element for each one (because f and g could map that same value to different elements of some S.) So it’s actually called the disjoint union. It’s a set that contains a copy of everything in A and everything in B, keeping two copies if A and B have elements in common.
In some sense, both the limit and the colimit of this diagram are about combining the elements of A and B, but they just combine them in different ways: the limit with a cartesian product that describes ways to choose one element from each, and the colimit with the disjoint union that describes ways to choose one elements from either one. In general, even though it was this time, the limit isn’t necessarily different from the colimit. For example, diagram with just one set has the same limit and colimit: both of them are just a set and a one-to-one correspondence between that set and the original set.
Colimits and dynamical systems
Let’s return to the earlier diagram of a dynamical system:
It’s now natural to ask what the colimit would be for this diagram. So let’s work that out with this picture.
Here’s what the definition of the colimit works out to this time: A colimit of this diagram is a set U together with a function k : A → U for which k = k ∘ f. Furthermore, for any other set S and function g : A → S where g = g ∘ f, there is a unique function h : U → S such that g = h ∘ k.
Let’s break down the meaning one piece at a time.
First, k = k ∘ f. What this tells us is that for any x in A, k(f(x)) = k(x). But that’s true for anything at all in A, including f(x), and f(f(x)) and so on, so we can write a chain of equalities: k(x) = k(f(x)) = k(f(f(x))) = k(f(f(f(x)))) = …, forever and ever. This brings us to another idea in dynamical systems: the orbit. An orbit (or sometimes trajectory) is a set of values that eventually map to the same things when iterating the transition function. So this says that k maps everything in the same orbit of f to the same result. (So do all valid choices of g, which has the same condition.)
The existence of h tells us that k maps distinct orbits of f to different places. (Otherwise, if g mapped them to different places, then g couldn’t be written as a composition with k.)
The uniqueness of h tells us that there’s nothing else in U other than one element for each orbit of f. (Otherwise, h would have a choice where to map these extra elements, so it wouldn’t be unique.)
Summing it all up, U is effectively the set of all orbits of the function f. An orbit can be thought of an an “eventual” behavior of f when applied over and over if you just ignore differences at the beginning of the process. In fact, a fixed point is one type of orbit! If x is a fixed point of f, then anything that ever reaches x will stay there forever since applying f more often doesn’t change the value. But it’s not the only kind of orbit. You can also have a longer cycle of repeating values. Or, despite what the name implies, an orbit need not be cyclic at all. For example, the function f(n) = n + 1 on the natural numbers has only one orbit: the one that counts off toward positive infinity increasing one number at a time.
Orbits come up all the time when one looks at iterated functions. The famous Collatz conjecture, for example, is just the statement that the function f on the positive integers defined by
has only one orbit. If the Collatz conjecture is true, then that one orbit cycles through the sequence 4, 2, 1, 4, 2, 1, … forever, and every possible starting point eventually ends up in that cycle. If the Collatz conjecture is false, then there is some other orbit, whether it’s another cycle or a sequence that ends up growing larger and larger forever.
Just like disjoint unions and cartesian products were two different takes on combining two sets, fixed points and orbits are related but different takes on understanding the eventual behaviors of a dynamical system. One is the limit, and one is the colimit, but both start from the same simple diagram.
What’s next?
This was just one example of a fundamental idea from a branch of mathematics that just pops out of a very abstract and general construction, fully defined as if by magic. Even though the diagram involved is trivial, I haven’t actually seen this mentioned in standard lists of examples of limits and colimits in textbooks! This is surprising, but I’m also glad, because I think I’d find this a little less exciting if it had just been example 2 in a textbook.
I would strongly encourage you to explore limits and colimits on your own by considering other simple diagrams you can draw, and checking whether there’s some natural characterization of the limit and colimit.
I’ve also stuck to sets and functions in this article, but the “game” where we didn’t talk about elements or membership did have a point. When you play that game with sets, you get an alternative kind of set theory for mathematics called the “Elementary Theory of the Category of Sets”, or ETCS for short. In the traditional ZF set theory, you assume there’s some notion of membership, and then work to build up ordered pairs, then relations, then functions. But in ETCS, you assume there’s a notion of functions, and work backward to recover ordered pairs and such. You can even recover an idea analogous to set membership: an element of a set is analogous to a function from some other one-element set, which we defined without reference to elements, into that set.
But because there was no direct reference to elements, the same definitions work in any other category; that is, a collection of things and arrows between them that compose in a manner like function composition. You can, for example, consider any algebraic structure (groups, rings, etc.) and the homomorphisms between them, or topological spaces and continuous functions between them, or even fundamentally non-function-like things such as partial orders (or any other reflexive transitive relation).
Although the same definitions work for any category, but they can mean very different things, and they may or may not even exist. One example: if you work out what sums and products of two things (that just means the colimit and limit of a diagram with two objects and no arrows) look like for the category of abelian groups, you’ll discover that they are the same thing! Both correspond to a notion that’s usually called a direct sum or direct product, which are the same thing for any finite number of groups. That’s very different from sets, where one was the cartesian product and the other the disjoint union.
Please do play around with your own diagrams, your own categories, and so on, and see where it takes you. If you find another interesting limit or colimit that isn’t a common example, please comment and let me know about it.
This blog post is covering two completely unrelated announcements about Stack that happened to land at the same time.
Haskell Foundation Slack chat rooms
I've long wanted to have a good discussion area for Stack in text chat. We've tried Gitter, Keybase, and Matrix. All of them lacked sufficient critical mass to get moving. We had a chatroom for Stack collaborators on the FP Complete Slack instance as well. However, for quite a while, we've been trying to move away from that, since we'd much prefer a community owned and operated area.
As I mentioned in my last blog post, such a place exists now: the Haskell Foundation Slack. I've created two new chat rooms over there: #stack-users for general discussion, and #stack-collaborators for discussion about working on the code base itself. If you're interested, come join us over there!
And in case anyone is wondering about more broad collaboration between Stack and Haskell Foundation: that's absolutely the plan. We're working through details on that. As someone sitting in both camps (Stack maintainer and Haskell Foundation board member) I can tell you that there's no real impediment to getting more official affiliation, it's simply a matter of times and priorities.
Also: apologies to everyone else out there, you'll now also be able to regularly confuse the words "Stack and Slack" :).
Linux ARM64 builds of Stack
We've had an on-again/off-again relationship with ARM builds of Stack for a while. It's never been much of a code issue; from my recollection, any time someone has tried to build Stack on an ARM machine it's worked without a hitch. The issue instead is getting a good CI setup working. GitHub Actions (and Azure Pipelines) really simplified the process of building and testing on Linux, Windows, and OS X. However, ARM builds were always a bit more painful.
That's why I was so excited to read about Oracle Cloud's Ampere A1 chips. It was fairly straightforward to get a GitHub Actions runner onto one of these machines and configure CI task to generate ARM builds. This is something I've wanted to play with some more for a while, but the relatively weak ARM machines I could get access to always made this tedious. Getting a successful CI build in under an hour is a game changer.
Some caveats, however. This is only a Linux, dynamically linked, ARM64 build. There's no Windows support here, or Alpine Linux (like we use for x86 Linux builds), or 32-bit ARM support. And this is all early days, and may not last. Hopefully this simply runs smoothly.
We have been looking for an opportunity to switch to GHC-9 on our nightly builds and have decided
that GHC-9 has been out long enough and we need to get on with preparing the ecosystem for its
eventual deployment as the Haskell toolchain of choice. To that end, in the coming weeks we will be
releasing LTS 18 based on GHC 8.10.4 (or GHC 8.10.5, should it make
an appearance) and
switching nightly to GHC 9.0.1 (or GHC 9.0.2 if it is released in time).
We can expect many packages to be removed from nightly due to build constraints being violated, or
even real API breakage, and we will need your help getting everything back into the new release.
We have prepared a ghc-9 branch which will become the first GHC 9 nightly and encourage you to check the diff to see if any of your packages are subject to removal. If you’d like them to be included in the initial GHC 9 nightly, please file a PR against that branch (example)
i dislike Haskell's syntax for type signatures because it requires writing the variable name twice (violating the adage Don't Repeat Yourself), and requires counting to figure out which type corresponds to which argument:
myfunction :: Typea -> Typeb -> Typec -> Typed -> Typee; myfunction x y z = ...;
one can attach type annotations to the arguments (with the ScopedTypeVariables LANGUAGE pragma), but that Repeats Yourself even more:
function declarations in C keep type information closely attached to variables, including return type, albeit in C's confusing syntax for types.
here is a somewhat ugly way to accomplish this without needing new syntax (other than ScopedTypeVariables):
myfunction (x :: Typea) (y :: Typeb) (z :: Typec) = returns :: Typed -> Typee where returns = ...;
you'll need -Wno-missing-signatures to quell ghc's "Top-level binding with no type signature" warning. of course, doing that deprives you of that warning when you actually don't have enough type annotation on a top-level binding, perhaps accidentally introducing more polymorphism than you intended.
because they are not top-level bindings, you can do this in let and where blocks with impunity.
if you do this nested, you'll get a warning (when -Wall) "The binding for 'returns' shadows the existing binding". -Wno-name-shadowing quells the warning. or, you could use different names for each returns variables, but that opens the possibility of using the wrong one with a typo.
type annotations on function arguments are not perfect. for example, this compiles:
f (x :: [a]) (y :: [b]) = x++y;
if you need typeclass contexts or forall, this will also not work: use traditional type signatures for fancy type stuff. but we can imagine a language extension that does allow fancy type features to work with type annotations on function arguments:
f (FANCYTYPESTUFF forall a m . (Context m)) (x :: m a) (y :: m a) = ...;
2021-05-24, Monday: Map.partitionWithKey's discriminator is p :: k -> a -> Bool But I have a function that discriminates only on the key: part :: k -> Bool write a function that translates my discriminator that can be used by Map.partitionWithKey: g :: (k -> Bool) -> (k -> a -> Bool)
Social Justice Cleric @noaheasterly: g = (const .)
2021-05-09, Sunday:
THE SEQUEL!
Okay, kinda the same, ... but not:
You have: f :: m a You want: g :: b -> m b
Where g runs f, but accepts an argument, b. g drops the result of f (... on the floor? idk) g returns the argument b lifted to the domain, m
GO!
Denis Stoyanov Ant @xgrommx: g (phantom . pure) This is just joke)
Social Justice Cleric @noaheasterly: (f $>)
2021-05-08, Saturday: two-parter
You have f :: a -> m b You want g :: a -> m a
That is to say: g is a function that returns the input and drops the output of f.
so:
blah :: (a -> m b) -> (a -> m a)
What is a gooder name for the blah-function?
Jonathan Cast #AJAA #Resist @jonathanccast: returnArg = (*>) <$> f <*> return
This post has nothing to do with marriage. Tying the knot is, in my opinion at least, a relatively obscure technique you can use in Haskell to address certain corner cases. I've used it myself only a handful of times, one of which I'll reference below. I preface it like this to hopefully make clear: tying the knot is a fine technique to use in certain cases, but don't consider it a general technique that you should need regularly. It's not nearly as generally useful as something like Software Transactional Memory.
That said, you're still interested in this technique, and are still reading this post. Great! Let's get started where all bad Haskell code starts: C++.
Doubly linked lists
Typically I'd demonstrate imperative code in Rust, but it's not a good idea for this case. So we'll start off with a very simple doubly linked list implementation in C++. And by "very simple" I should probably say "very poorly written," since I'm out of practice.
Anyway, reading the entire code isn't necessary to get the point across. Let's look at some relevant bits. We define a node of the list like this, including a nullable pointer to the previous and next node in the list:
When you add the first node to the list, you set the new node's previous and next values to NULL, and the list's first and last values to the new node. The more interesting case is when you already have something in the list. To add a new node to the back of the list, you need some code that looks like the following:
For those (like me) not fluent in C++, I'm making three mutations:
Mutating the new node's prev member to point to the currently last node of the list.
Mutating the currently last node's next member to point at the new node.
Mutating the list itself so that its last member points to the new node.
Point being in all of this: there's a lot of mutation going on in order to create a double linked list. Contrast that with singly linked lists in Haskell, which are immutable data structures and require no mutation at all.
Anyway, I've written my annual quota of C++ at this point, it's time to go back to Haskell.
RIIH (Rewrite it in Haskell)
Using IORefs and lots of IO calls everywhere, it's possible to reproduce the C++ concept of a mutable doubly linked list in Haskell. Full code is available in a Gist, but let's step through the important bits. Our core data types look quite like the C++ version, but with IORef and Maybe sprinkled in for good measure:
data Node a = Node
{ prev :: IORef (Maybe (Node a))
, value :: a
, next :: IORef (Maybe (Node a))
}
data List a = List
{ first :: IORef (Maybe (Node a))
, last :: IORef (Maybe (Node a))
}
And adding a new value to a non-empty list looks like this:
Notice that, like in the C++ code, we need to perform mutations on the existing node and the last member of the list.
This certainly works, but it probably feels less than satisfying to a Haskeller:
I don't love the idea of mutations all over the place.
The code looks and feels ugly.
I can't access the values of the list from pure code.
So the challenge is: can we write a doubly linked list in Haskell in pure code?
Defining our data
I'll warn you in advance. Every single time I've written code that "ties the knot" in Haskell, I've gone through at least two stages:
This doesn't make any sense, there's no way this is going to work, what exactly am I doing?
Oh, it's done, how exactly did that work?
It happened while writing the code below. You're likely to have the same feeling while reading this of "wait, what? I don't get it, huh?"
Anyway, let's start off by defining our data types. We didn't like the fact that we had IORef all over the place. So let's just get rid of it!
data Node a = Node
{ prev :: Maybe (Node a)
, value :: a
, next :: Maybe (Node a)
}
data List a = List
{ first :: Maybe (Node a)
, last :: Maybe (Node a)
}
We still have Maybe to indicate the presence or absence of nodes before or after our own. That translation is pretty easy. The problem is going to arise when we try to build such a structure, since we've seen that we need mutation to make it happen. We'll need to rethink our API to get going.
Non-mutable API
The first change we need to consider is getting rid of the concept of mutation in the API. Previously, we had functions like pushBack and popBack, which were inherently mutating. Instead, we should be thinking in terms of immutable data structures and APIs.
We already know all about singly linked lists, the venerable [] data type. Let's see if we can build a function that will let us construct a doubly linked list from a singly linked list. In other words:
buildList :: [a] -> List a
Let's knock out two easy cases first. An empty list should end up with no nodes at all. That clause would be:
buildList [] = List Nothing Nothing
The next easy case is a single value in the list. This ends up with a single node with no pointers to other nodes, and a first and last field that both point to that one node. Again, fairly easy, no knot tying required:
buildList [x] =
let node = Node Nothing x Nothing
in List (Just node) (Just node)
OK, that's too easy. Let's kick it up a notch.
Two-element list
To get into things a bit more gradually, let's handle the two element case next, instead of the general case of "2 or more", which is a bit more complicated. We need to:
Construct a first node that points at the last node
Construct a last node that points at the first node
Construct a list that points at both the first and last nodes
Step (3) isn't too hard. Step (2) doesn't sound too bad either, since presumably the first node already exists at that point. The problem appears to be step (1). How can we construct a first node that points at the second node, when we haven't constructed the second node yet? Let me show you how:
buildList [x, y] =
let firstNode = Node Nothing x (Just lastNode)
lastNode = Node (Just firstNode) y Nothing
in List (Just firstNode) (Just lastNode)
If that code doesn't confuse or bother you you've probably already learned about tying the knot. This seems to make no sense. I'm referring to lastNode while constructing firstNode, and referring to firstNode while constructing lastNode. This kind of makes me think of an Ouroboros, or a snake eating its own tail:
In a normal programming language, this concept wouldn't make sense. We'd need to define firstNode first with a null pointer for next. Then we could define lastNode. And then we could mutate firstNode's next to point to the last node. But not in Haskell! Why? Because of laziness. Thanks to laziness, both firstNode and lastNode are initially created as thunks. Their contents need not exist yet. But thankfully, we can still create pointers to these not-fully-evaluated values.
With those pointers available, we can then define an expression for each of these that leverages the pointer of the other. And we have now, successfully, tied the knot.
Expanding beyond two
Expanding beyond two elements follows the exact same pattern, but (at least in my opinion) is significantly more complicated. I implemented it by writing a helper function, buildNodes, which (somewhat spookily) takes the previous node in the list as a parameter, and returns back the next node and the final node in the list. Let's see all of this in action:
buildList (x:y:ys) =
let firstNode = Node Nothing x (Just secondNode)
(secondNode, lastNode) = buildNodes firstNode y ys
in List (Just firstNode) (Just lastNode)
-- | Takes the previous node in the list, the current value, and all following
-- values. Returns the current node as well as the final node constructed in
-- this list.
buildNodes :: Node a -> a -> [a] -> (Node a, Node a)
buildNodes prevNode value [] =
let node = Node (Just prevNode) value Nothing
in (node, node)
buildNodes prevNode value (x:xs) =
let node = Node (Just prevNode) value (Just nextNode)
(nextNode, lastNode) = buildNodes node x xs
in (node, lastNode)
Notice that in buildList, we're using the same kind of trick to use secondNode to construct firstNode, and firstNode is a parameter passed to buildNodes that is used to construct secondNode.
Within buildNodes, we have two clauses. The first clause is one of those simpler cases: we've only got one value left, so we create a terminal node that points back at previous. No knot tying required. The second clause, however, once again uses the knot tying technique, together with a recursive call to buildNodes to build up the rest of the nodes in the list.
The full code is available as a Gist. I recommend reading through the code a few times until you feel comfortable with it. When you have a good grasp on what's going on, try implementing it from scratch yourself.
Limitation
It's important to understand a limitation of this approach versus both mutable doubly linked lists and singly linked lists. With singly linked lists, I can easily construct a new singly linked list by consing a new value to the front. Or I can drop a few values from the front and cons some new values in front of that new tail. In other words, I can construct new values based on old values as much as I want.
Similarly, with mutable doubly linked lists, I'm free to mutate at will, changing my existing data structure. This behaves slightly different from constructing new singly linked lists, and falls into the same category of mutable-vs-immutable data structures that Haskellers know and love so well. If you want a refresher, check out:
None of these apply with a tie-the-knot approach to data structures. Once you construct this doubly linked list, it is locked in place. If you try to prepend a new node to the front of this list, you'll find that you cannot update the prev pointer in the old first node.
There is a workaround. You can construct a brand new doubly linked list using the values in the original. A common way to do this would be to provide a conversion function back from your List a to a [a]. Then you could append a value to a doubly linked list with some code like:
However, unlike singly linked lists, we lose all possibilities of data sharing, at least at the structure level (the values themselves can still be shared).
Why tie the knot?
That's a cool trick, but is it actually useful? In some situations, absolutely! One example I've worked on is in the xml-conduit package. Some people may be familiar with XPath, a pretty nice standard for XML traversals. It allows you to say things like "find the first ul tag in document, then find the p tag before that, and tell me its id attribute."
A simple implementation of an XML data type in Haskell may look like this:
data Element = Element Name (Map Name AttributeValue) [Node]
data Node
= NodeElement Element
| NodeContent Text
Using this kind of data structure, it would be pretty difficult to implement the traversal that I just described. You would need to write logic to keep track of where you are in the document, and then implement logic to say "OK, given that I was in the third child of the second child of the sixth child, what are all of the nodes that came before me?"
Instead, in xml-conduit, we use knot tying to create a data structure called a Cursor. A Cursor not only keeps track of its own contents, but also contains a pointer to its parent cursor, its predecessor cursors, its following cursors, and its child cursors. You can then traverse the tree with ease. The traversal above would be implemented as:
#!/usr/bin/env stack
-- stack --resolver lts-17.12 script
{-# LANGUAGE OverloadedStrings #-}
import qualified Text.XML as X
import Text.XML.Cursor
main :: IO ()
main = do
doc <- X.readFile X.def "input.xml"
let cursor = fromDocument doc
print $ cursor $// element "ul" >=> precedingSibling >=> element "p" >=> attribute "id"
You can test this out yourself with this sample input document:
Like most techniques in programming in general, and Haskell in particular, it can be tempting to go off and look for a use case to throw this technique at. The use cases definitely exist. I think xml-conduit is one of them. But let me point out that it's the only example I can think of in my career as a Haskeller where tying the knot was a great solution to the problem. There are similar cases out there that I'd include too (such as JSON document traversal).
Is it worth learning the technique? Yeah, definitely. It's a mind-expanding move. It helps you internalize concepts of laziness just a bit better. It's really fun and mind-bending. But don't rush off to rewrite your code to use a relatively niche technique.
If anyone's wondering, this blog post came out of a question that popped up during a Haskell training course. If you'd like to come learn some Haskell and dive into weird topics like this, come find out more about FP Complete's training programs. We're gearing up for some intermediate Haskell and Rust courses soon, so add your name to the list if you want to get more information.
Here are two theorems from game theory, which initially seem to contradict each other. By comparing them, we can uncover some hidden nuance in the situation.
Gibbard and Roth face off!
Gibbard’s Theorem: Strategy is Inevitable
The first theorem, initially by Allan Gibbard and generalized by later work by Gibbard and others, showed that any collective decision-making process must have one of three properties:
There are only two possible outcomes.
There is a dictator, who can choose the outcome regardless of any choices made by anyone else.
It is strategic, meaning that the best way to get what you want depends not just on your preferred outcome but also on anticipating what others will do.
This is a pretty big deal. Think about elections, for example. Assuming we have more than two candidates and don’t have a dictatorship, strategic voting must be possible. That’s too bad, because we’d like to be able to tell people that they can always just vote for their favorite candidate. Alas, this can never be the case. It’s widely known that hopeless third parties can “spoil” the election by siphoning votes from major candidates. Clever systems like instant-runoff are supposed to fix that problem, but Gibbard showed that in general, it can never completely be fixed. (At least, not without accepting a limited two-party system or a dictatorship.)
It’s not strictly part of Gibbard’s theorem, but Gibbard and others showed later, that this remains true even if there is randomness is involved. You can wait until after the voting happens to choose the dictator, or to decide which two outcomes to allow; but ultimately the same conditions apply. That won’t be important for the rest of this article, though.
Roth’s Theorem: DA is strategy-proof!
While Gibbard’s theorem addresses any collective decision-making process, Gale and Shapley’s Deferred Acceptance (DA) algorithm is a solution for a specific kind of decision-making problem: matching. The idea is to match up some set of candidates with some set of positions. Each candidate has preferences about which position they want, and each position has preferences on which candidates they prefer. (The presentation is historically given in terms of marriage, but that presents difficulties when you consider same-sex marriage, gender identity, etc., so let’s call it candidates and positions!)
Gale and Shapley proved that there exists a matching of candidates to positions which is stable, in the sense that no candidate would prefer a position which would also prefer them. (The word “stable” perhaps comes from the fact that such a candidate could then switch to their preferred position, so an unstable matching wouldn’t last long.) They gave a specific algorithm, called DA or Deferred Acceptance, which is described in the previous link. It starts from a list of each candidate and position’s preferences, and generates a stable matching. They went on to prove that the matching it generates is the unique one that all candidates would agree is best.
You might imagine that someone very clever could find a way to get a better position by being untruthful about their preferences with this matching algorithm. For example, if someone wants a position but doesn’t think they’ll be accepted, would it be wise to instead express a more realistic preference? Surprisingly, Alvin Roth showed that this is impossible! The DA algorithm is strategy-proof, in the sense that the best way to get the position you most prefer is to provide your honest-to-goodness true list of preferences.
To strategize or not?
So now we have a dilemma. The stable matching definitely has more than two possible outcomes. It definitely doesn’t have a dictator. So it must, by Gibbard’s theorem, be strategic. But hold on… Roth showed that an honest list of preferences is always best, regardless of the preferences of others. How can both be true? The answer is subtle, and it will show that we need to be careful about exactly what words mean in each case.
Here’s the key:
Gibbard’s theorem assumes that each participant has a set of preferences about the entire outcome of the process. In this case, the outcome of the process is the matching of all candidates to all positions.
Roth, on the other hand, assumes that each candidate has a set of preferences about which position they individually will be assigned to. They are not entitled to a preference about which positions are given to which other candidates.
Roth’s preferences are limited enough that they only produce conflict indirectly, such as when there are more candidates who want a certain position than there are slots to accept them. And that turns out to yield just enough slack to escape the consequences of Gibbard’s theorem, and allow the decision process to be non-strategic in Roth’s sense.
But be careful! This means that if candidates do have preferences that depend on the placement of others, Roth’s result no longer applies. To take a simple example, suppose two close friends want to work together in the same position. Despite Roth’s dismissal of strategy, they will need to strategize to accomplish this. For example, they would be well-advised to favor applying for positions that are likely to be generally unpopular. They should also avoid applying to positions that wouldn’t want also want their friend. Both of these are strategic choices that involve anticipating the preferences of others.
In the end, of course, Gibbard and Roth’s results can co-exist, but one must be careful in exactly what they mean.
Some final notes
I want to be clear that this isn’t new. In fact, Roth cited Gibbard’s result in his paper on the matching problem, and is obviously aware of this nuance.
For this analysis, I’m considering only candidates as the participants in the decision-making process, and the employer preferences as being baked into the decision algorithm. Roth’s result does not extend to employers expressing their true preferences! In fact, Roth shows that no process can exist that also incentivizes employers to indicate their true preferences among candidates.
This is a repost of this blog post originally published on the Nomadic Labs blog.
The consensus algorithm is a crucial part of any blockchain project. Because
of the distributed nature of blockchains, different nodes can have different
ideas of what the current state of the blockchain is supposed to be. The
role of the consensus algorithm is to decide which of these possible states,
called forks or branches, will be selected globally. In the world of
distributed systems, there are two distinctive families of consensus
algorithms: Nakamoto-style and BFT-style. Most blockchain solutions use
Nakamoto-style algorithms that allow the existence of any number of
forks of any length, but make longer forks increasingly unstable, so that
they eventually collapse to a single branch. We say that these algorithms
have probabilistic finality. Byzantine fault tolerance
(BFT) algorithms have deterministic finality. They stipulate
definite conditions that must be fulfilled for a block to become final.
Nomadic Labs is intending to propose
Tenderbake — a BFT-style algorithm — as the next
consensus algorithm of Tezos. Its deterministic finality allows us to make
solid claims about the period of time that should pass for a transaction to
become final. In Tenderbake, a block becomes final when there are two blocks
on top of it — this is the only condition. So, if the system produces one
block per 15 seconds, a transaction will become final in about 30 seconds.
This is the kind of performance that users expect from a successful
blockchain solution.
Historically, Tenderbake started as a variant of
Tendermint and was subsequently adapted to fit into the
existing Tezos system. However, as time passed, the description from the
paper and its nascent implementation in the Tezos
codebase started to differ. Thus, for a person starting working on
Tenderbake, reading the paper is not enough. It is also hard to study the
algorithm by reading the Tezos code, because the consensus algorithm is not
implemented in isolation from other components of the system. This is where
the Tenderbake simulator project comes into play.
The simulation framework
Nomadic Labs asked Tweag to develop a framework that would be general enough
to model any consensus algorithm (be it BTF-style, Nakamoto-style or in
styles yet to be invented) in a clear way to facilitate onboarding of
newcomers and for exploration of consensus algorithms in the future. The
results of our work can be found in this repository. The
language of choice at Nomadic Labs is OCaml and we saw no reason not to use
it for this project. Similar to the Tezos code itself, the simulator is
distributed under the MIT license.
Principles of operation
We provide here a simplified description that nevertheless should give the
reader an idea of what the framework can do. The project comes with a
guide that explains in detail how the simulation framework works
and how to implement a consensus algorithm in it.
The simulator allows us to observe the evolution of a system that comprises
a collection of nodes — independent processes that do not share memory but
can exchange messages by means of asynchronous broadcast that can be
fine-tuned by the user if desired.
Consensus algorithms in the framework are implemented as event handlers —
functions that are called when an event occurs at a particular node. A call
of an event handler is called an iteration. Event handlers have the
following type signature:
type event_handler =Algorithm.params ->Time.t ->Event.t ->Signature.private_key ->Algorithm.node_state ->Effect.t list *Algorithm.node_state
Let’s go over the arguments of the function:
Algorithm.params is the parameters of the algorithm such as e.g. round
duration in seconds.
Time.t is the current time.
Event.t is the event the node needs to react to. Currently there are two
kinds of events: reception of a message and a “wake up” call that the node
can schedule for itself. Message types are defined per consensus
algorithm.
Signature.private_key is the private key that every node magically
knows. It is used for signing of messages. This is important because the
framework allows us to program and use Byzantine versions of nodes, too.
Algorithm.node_state is the node state. The type is defined per
consensus algorithm.
The return type of an event_handler is an effect list and an updated node
state. An effect in the list can be one of the following:
Broadcast a message to all nodes.
Schedule a wake up call.
Shut down.
Testing
We have written two kinds of tests: stress tests and scenario tests.
Stress tests are about letting an algorithm run for a number of
iterations with a large enough network and realistic propagation of
messages, including messages getting lost and messages arriving out of
order. The framework allows us to specify a set of predicates that must hold
at each iteration in such a test. This way we can determine if an algorithm
satisfies liveness and safety properties.
According to the tests, all models that we have written satisfy both
properties.
Scenario tests are about adjusting propagation of messages and/or
lifetime of nodes in order to model a situation of interest. We can then
inspect execution logs and check whether the nodes behaved in the expected
way. It is easy to do because simulations return typed logs that we can
pattern match on.
Implemented algorithms
We have implemented four consensus algorithms (listed below roughly in order
of increasing complexity), and applied stress tests and scenario tests as
discussed above:
Ouroboros (the simple BFT version), see
src/ouroboros.
Emmy+, see
src/emmy_plus;
this is the current consensus algorithm used by Tezos.
Tenderbake, see
src/tenderbake;
this is the algorithm Nomadic Labs is planning to propose as a future
amendment to the Tezos blockchain.
Every algorithm is explained in its README.md. Our focus was not only
explaining how a particular algorithm works in principle, but also how it
translates to code that the simulator framework can run.
Conclusion
The simulator has already proven useful. People who have tried it out report
that it has helped them understand the algorithm of Tenderbake and
experiment with it. In the future, we can expect new consensus algorithms to
be implemented and explored using this framework. Please, feel free to give
it a try and contribute!
This post collects a random assortment of guidelines I commonly share for how to organize Haskell projects.
Organize modules “vertically”, not “horizontally”
The glib summary of this rule is: don’t create a “Types” or “Constants” module.
“Vertically” organized modules are modules that group related functionality within the same module. For example, vertically-oriented modules for a simple interpreter might be:
A Syntax module
… which contains the concrete syntax tree type and utilities for traversing, viewing, or editing that tree.
A Parsing module
… which contains (or imports/re-exports) the parser type, a parser for the syntax tree, and error messages specific to parsing.
An Infer module
… which contains the the type inference logic, exception types, and error messages specific to type-checking.
An Evaluation module
… which logic for evaluating an expression, including possibly a separate data structure for a fully-evaluated abstract syntax tree.
A Pretty module
… which specifies how to pretty-print or otherwise format expressions for display.
“Horizontally” organized modules mean that you organize code into modules based on which language features or imports the code relies upon. For example, horizontally-oriented modules for the same interpreter might be:
A Types module
… which contains almost all types, including the concrete syntax tree, abstract syntax tree, parsing-related types, and exception types.
A Lib module
… which contains almost all functions, including the parsers, type inference code, evaluator, and prettyprinter.
A Constants module
… which contains almost all constants (including all error messages, timeouts, and help text).
An App module
… which contains the main entry point for the program.
There are a few reasons I prefer vertical module organization over horizontal module organization:
Vertically-organized modules are easier to split into smaller packages
For example, in a vertically-organized project I could separate out the Syntax, Parser, and Pretty modules into a separate standalone package, which could be used by other people to create an automatic code formatter for my language without having to depend on the type-checking or evaluation logic.
Conversely, there would be little benefit in separating out a Types module from the equivalent horizontally-organized package. Typically, horizontal modules are so tightly coupled to one another that no subset of the modules is useful in isolation.
The separability of vertical modules is an even bigger feature for proprietary projects that aspire to eventually open source their work. Vertically-organized projects are easier to open source a few modules at a time while keeping the proprietary pieces internal.
Vertically-organized modules tend to promote more granular and incremental build graphs
In a horizontally-organized project, each new type you add to the Types module forces a rebuild of the entire package. In a vertically-organized project, if I completely rewrite the type-checking logic then only other modules that depend on type-checking will rebuild (and typically very few depend on type-checking).
Vertically-organized modules tend to group related changes
A common issue in a horizontally-organized project is that every change touches almost every module, making new contributions harder and making related functionality more difficult to discover. In a vertically-organized project related changes tend to fall within the same module.
Naming conventions
I like to use the convention that the default module to import is the same as the package name, except replacing - with . and capitalizing words.
For example, if your package name is foo-bar-baz, then the default module the user imports to use your package is Foo.Bar.Baz.
Packages following this module naming convention typically do not have module names beginning with Control. or Data. prefixes (unless the package name happens to begin with a control- or data- prefix).
There are a few reasons I suggest this convention:
Users can easily infer which module to import from the package name
It tends to lead to shorter module names
For example, the prettyprinter package recently switched to this idiom, which changed the default import from Data.Text.Prettyprint.Doc to Prettyprinter.
It reduces module naming clashes between packages
The reasoning is that you are already claiming global namespace when naming a package, so should why not also globally reserve the module of the same name, too?
However, this won’t completely eliminate the potential for name clashes for other non-default modules that your package exports.
The “God” library stanza
This is a tip for proprietary projects only: put all of your project’s code into one giant library stanza in your .cabal file, including the entrypoint logic (like your command-line interface), tests, and benchmarks. Then every other stanza in the .cabal file (i.e. the executables, test suites, and benchmarks) should just be a thin wrapper around something exported from one of your “library” modules.
For example, suppose that your package is named foo which builds an executable named bar. Your foo.cabal file would look like this (with only the relevant parts shown):
executable bar hs-source-dirs: bar main-is: Main.hs …
… where src/Foo/Bar.hs would look like this:
-- ./src/Foo/Bar.hs moduleFoo.Barwhere … main ::IO () main = … -- Your real `main` goes here
… and bar/Main.hs is a trivial wrapper around Foo.Bar.main:
-- ./bar/Main.hs moduleMainwhere importqualifiedFoo.Bar main ::IO () main = Foo.Bar.main
This tip specifically works around cabal repl’s poor support for handling changes that span multiple project stanzas (both cabal v1-repl and cabal v2-repl appear to have the problem).
To illustrate the issue, suppose that you use cabal repl to load the executable logic for the project like this:
$ cabal repl exe:bar … *Main>
Now if you change the Foo.Bar module and :reload the REPL, the REPL will not reflect the changes you made. This is a pain whenever you need to test changes that span your library and an executable, your library and a test suite, or your library and a benchmark.
Also, if you load an executable / test suite / benchmark into the REPL that depends on a separate library stanza then cabal repl will force you to generate object code for the library stanza, which is slow. Contrast that with using cabal repl to only load the library stanza, which will be faster because it won’t generate object code.
Moreover, ghcid uses cabal repl to power its fast type-checking loop, which means that ghcid also does not work well if you need to quickly switch between changes to the library stanza and other project stanzas.
The fix to all of these problems is: put all of your project’s logic into the library stanza and use only the library stanza as basis for your interactive development. Everything else (your executables, your test suites, and your benchmarks) is just a trivial wrapper around something exported from the library.
I don’t recommend this solution for open source projects, though. If you do this for a public package then your end users will hate you because your package’s library section will depend on test packages or benchmarking packages that can’t be disabled. In contrast, proprietary codebases rarely care about gratuitous dependencies (in my experience).
Conclusion
Those are all of the tips I can think of at the moment. Leave a comment if you think I missed another common practice.
There are two widely used associative containers in Haskell:
Map from containers
HashMap from unordered-containers
Map is using Ord (total order), and most operations are
\mathcal{O}(\log n)
. HashMap is using Hashable (some hash), and most operations are
\mathcal{O}(1)
. HashMap is obviously better!
It depends.
Sometimes HashMap is the only choice, for example when the keys are Unique which has Hashable instance but does not have Ord one. 1 Let us discard such cases and only consider situations when key type has both Ord and Hashable instances.
The thing we often forget to consider is unordered in unordered-containers. toList :: HashMap k v -> [(k, v)]2 doesn't not respect the equality. The order in the resulting list is arbitrary, it depends on how HashMap is constructed.
We should be aware that we trade stability for performance, it is a trade-off, not a pure win.
And not only on that, hash functions provided by hashableare not stable. They depend on a library version, operating system, architecture. The reason is simple: we should be able to tweak hash functions to whatever is the best for performance related properties (speed and collision resistance is own tradeoff already).
Today released hashable-1.3.2.0 optionally could randomise the hash seed on startup of final executable. If random-init-seed flag is enabled, the initial seed is not a compile-time constant but a value initialized on the first use. I added the random-init-seed flag to help find issues when we (unintentionally) depend on (non-existent) stability of the hash function. I immediately found a bug in the lucid test suite. Even the test-suite took care of different orders already, there was a typo in test case.
I do not recommend enabling that flag in production, only in tests.
Why lucid uses HashMap (for HTML tag attributes) instead of Map. I actually don't know. These attribute dictionaries are usually of "human size", at which point the performance difference doesn't show up. (Maybe a Trie Text would be the best option. I'm not sure.) See my previous blog post on benchmarking discrimination package. List.nubis faster then Set or HashSet based ordNub and hashNub with small inputs, even the member check in List has linear complexity. Constant factors matter. If lucid used Map, its output would be stable. Now it's not, attributes can appear in arbitrary order, and that causes problems, e.g. for dhall-docs.3
Another example is aeson. It is using HashMap for the representation of JSON objects. I don't have any data whether using Map would be worse for performance in typical cases, I doubt. It might matter when JSON contains an actual dictionary mapping from keys to values, but "record objects" are small. Whether key-value mappings would be that big that it mattered, is also not unknown. We can argue that it would be better in worst case, as there is known DDoS attacks on HashMaps with deterministic (and weak, such hashable) hash functions. If we would use stronger hash function, the constant factor will go up, and then Map might be faster and safer by construction.
To conclude, I'm sorry that I broke all of your test-suites with hashable-1.3.1.0 release which changed initial seed of the default hash. On the other hand, I think it was good to realise how much we rely on something we shouldn't. Therefore hashable-1.3.2.0 introduces the random-init-seed flag which you can use in your tests to find those issues. (N.B. I don't consider that flag to be a security feature). That release also finally mentions in haddocks the "known but hidden" fact that hash in hashable is not stable. Don't rely on that.
I suggest you default to Map when you need an associative container, and only consider HashMap when you actually have a case for it Benchmark, don't guess, base your decision on data, rerun these benchmark periodically.
And concurrent one as in unique package cannot have Ord instance.↩︎
If you rely (directly or indirectly) on HashMap.toList or HashSet.toList where the order may leak, they are likely wrong data structures. It is ok to use toList if we remove order-dependency later for example by sorting the data at the end or folding with a commutative operation.↩︎
I think dhall-docs should use e.g. tagsoup to reparse the output, (normalise), and compare it as "tag tree", not as text. That's not very convenient, but it is more correct — even if lucid had fully stable output.↩︎
In this talk from the Cardano 2020 Virtual Summit, I am explaining the design of the custom native asset functionality of Cardano (starting at 18:40min). This is based on the work published in the following two research papers at ISoLA 2020: UTXOma: UTXO with Multi-Asset Support and Native Custom Tokens in the Extended UTXO Model. Custom native assets are supported on the Cardano mainnet since the Mary hardfork that happened in March 2021.
That is, takes a stream of candidate primes; the head of this stream is a prime, and the remaining primes are obtained by removing all multiples of from the candidates and sieving what’s left. (It’s also a nice unfold.)
Unfortunately, as Melissa O’Neill observes, this nifty program is not the true Sieve of Eratosthenes! The problem is that for each prime , every remaining candidate is tested for divisibility. O’Neill calls this bogus common version “trial division”, and argues that the Genuine Sieve of Eratosthenes should eliminate every multiple of without reconsidering all the candidates in between. That is, only of the natural numbers are touched when eliminating multiples of 2, less than of the remaining candidates for multiples of 3, and so on. As an additional optimization, it suffices to eliminate multiples of starting with , since by that point all composite numbers with a smaller nontrivial factor will already have been eliminated.
O’Neill’s paper presents a purely functional implementation of the Genuine Sieve of Eratosthenes. The tricky bit is keeping track of all the eliminations when generating an unbounded stream of primes, since obviously you can’t eliminate all the multiples of one prime before producing the next prime. Her solution is to maintain a priority queue of iterators; indeed, the main argument of her paper is that functional programmers are often too quick to use lists, when other data structures such as priority queues might be more appropriate.
O’Neill’s paper was published as a Functional Pearl in the Journal of Functional Programming, when Richard Bird was the handling editor for Pearls. The paper includes an epilogue that presents a purely list-based implementation of the Genuine Sieve, contributed by Bird during the editing process. And the same program pops up in Bird’s textbook Thinking Functionally with Haskell (TFWH). This post is about Bird’s program.
The Genuine Sieve, using lists
Bird’s program appears in §9.2 of TFWH. It deals with lists, but these will be infinite, sorted, duplicate-free streams, and you should think of these as representing infinite sets, in this case sets of natural numbers. In particular, there will be no empty or partial lists, only properly infinite ones.
Now, the prime numbers are what you get by eliminating the composite numbers from the “plural” naturals (those greater than one), and the composite numbers are the proper multiples of the primes—so the program is cleverly cyclic:
We’ll come back in a minute to , which unions a set of sets to a set; but is the obvious implementation of list difference of sorted streams (hence, representing set difference):
and generates the multiples of starting with :
Thus, the composites are obtained by merging together the infinite stream of infinite streams . You might think that you could have defined instead , but this doesn’t work: this won’t compute the first prime without first computing some composites, and you can’t compute any composites without at least the first prime, so this definition is unproductive. Somewhat surprisingly, it suffices to “prime the pump” (so to speak) just with 2, and everything else flows freely from there.
Returning to , here is the obvious implementation of , which merges two sorted duplicate-free streams into one (hence, representing set union):
Then is basically a stream fold with . You might think you could define this simply by , but again this is unproductive. After all, you can’t merge the infinite stream of sorted streams into a single sorted stream, because there is no least element. Instead, we have to exploit the fact that we have a sorted stream of sorted streams; then the binary merge can exploit the fact that the head of the left stream is the head of the result, without even examining the right stream. So, we define:
This program is now productive, and yields the infinite sequence of prime numbers, using the genuine algorithm of Eratosthenes.
Approximation Lemma
Bird uses the cyclic program as an illustration of the Approximation Lemma. This states that for infinite lists , ,
where
Note that is undefined; the function preserves the outermost constructors of a list, but then chops off anything deeper and replaces it with (undefined). So, the lemma states that to prove two infinite lists equal, it suffices to prove equal all their finite approximations.
Then to prove that does indeed produce the prime numbers, it suffices to prove that
for all , where is the th prime (so ). Bird therefore defines
and claims that
To prove the claim, he observes that it suffices for to be well defined at least up to the first composite number greater than , because then delivers enough composite numbers to supply , which will in turn supply , and so on. It is a “non-trivial result in Number Theory” (in fact, a consequence of Bertrand’s Postulate) that ; therefore it suffices that
where is the th composite number (so ) and . Completing the proof is set as Exercise 9.I of TFWH, and Answer 9.I gives a hint about using induction to show that is the result of merging with . (Incidentally, there is a typo in TFWH: both the body of the chapter and the exercise and its solution have “” instead of “”.)
Unfortunately, the hint in Answer 9.I is simply wrong. For example, it implies that (which equals ) could be constructed from (which equals ) and (which equals ); but this means that the 6 and 8 would have to come out of nowhere. Nevertheless, the claim in Exercise 9.I is valid. What should the hint for the proof have been?
Fixing the proof
The problem is made tricky by the cyclicity. Tom Schrijvers suggested to me to break the cycle by defining
so that ; this allows us to consider in isolation from . In fact, I claim (following a suggestion generated by Geraint Jones and Guillaume Boisseau at our Algebra of Programming research group) that if where , then has a well-defined prefix consisting of all the composite numbers such that and is a multiple of some with , in ascending order, then becomes undefined. That’s not so difficult to see from the definition: contains the multiples of starting from ; all these streams get merged; but the result gets undefined (in ) once we pass the head of the last stream. In particular, has a well-defined prefix consisting of all the composites up to , then becomes undefined, as required.
However, I must say that I still find this argument unsatisfactory: I have no equational proof about the behaviour of . In fact, I believe that the Approximation Lemma is unsufficient to provide such a proof: works along the grain of , but against the grain of . I believe we want something instead like an “Approx While Lemma”, where is to as is to :
Thus, retains elements of as long as they satisfy , but becomes undefined as soon as they don’t (or when the list runs out). Then for all sorted, duplicate-free streams and unbounded streams (that is, ),
The point is that works conveniently for and and hence for too. I am now working on a proof by equational reasoning of the correctness (especially the productivity) of the Genuine Sieve of Eratosthenes…
I'm now writing the second Haskell service using Redis to store data. There are
a few packages on Hackage related to Redis but I only found 2 client libraries,
redis-io and hedis. I must say I like the API of redis-io better, but it breaks
a rule I hold very dear:
Libraries should never log, that's the responsibility of the application.
So, hedis it is. I tried using the API as is, but found it really cumbersome so
looked around and after some inspiration from hedis-simple I came up with the
following functions.
First a wrapper around a Redis function that put everything into ExceptionT
with a function that transforms a reply into an Exception.
lpush::Exception e =>(Reply-> e)->ByteString->[ByteString]->ExceptionTRedisIntegerlpush mapper key element =ExceptionT$ replyToExc <$> R.lpush key element
where
replyToExc = first (toException . mapper)
I found wrapping up functions like this is simple, but repetitive.
Finally I need a way to run the whole thing and unwrap it all back to IO:
runRedis::Connection->ExceptionTRedis a ->IO(EitherSomeException a)runRedis conn = R.runRedis conn . runExceptionT
<quick-intro> In my previous post, I mentioned that we were opening up the Haskell Foundation Slack instance for more people. I'd like to emphasize that a second time, and mention that we are trying to use Slack as a tech track communication hub. We're actively looking for contributors to join up on some of these initiatives and bring them to reality. If the previous topics I've mentioned (like the Vector Types Proposal) or topics like improving Haskell educational material sound interesting, please join the Slack, introduce yourself in #tech-track, and tell us what you'd be interested in helping out with. You'll get to be part of some hopefully significant improvements to the Haskell ecosystem, and have access to a lot of other collaborators and mentors to make this happen. </quick-intro>
Also, we're working to get a unified listing of the active projects and what kind of contributors we're looking for, so stay tuned for that too. Anyway, on to the newest proposal!
Stack
There are currently multiple ways to install Haskell tooling, and not all of them work for everyone. Personally, I've worked on two such methods: first MinGHC (a Windows installer), and later Stack, which subsumed MinGHC. Stack combines both a build tool and a toolchain installer, and manages GHC installations as well as—on Windows—install msys. And if it seems like Windows is coming up a lot here... that's a theme.
Side note What's msys? Why do we need it? msys is essentially a bunch of POSIX utilities ported to run on Windows. The most basic problem is solves is allowing packages with configure scripts, such as network, to be installed on Windows. It can also be used to provide a package manager for getting shared libraries on Windows. For this post, take as a given that having msys on Windows is basically a necessity.
Anyway, with Stack, you have a set of different installers for the Stack binary itself: a GUI installer for Windows or a curl-able script for POSIX-like systems. Stack then can install GHC and msys2 into user-specific directories, and provides commands like stack ghc or stack --resolver ghc-8.10.4 exec to run commands with the appropriate toolchain configured in environment variables.
There are basically three problems with this setup:
Some people really don't like user-specific installs, and want tooling installed system-globally. Or they really want package managers to maintain toolchains. Or a few other such goals. There are inherent problems with some of those goals, such as distributions having out-of-date installers, or permissions management, or others. The per-user toolchain installer has become pretty standards across different languages (Python, Ruby, Rust, etc), and is unlikely to change any time soon.
Some people really don't like typing in stack ghc instead of ghc. I'm overall a fan of needing the wrapper tooling, since it allows you to trivially switch which toolchain you're using, and makes it more blatant that certain environment variables (like STACK_YAML) and files (like stack.yaml) may affect behavior.
Stack isn't the only tool in town. For example, many people want to use cabal-install instead. Stack doesn't have support for downloading cabal-install binaries right now. Even if it did, it's not clear that the cabal-install team will consent to an installer that leverages Stack like this. But there are even more binaries we may wish to install and manage going forward, such as linters, stylers, IDE integration tools, etc.
But if you're using Stack to build your projects, "how do I install GHC and associated tools" is basically a solved problem. Perhaps (2) irritates some people, but we'll get back to that. Perhaps you don't like curl-able scripts, or Windows installers, or user-specific installs. You can go ahead and manually download the Stack executable and use things like { system-ghc: true, install-ghc: false }. But from my experience, most people just use the defaults and are happy enough.
Everything else
But as mentioned, not everyone is on Stack. And there's other tooling we'd like to be able to install. And we don't want to bifurcate our install methods too much. And it's kind of weird that Stack is both a toolchain manager and a build tool. And frankly, it would be nice to finally get a downloads page that everyone can agree with.
There are other approaches to installing toolchains available today. ghcup, Haskell Platform (to some extent), manually ./configure && make installing GHC bindists, distro specific methods, etc. Some of these solves some of the three problems I mentioned with Stack. Some solve other problems as well. Some create new problems. But a recurring theme in particular that the Haskell Foundation members have heard is that the story for non-Stack users on Windows is particularly painful.
Let me clarify that I'm wearing two hats with what I'm about to say:
A maintainer of Stack and the original author of much of Stack's toolchain install logic, who would always rather see the Stack codebase size go down instead of up.
A member of the Haskell Foundation board, who is looking to improve the Haskell adoption story.
So the problem statement: can't we standardize an install method that meets most people's goal, most of the time?
The goal
I'll start by laying out the ultimate goal that's in my head. This idea's been bouncing around for a few months now, but only came up in May 4, 2021 tech track meeting for the Haskell Foundation. I don't think it's that ambitious of a goal. But in the next section, I'm also going to lay out some shorter term milestones.
What I'd like to see is a well maintained Haskell library for installing Haskell toolchains, generally acknowledged as the de facto standard way to install GHC and other tools. It should be Haskell, so that it's easy for Haskellers to maintain. It should support as many operating systems as possible. It should provide an easy-to-use Haskell API that other tooling can leverage.
In addition, we should have some executables built against that library that provide a nice command line interface for managing toolchains. Some examples of these executables would be:
The primary installer executable, that let's you say things like "install GHC 9.0.1" and "run the following command with env vars set so that it will use GHC 9.0.1"
A ghc/ghc.exe shim executable that does the same thing as above, but doesn't require a longer command line like haskell-install --ghc 9.0.1 --install-ghc exec -- ghc -O2 Main.hs. Instead, with this ghc/ghc.exe shim, all of that would be implied. How does it know what GHC version to use? How do you override it? All fair questions I'm going to gloriously punt on for now.
We can build out additional shim executables for other common tools, like runghc, or ghci, or... and this is the cool part... shim executables for Stack and cabal-install.
Next, we're going to build installers for all of these executables. On Windows, it will be a GUI installer. On Linux, OS X, and others, we can take our pick, though starting with curl-able scripts makes the most sense in my opinion.
Stack no longer will have logic in its codebase for installing GHC. Instead, it will rely on this new library. Or, perhaps, it won't even do that. Perhaps it will be the new tool's job to ensure the correct GHC is available for Stack invocations. I'm not quite sure.
cabal-install doesn't currently maintain a GHC toolchain, so I think it would fit into this model perfectly without any need for modification.
The milestones
The biggest pain point we want to immediately solve is to get a Windows installer out the door. The short term goal for this is to provide a brand new executable, build against Stack.Setup, and build an installer around it. Maybe that installer will include recent cabal-install and Stack executables. The details on what exactly it will do are still slightly fuzzy. But the point is: we're just writing an executable that reuses existing installer logic within Stack's library.
The next step is creating the new installer library. My recommendation is to simply cleave off the aforementioned Stack.Setup module into a new package and then modify as needed. Modifications may include adding support for installing other tools, updating the location of config files for specifying toolchain binaries, or more. But Stack.Setup is a module that's been around for a long time now, and has worked out a lot of these details. I'd advise against recreating the wheel here.
With that in place, we can begin to work on the other shim executables mentioned above, and begin to look at how this new installer library and the shim executables interact with the rest of the installer ecosystem. That's an open discussion yet to happen.
And in parallel to that, the Stack codebase itself can begin its move over to the new library.
Inspiration
I'm trying to avoid sounding like a broken record, but a lot of the inspiration for my thoughts here come from rustup. Honestly though, most of the ideas of how this installer should work predate Rust, and come from a combination of experience with MinGHC and seeing how tools like virtualenv and RVM work.
Get involved!
I'm going to put this blog post on Discourse, as I've been doing with previous posts. But I'm going to strongly encourage anyone interested in this to join up on Slack too. We've created a new channel for this topic, #tech-track-installer. If you're interested in the topic, please join up!
I'm not just asking for people to provide some feedback, though that's welcome. I'm looking for people to contribute, work on code, and more broadly, take ownership of this work. I think this is a relatively small, but massively vital, piece of Haskell infrastructure we're trying to solidify. Someone looking to get deeper into open source maintenance would be great. And you'll have plenty of people available to provide guidance when desired.
Stack/cabal-install/GHC mapping
One final idea I was playing with, and thought I'd leave as more of a footnote. One of the painpoints in maintaining build tooling (Stack or cabal-install) is maintaining backwards compatibility with old versions of GHC. However, other languages don't always do that. For example, with Rust, each new compiler version gets a new Cargo (the build tool) version:
With a toolchain installer in place like I'm describing, we could end up in that reality for Haskell too. Perhaps we could have a config file that essentially says, for each GHC version, what the recommended version is of cabal-install, Stack, msys2, stylish-haskell, and a slew of other tools. You could override it if desired, but the default would be the right thing for most people. Talking about a unified "version 9.2.1" Haskell toolchain that represents the compiler, standard libraries, and add-on tooling sounds like a huge step forward for easy adoption, at least to me.
I'm launching a new podcast, the PyTorch Developer Podcast. The idea is to be a place for the PyTorch dev team to do bite sized (10-20 min) topics about all sorts of internal development topics in PyTorch. For now, it's just me monologuing for fifteen minutes about whatever topic I decide. The plan is to release an episode daily, five days a week, until I run out of things to say (probably not for a while, I have SO MANY THINGS TO SAY). I don't edit the podcasts and do minimal planning, so they're a bit easier to do than blog posts. Check it out! There's two episodes out already, one about how we do Python bindings for our C++ objects and another about history and constraints of the dispatcher. If there are any topics you'd like me to cover, give a shout.
This post documents a small trick that I use to avoid deeply-nested error-handling code. This trick is a common piece of Haskell folklore that many people either learn from others or figure out on their own, but I’m not sure what the official name of this trick is (so I had difficulty searching for prior art explaining this trick). However, I’ve taught this trick to others enough times that I think it merits a blog post of its own.
This post assumes some familiarity with Haskell’s Either type and do notation, but the Appendix at the end of the post will walk through all of the details using equational reasoning if you’re having trouble following along with how things work.
The motivating example
Let’s begin from the following contrived Either-based example that uses deeply nested error-handling:
{-# LANGUAGE NamedFieldPuns #-} importText.Read (readMaybe) dataPerson=Person { age ::Int, alive ::Bool } deriving (Show) example ::String->String->EitherStringPerson example ageString aliveString =do case readMaybe ageString of Nothing->do Left"Invalid age string" Just age ->do if age <0 thendo Left"Negative age" elsedo case readMaybe aliveString of Nothing->do Left"Invalid alive string" Just alive ->do returnPerson{ age, alive }
… which we can use like this:
>>> example "24""True" Right (Person {age =24, alive =True}) >>> example "24""true" Left"Invalid alive string" >>> example """True" Left"Invalid age string" >>> example "-5""True" Left"Negative age"
Notice how the above coding style increases the nesting / indentation level every time we parse or validate the input in some way. We could conserve indentation by using 2-space indents or indenting only once for each level of nesting:
{-# LANGUAGE NamedFieldPuns #-} importText.Read (readMaybe) dataPerson=Person { age ::Int, alive ::Bool } deriving (Show) example ::String->String->EitherStringPerson example ageString aliveString =case readMaybe ageString of Nothing->Left"Invalid age string" Just age ->if age <0 thenLeft"Negative age" elsecase readMaybe aliveString of Nothing->Left"Invalid alive string" Just alive ->returnPerson{ age, alive }
However, I think most people writing code like that would prefer to keep the indentation level stable, no matter how many validations the code requires.
The trick
Fortunately, we can avoid nesting the code with the following change:
{-# LANGUAGE NamedFieldPuns #-} importText.Read (readMaybe) dataPerson=Person { age ::Int, alive ::Bool } deriving (Show) example ::String->String->EitherStringPerson example ageString aliveString =do age <-case readMaybe ageString of Nothing->Left"Invalid age string" Just age ->return age if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe aliveString of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive }
Here we make use of several useful properties:
return in Haskell does not short-circuit or exit from the surrounding code
In fact, some Haskell programmers prefer to use pure (a synonym for return) to better convey the absence of short-circuiting behavior:
{-# LANGUAGE NamedFieldPuns #-} importText.Read (readMaybe) dataPerson=Person { age ::Int, alive ::Bool } deriving (Show) example ::String->String->EitherStringPerson example ageString aliveString =do age <-case readMaybe ageString of Nothing->Left"Invalid age string" Just age ->pure age if age <0 thenLeft"Negative age" elsepure () alive <-case readMaybe aliveString of Nothing->Left"Invalid alive string" Just alive ->pure alive purePerson{ age, alive }
Leftdoes short-circuit from the surrounding code
In fact, we can define a synonym for Left named throw if we want to better convey the presence of short-circuiting behavior::
{-# LANGUAGE NamedFieldPuns #-} importText.Read (readMaybe) dataPerson=Person { age ::Int, alive ::Bool } deriving (Show) example ::String->String->EitherStringPerson example ageString aliveString =do age <-case readMaybe ageString of Nothing-> throw "Invalid age string" Just age ->pure age if age <0 then throw "Negative age" elsepure () alive <-case readMaybe aliveString of Nothing-> throw "Invalid alive string" Just alive ->pure alive purePerson{ age, alive } throw ::String->EitherString a throw =Left
Left / throw “return” any type of value
If you look at the type of throw, the “return” type is a polymorphic type a because throw short-circuits (and therefore doesn’t need to return a real value of that type).
This is why the type checker doesn’t complain when we do this:
age <-case readMaybe ageString of Nothing-> throw "Invalid age string" Just age ->pure age
… because both branches of the case expression type-check as an expression that “return”s an Int. The left branch type-checks as a branch that returns an Int because it cheats and never needs to return anything and the right branch returns a bona-fide Int (the age).
We can make this explicit by giving both branches of the case expression an explicit type annotation:
age <-case readMaybe ageString of Nothing-> throw "Invalid age string" ::EitherStringInt Just age ->pure age ::EitherStringInt
This means that both branches of a case expression will always share the same return type if at least one branch is a Left / throw. Similarly, both branches of an if expression will share the same return type if at least one branch is a Left / throw:
if age <0 then throw "Negative age" ::EitherString () elsepure () ::EitherString ()
We can return a value from a case expression
New Haskell programmers might not realize that case expressions can return a value (just like any other expression), which means that their result can be stored as a new value within the surrounding scope:
age <-case readMaybe ageString of Nothing-> throw "Invalid age string" ::EitherStringInt Just age ->pure age ::EitherStringInt
The type-checker doesn’t mind that only the second branch returns a valid age because it knows that the outer age is unreachable if the first branch short-circuits. This understanding is not built into the compiler, but is rather an emergent property of how do notation works for Either. See the Appendix for a fully-worked equational reasoning example showing why this works.
Combinators
You can build upon this trick by defining helpful utility functions to simplify things further. For example, I sometimes like to define the following helper function:
orDie ::Maybe a ->String->EitherString a Just a `orDie` _ =return a Nothing`orDie` string =Left string {- Equivalent, more explicit, implementation: maybe `orDie` string = case maybe of Nothing -> Left string Just a -> return a -}
… which you can use like this:
{-# LANGUAGE NamedFieldPuns #-} importText.Read (readMaybe) dataPerson=Person { age ::Int, alive ::Bool } deriving (Show) orDie ::Maybe a ->String->EitherString a Just a `orDie` _ =Right a Nothing`orDie` string =Left string example ::String->String->EitherStringPerson example ageString aliveString =do age <- readMaybe ageString `orDie`"Invalid age string" if age <0 thenLeft"Negative age" elsereturn () alive <- readMaybe aliveString `orDie`"Invalid alive string" returnPerson{ age, alive }
… which is even more clear (in my opinion).
Conclusion
That is the entirety of the trick. You can return values from case expressions to avoid deeply-nesting your Either code, or you can define utility functions (such as orDie) which do essentially the same thing.
This trick applies equally well to any other Monad that supports some notion of short-circuiting on failure, such as ExceptT (from transformers / mtl). The only essential ingredient is some throw-like function that short-circuits and therefore has a polymorphic return type.
Appendix
You can build a better intuition for why this works using equational reasoning. We’ll begin from an example usage of our function and at each step of the process we will either desugar the code or substitute one subexpression with another equal subexpression.
In all of the examples, we’ll begin from this definition for the example function:
example ::String->String->EitherStringPerson example ageString aliveString =do age <-case readMaybe ageString of Nothing->Left"Invalid age string" Just age ->return age if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe aliveString of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive }
Let’s first begin with the example that fails the most quickly:
example """True" -- Substitute `example` with its definition: -- -- … replacing `ageString` with `""` -- -- … replacing `aliveString` with `"True"` do age <-case readMaybe ""of Nothing->Left"Invalid age string" Just age ->return age if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Definition of `readMaybe` (not shown): -- -- (readMaybe "" :: Maybe Int) = Nothing do age <-caseNothingof Nothing->Left"Invalid age string" Just age ->return age if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Simplify the first `case` expression do age <-Left"Invalid age string" if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Desugar `do` notation one step -- -- (do age <- m; n) = (m >>= \age -> n) Left"Invalid age string">>= \age -> doif age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Definition of `>>=` for `Either` -- -- (Left x >>= _) = (Left x) Left"Invalid age string"
… and we’re done! The key step is the last bit where we simplify (Left … >>= _) to (Left …), which is how Either short-circuits on failure. Because this simplification does not use the downstream code everything type-checks just fine even though we never supply a valid age.
For completeness, let’s also walk through the example where everything succeeds:
example "24""True" -- Substitute `example` with its definition: -- -- … replacing `ageString` with `"24"` -- -- … replacing `aliveString` with `"True"` do age <-case readMaybe "24"of Nothing->Left"Invalid age string" Just age ->return age if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Definition of `readMaybe` (not shown): -- -- (readMaybe "24" :: Maybe Int) = (Just 24) do age <-caseJust24of Nothing->Left"Invalid age string" Just age ->return age if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Simplify the first `case` expression do age <-return24 if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- return = Right do age <-Right24 if age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Desugar `do` notation one step -- -- (do age <- m; n) = (m >>= \age -> n) Right24>>= \age -> doif age <0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age, alive } -- Definition of `>>=` for `Either` -- -- (Right x >>= f) = (f x) -- -- This means that we substitute `age` with `24` doif24<0 thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- (24 < 0) = False doifFalse thenLeft"Negative age" elsereturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- (if False then l else r) = r doreturn () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- return = Right doRight () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- (do m; n) = (do _ <- m; n) do _ <-Right () alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- Desugar `do` notation one step -- -- (do age <- m; n) = (m >>= \age -> n) Right () >>= _ -> do alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- Definition of `>>=` for `Either` -- -- (Right x >>= f) = (f x) -- -- This means that we substitute `_` (unused) with `()` do alive <-case readMaybe "True"of Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- Definition of `readMaybe` (not shown): -- -- (readMaybe "True" :: Maybe Bool) = (Just True) do alive <-caseJustTrueof Nothing->Left"Invalid alive string" Just alive ->return alive returnPerson{ age =24, alive } -- Simplify the `case` expression do alive <-returnTrue returnPerson{ age =24, alive } -- return = Right do alive <-RightTrue returnPerson{ age =24, alive } -- Desugar `do` notation one step -- -- (do age <- m; n) = (m >>= \age -> n) RightTrue>>= \alive -> doreturnPerson{ age =24, alive } -- Definition of `>>=` for `Either` -- -- (Right x >>= f) = (f x) -- -- This means that we substitute `alive` with `True` doreturnPerson{ age =24, alive =True } -- Desugar `do` notation -- -- do m = m returnPerson{ age =24, alive =True } -- return = Right RightPerson{ age =24, alive =True }
As an exercise, you can try to extrapolate between those two examples to reason through what happens when we evaluate the remaining two examples which fail somewhere in between:
Quickstrom uses linear temporal logic (LTL) for specifying web applications. When explaining how it works, I’ve found that the basics of LTL are intuitive to newcomers. On the other hand, it’s not obvious how to specify real-world systems using LTL. That’s why I’m sharing some of my learnings and ideas from the past year in the form of blog posts.
This post focuses on how to use LTL to specify systems in terms of state machines. It’s a brief overview that avoids going into too much detail. For more information on how to test web applications using such specifications, see the Quickstrom documentation.
To avoid possible confusion, I want to start by pointing out that a state machine specification in this context is not the same as a model in TLA+ (or similar modeling languages.) We’re not building a model to prove or check properties against. Rather, we’re defining properties in terms of state machine transitions, and the end goal is to test actual system behavior (e.g. web applications, desktop applications, APIs) by checking that recorded traces match our specifications.
Linear Temporal Logic
In this post, we’ll be using an LTL language. It’s a sketch of a future specification language for Quickstrom.
A formula (plural formulae) is a logical expression that evaluates to true or false. We have the constants:
true
false
We combine formulae using the logical connectives, e.g:
&&
||
not
==>
The ==> operator is implication. So far we have propositional logic, but we need a few more things.
Temporal Operators
At the core of our language we have the notion of state. Systems change state over time, and we’d like to express that in our specifications. But the formulae we’ve seen so far do not deal with time. For that, we use temporal operators.
To illustrate how the temporal operators work, I’ll use diagrams to visualize traces (sequences of states). A black circle denotes a state in which the formula is true, and a white circle denotes a state where the formula is false.
For example, let’s say we have two formulae, P and Q, where:
P is true in the first and second state
Q is true in the second state
Both formulae are false in all other states. The formulae and trace would be visualized as follows:
P ●───●───○
Q ○───●───○
Note that in these diagrams, we assume that the last state repeats forever. This might seem a bit weird, but drawing an infinite number of states is problematic.
All of the examples explaining operators have links to the Linear Temporal Logic Visualizer, in which you can interactively experiment with the formulae. The syntax is not the same as in the article, but hopefully that’s not a problem.
Next
The next operator takes a formula as an argument and evaluates it in the next state.
The next operator is relative to the current state, not the first state in the trace. This means that we can nest nexts to reach further into the future.
next (next P) ●───●───○───○───○next P ○───●───●───○───○P ○───○───●───●───○
All right, time for a more concrete example, something we’ll evolve throughout this post. Let’s say we have a formula gdprConsentIsVisible which is true when the GDPR consent screen is visible. We specify that the screen should be visible in the current and next state like so:
gdprConsentIsVisible && next gdprConsentIsVisible
A pair of consecutive states is called a step. When specifying state machines, we use the next operator to describe state transitions. A state transition formula is a logical predicate on a step.
In the GDPR example above, we said that the consent screen should stay visible in both states of the step. If we want to describe a state change in the consent screen’s visibility, we can say:
gdprConsentIsVisible && next (not gdprConsentIsVisible)
The formula describes a state transition from a visible to a hidden consent screen.
Always
But interesting state machines usually have more than one possible transition, and interesting behaviors likely contain multiple steps.
While we could nest formulae containing the next operator, we’d be stuck with specifications only describing a finite number of transitions.
Consider the following, where we like to state that the GDPR consent screen should always be visible:
gdprConsentIsVisible && next (gdprConsentIsVisible && next ...)
This doesn’t work for state machines with cycles, i.e. with possibly infinite traces, because we can only nest a finite number of next operators. We want state machine specifications that describe any number of transitions.
This is where we pick up the always operator. It takes a formula as an argument, and it’s true if the given formula is true in the current state and in all future states.
always P ●───●───●───●───●P ●───●───●───●───●always Q ○───○───●───●───●Q ●───○───●───●───●
Note how always Q is true in the third state and onwards, because that’s when Q becomes true in the current and all future states.
Let’s revisit the always-visible consent screen specification. Instead of trying to nest an infinite amount of next formulae, we instead say:
always gdprConsentIsVisible
Neat! This is called an invariant property. Invariants are assertions on individual states, and an invariant property says that it must hold for every state in the trace.
Always for State Machines
Now, let’s up our game. To specify the system as a state machine, we can combine transitions with disjunction (||) and the always operator. First, we define the individual transition formulae open and close:
let open = not gdprConsentIsVisible && next gdprConsentIsVisible;let close = gdprConsentIsVisible && next (not gdprConsentIsVisible);
Our state machine formula says that it always transitions as described by open or close:
always (open || close)
We have a state machine specification! Note that this specification only allows for transitions where the visibility of the consent screen changes back and forth.
So far we’ve only seen examples of safety properties. Those are properties that specify that “nothing bad happens.” But we also want to specify that systems somehow make progress. The following two temporal operators let us specify liveness properties, i.e. “good things eventually happen.”
Quickstrom does not support liveness properties yet.1
Eventually
We’ve used next to specify transitions, and always to specify invariants and state machines. But we might also want to use liveness properties in our specifications. In this case, we are not talking about specific steps, but rather goals.
The temporal operator eventually takes a formula as an argument, and it’s true if the given formula is true in the current or any future state.
eventually P ○───○───○───○───○P ○───○───○───○───○eventually Q ●───●───●───●───○Q ○───○───○───●───○
For instance, we could say that the consent screen should initially be visible and eventually be hidden:
gdprConsentIsVisible && eventually (not gdprConsentIsVisible)
This doesn’t say that it stays hidden. It may become visible again, and our specification would allow that. To specify that it should stay hidden, we use a combination of eventually and always:
gdprConsentIsVisible && eventually (always (not gdprConsentIsVisible))
Let’s look at a diagram to understand this combination of temporal operators better:
eventually (always P) ○───○───○───○───○P ○───○───●───●───○eventually (always Q) ●───●───●───●───●Q ○───○───●───●───●
The formula eventually (always P) is not true in any state, because P never starts being true forever. The other formula, eventually (always Q), is true in all states because Q becomes true forever in the third state.
Until
The last temporal operator I want to discuss is until. For P until Q to be true, P must be true until Q becomes true.
P until Q ●───●───●───●───○P ●───●───○───○───○Q ○───○───●───●───○
Just as with the eventually operator, the stop condition (Q) doesn’t have to stay true forever, but it has to be true at least once.
The until operator is more expressive than always and eventually, and they can both be defined using until.2
Anyway, let’s get back to our running example. Suppose we have another formula supportChatVisible that is true when the support chat button is shown. We want to make sure it doesn’t show up until after the GDPR consent screen is closed:
not supportChatVisible until not gdprConsentIsVisible
The negations make it a bit harder to read, but it’s equivalent to the informal statement: “the support chat button is hidden at least until the GDPR consent screen is hidden.” It doesn’t demand that the support chat button is ever visible, though. For that, we instead say:
gdprConsentIsVisible until (supportChatVisible && not gdprConsentIsVisible)
In this formula, supportChatVisible has to become true eventually, and at that point the consent screen must be hidden.
Until for State Machines
We can use the until operator to define a state machine formula where the final transition is more explicit.
Let’s say we want to specify the GDPR consent screen more rigorously. Suppose we already have the possible state transition formulae defined:
allowCollectedData
disallowCollectedData
submit
We can then put together the state machine formula:
let gdprConsentStateMachine = gdprConsentIsVisible && (allowCollectedData || disallowCollectedData) until (submit && next (not gdprConsentIsVisible));
In this formula we allow any number of allowCollectedData or disallowCollectedData transitions, until the final submit resulting in a closed consent screen.
What’s next?
We’ve looked at some temporal operators in LTL, and how to use them to specify state machines. I’m hoping this post has given you some ideas and inspiration!
Another blog post worth checking out is TLA+ Action Properties by Hillel Wayne. It’s written specifically for TLA+, but most of the concepts are applicable to LTL and Quickstrom-style specifications.
I intend to write follow-ups, covering atomic propositions, queries, actions, and events. If you want to comment, there are threads on GitHub, Twitter, and on Lobsters. You may also want to sponsor my work.
2021-05-28: Added links to the Linear Temporal Logic Visualizer matching the relevant examples. Note that the syntax is different in the visualizer.
Footnotes
A future version of Quickstrom will use a different flavor of LTL tailored for testing, and that way support liveness properties.↩︎
We can define eventually P = true until P, and perhaps a bit harder to grasp, always P = not (true until not P). Or we could say always P = not (eventually not P).↩︎
Recently, I wanted to better understand how the Cardano blockchain works on a technical level, in particular how the ledger and smart contracts work. Cardano is a rapidly moving target, and I had some trouble finding an accessible, yet detailed introduction. So, I decided to write up an introduction myself, after browsing through the documentation for Cardano, the (separate?) documentation for Plutus, the research papers, and the formal specifications. For me, the most accessible summary was M. Chakravarty et. al.’s paper “Functional Blockchain Contracts”, though for some reason, this paper still has draft status.
Cardano is a decentralized blockchain platform built on a proof-of-stake consensus mechanism. The main cryptocurrency on the Cardano blockchain is called ADA, named after lady Ada Lovelace.
Cardano is developed by the Cardano Foundation. You can find more information about their goals and ecosystem on their website. In this post, I would like to focus on the technology and give a brief, yet accessible introduction to how the Cardano ledger works, i.e. how transactions and smart contracts work. For more insight into the consensus protocol that is used to add transactions to the blockchain in real time, I refer to a blogpost by E. de Vries.
The Cardano Ledger
Basic transactions
This section presents the information you need to make sense of basic transactions on the blockchain, which you can view with e.g. the Cardano explorer.
A blockchain is a decentralized ledger. This ledger records transactions between different parties. A simple example of such a transaction is one where Alice (A) sends some quantity of digital currency to Bob (B). In the Cardano blockchain, this digital currency is called ADA. The following figure illustrates a transfer of four ADA from Alice to Bob:
The illustration shows the state before the transaction above the horizontal line, and the state after the transaction below the line: Before, Alice owns 4 ADA, while afterwards, Bob owns 4 ADA.
Unlike in the traditional banking system, where the bank authorizes who can submit transactions, anybody can submit a transaction to the blockchain. However, we have to make sure that only Alice is authorized to add this transaction to the ledger; otherwise, the system becomes useless, as e.g. Bob could attempt to add this transactions for his own benefit without confirmation from Alice. In a blockchain, this authorization is done using public key cryptography: Alice has a private key, which she uses to digitally sign the contents of the transaction. Using Alice’s public key, anyone can verify that it was indeed Alice who signed the transaction, but only someone who knows the private key is able to produce this signature. Typically, Alice uses a wallet software to manage her private key and sign transactions. The following figure illustrates the signed transaction:
If we know Alice’s public key, we can check that the signature is valid and that it was indeed Alice who authorized the transaction. In fact, the blockchain does not keep track of which key belongs to whom; Alice’s public key is simply included in the transaction data. Nothing prevents Alice from using multiple public and private key pairs, corresponding to different “accounts” or “wallets”.
The flip side of not relying on a bank to authorize transactions, and possibly revert them in the case of abuse, is that the security of your money is tied to the security of your private key. Specifically, a) you need to keep your private key secret, because other people can spend your money if they know it, and b) if you lose / forget / hide too well your private key, nobody, including you, can spend your money anymore. It cannot be stressed often enough: Your private key = your money!
In the real Cardano blockchain, parties are not identified by their public keys; instead, money is transferred between addresses. A party can use as many addresses as they like. While the public key of Alice is included in the transaction, the public key of Bob is not included, only one of his addresses. We will explain the relation between keys and addresses in more detail below.
In addition, the Cardano blockchain allows transactions between multiple addresses at once. For instance, a transaction may transfer money from one address to, say, two different addresses. The following figure illustrates a transfer of four ADA from address A, three of which go to address B, and one of which goes to address C:
The Cardano blockchain is public. Anyone can view the transactions on the Cardano explorer. For example, the transaction with ID 825c63… takes 42.42… ADA from an address and transfers 5 ADA to one address and 37.25… to another. In addition, 0.16… ADA is taken as a fee for including the transaction in the blockchain.
It is worth noting that the Cardano ledger does not record total balances of money owned by different addresses. It turns out that this simplifies the ledger model; we will discuss this below. Here, we merely note that storing total balances is not necessary, because the blockchain is public information, and one can compute the address balance by following all transactions from the beginning. The Cardano explorer does that automatically; for example, at the time of this writing, the address addr1q8kzwv… was involved in two transactions and has a total balance of 0 ADA.
On the Cardano blockchain, transactions are grouped into blocks, and blocks are grouped into epochs. This is relevant for determining which of the submitted transactions are to be included in the ledger, via the proof-of-stake consensus protocol, but as far as the currency is concerned, the grouping is irrelevant.
UTXO
While the previous section explained how to read basic transactions on the Cardano blockchain explorer, this section will present more details on how transactions are constructed. For example, many transaction which represent a simple transfer of currency between two parties are built using three addresses, one input and two output addresses, and sometimes even many more addresses. The reason is that Cardano uses an unspent transaction output (UTXO) model, similar to Bitcoin, which we will explain here.
On the surface, transactions in the unspent transaction output (UTXO) model may look like transfers of funds (money) between addresses, as described previously. However, on a deeper level, the UTXO model is better understood if we take a radical departure from the idea that addresses act as containers for funds. Instead, in the UTXO model, the containers for funds are the transaction outputs. Each transaction consists of a unique transaction ID, a set of inputs, and a list of outputs. Conversely, each output is uniquely identified by its position in the list of outputs and the transaction ID of the associated transaction. The inputs of a new transaction that is added to the blockchain are always references to previous outputs; moreover, once an output has been used as an input to a new transaction, it cannot be used a second time. Put differently, the inputs of new transaction may only refer to unspent transactions, which have not been used as input to another transaction yet. The fact that outputs are associated with addresses and funds is not relevant for being able to connect the inputs and outputs of transactions. The following figure illustrates the blockchain as a set of transactions with connected inputs and outputs
Here, the grey boxes within each figure correspond to inputs (above the line) and outputs (below the line). Each output can be connected to exactly one input; this is visualized by a dangling black line (output) which connects to a dangling red line (input).
The Cardano explorer is not a direct visualization of the UTXO model, because it does not directly show which transaction IDs the inputs reference. However, one can often infer this information from the addresses and funds associated with the input. For example, the address addr1q8kzwv… is involved in two transactions. The last output of the first transaction 34d8… is the input to the second transaction 825c…, because both the address and the amount of ADA are identical.
Funds and addresses do become important when validating transactions. For example, a transaction must conserve funds: The sum of funds of the inputs must be equal to the sum of funds of the outputs (minus a small transaction fee). We will discuss this in the next section.
Transferring funds between parties within the UTXO model often involves multiple inputs and outputs. For example, consider an unspent transaction output which contains 5 ADA. Any transaction that wants to use this output must use the entire sum. If we want to transfer 4 ADA to another party, we cannot do this with a transaction that expects 4 ADA as an input — we would not be able to connect it to the output, because an input and an output must match identically. However, we can create a transaction which takes an input with 5 ADA and transfers 4 ADA to one output with an address belonging to the other party, and 1 ADA to a second output with an address that is under our control. Many transactions on the Cardano blockchain are of this kind, for example the transaction 825c63…. However, it may also happen that we want to transfer 4 ADA, but only three outputs with 1.5 Ada each are available that are owned by us. In this case, we create a transactions with these three inputs and with two outputs, one that sends 3 Ada to the other party, and one with the remaining 0.5 Ada and an address owned by us. For example, the transaction 1bab… uses this scheme, as all the inputs have the same address.
This UTXO approach to transferring funds may seem a little roundabout at first, but the advantage is that all validity checks are confined to transactions — we can connect transactions in any way that adheres to the UTXO principle (each output is spent fully and at most once), and be guaranteed that the ledger conserves money as long as each individual transaction is valid. This simplifies the ledger model, and is an example of “compositionality”.
Transaction Validation
Having explained how inputs and outputs are connected via the UTXO model, we next turn to the validation of transactions, which is arguably the most important task of a blockchain. We will look at addresses and public keys for authorizing transactions in this sections, and discuss smart contracts in the next section.
The validation of transactions is what enables blockchain ledgers to store monetary value. To be considered valid, a transaction must pass several checks; only then can it be included in the blockchain ledger.
First, the transaction must conform to the UTXO model, i.e. every input refers to a unique unspent output. This effectively prevents double spending of funds.
Second, money must be conserved. The funds associated to the inputs may be redistributed and be assigned differently to the outputs, but the sum of funds of the inputs must be equal to the sum of funds of the outputs, minus a small transaction fee.
Third, the transaction must be authorized by the parties that own the addresses associated to the inputs. As mentioned in a previous section, this is done using public key cryptography. However, an address is not a public key, instead, it is the hash of a public key. This is a neat trick to reduce the size of the addresses. That said, it is impossible to check a signature by using just the hash of the public key. Thus, the transaction explicitly lists all the public keys that hash to the input addresses, thereby revealing the public keys. The transaction must be signed by all corresponding private keys (which must be kept secret by the parties). However, the public keys of the output addresses are not revealed; this saves space, and adds an extra layer of cryptographic protection by not exposing the public keys to a possible (though likely hopeless) offline cryptanalysis.
With these validation checks, the ledger is suitable for storing monetary value.
However, by allowing more complicated validation logic, we can support smart contracts, which are automated contracts for transferring funds and data. To this end, the Cardano ledger associates inputs and outputs not just with funds and addresses, but with additional, custom data which encodes the state of the smart contract. This leads to the Extended UTXO (EUTXO) model, which we discuss in the next section.
At the time of this writing, the data supported by the Cardano ledger, and also its data format, are not set in stone: From time to time, the Cardano blockchain will undergo a hard fork where the ledger format changes. However, thanks to a special programming technique known as a hard fork combinator, these forks are almost seamless: existing transactions on the ledger will be unchanged, and are compatible with all future transactions that are added to the chain. The Cardano ledger has already undergone a number of hard forks; the data formats are documented in formal specifications. The Cardano blockchain started with the Byron format, which supported UTXO model transactions of Ada. Then came the Shelly format, which enabled stake pools and staking rewards. The Mary hard fork enabled multi-assets (Shelly-MA), i.e. the transfer of tokens or currencies besides Ada. The Alonzo hard fork will enable smart contracts.
Smart Contracts and EUTXO
Blockchains can automate not just simple transfers of money, but also more complex exchanges involving multiple parties and conditions. Smart contracts are pieces of software that describe such exchanges.
On the Cardano blockchain, smart contracts are identified by their addresses. Instead of sending funds to an address controlled by a human, you can send funds to an address representing a smart contract, which will then perform some automated transactions involving your funds. By looking at the source code for this contract beforehand, you can understand exactly what the contract is going to do. The validity checks on the blockchain ensure that the contract is executed exactly as specified.
A simple example of an exchange suitable for a smart contract would be donation matching: Whenever Bob receives a donation, Alice wants to match that donation with an equal amount of money. One possible implementation in terms of a smart contract would would work as follows: First, Alice sends the funds she wants to use for donation matching to the contract. Then, any willing donor, say Dorothee, sends money to the contract, which will then automatically forward this money plus an equal amount of matching funds to the address owned by Bob.
This example shows that smart contracts often involve multiple transactions: Alice makes a transaction that transfers funds to the contract, and Dorothee makes a transaction that donates to Bob. These transactions need to adhere to the contract rules, e.g. that Alice is the first to make the transaction, and Dorothee must come second. Any smart contract consists of two components: an on-chain component, which ensures that transactions adhere to the contract rules, and an off-chain component, which is a program that Alice or Dorothee use to submit rule-abiding transactions. The on-chain component never submits any transactions to the blockchain, it only checks existing transactions for validity. The on-chain component is run on the nodes that validate the blockchain ledger, while the off-chain component is run by the user, e.g. as part of his wallet. For Cardano, the Plutus platform allows us to write both components using the same programming language, Haskell.
The on-chain component of a smart contract operates like a state machine. When an output of a transaction is a smart contract address, this address has to be accompanied by additional, custom state data. This data represents a new state of the contract. Here, the same hashing trick as for addresses is used: Only a hash of the data, not the data itself is included in the transaction. But when the input of transaction references this output, via the UTXO model, the data is included in full. In addition, this input has to come with additional, custom data called a redeemer, which has the role of user input, and corresponds to a state transition label for the state machine. The UTXO-like ledger model which includes state data and redeemer data is known as the Extended UTXO (EUTXO) model.
For example, the donation matching example above can be implemented with the following state machine:
The machine has a single state Funded addrA addrB, which indicates that the contract is funded and ready to match donations for Bob. This state stores both Alice’s address addrA , and Bob’s address addrB. The possible transitions are Donate for making a matched donation to Bob, and Refund to return Alice’s funds when she wants to end the fundraiser. The state machine may seem somewhat unusual in that the contract state does not contain information about Alice’s total funds, and that the donation amount is not recorded in the Donate transition. However, there is no need for this, because transaction inputs and outputs always store an amount of funds.
The on-chain component can implement this state machine by only accepting the following transaction (schemas) as valid:
Here, the first transaction schema corresponds to a Donate transition. It takes an amount y from Dorothee and transfers twice that amount to Bob, using the funds held by the smart contract input. The second transaction schema corresponds to the Refund transition. It returns all the funds in the contract to Alice by spending the input contract. Here, we have also added an input from Alice (with zero funds) in order to make sure that not only does all the money go to back to Alice, but also that only Alice can initiate a refund: As her address appears in the input, this transaction is only valid with her signature. Similarly, as the smart contract address appears as an input, the validator code of the smart contract is run on this transaction. By refusing all transactions that do not fit into these two schema, only valid transitions of the smart contract state are possible. (The description here is a bit oversimplified in that we have ignored transaction fees and the fact unspent outputs with zero funds are not allowed. Transactions on the blockchain always involve a small fee.)
Finally, to start a fundraiser for Bob, Alice can submit the following transaction to the chain.
This transaction creates a smart contract output with funds, ready to be spent as an input.
Conclusion
We have taken a look at the ledger for the Cardano blockchain. We have looked at basic transactions, the unspent transaction output (UTXO) model popularized by bitcoin, validation with addresses and public keys, and finally introduced the basics of smart contracts and how their on-chain components can be understood as state machines. That said, we have also skipped over some features of the ledger, such as multiple assets and staking, because I believe that they do not affect the substance of the EUTXO model and can be understood separately if desired. I hope this helps!
If this were a YouTube channel, I would ask you to smash the like button and click subscribe. Oh well. I guess I can ask you to leave comments below, though.
regular (positive) Pell equation is x^2 - D*y^2 = +1.
negative Pell equation is x^2 - D*y^2 = -1. if you have a solution (x,y) to the negative Pell equation, run Newton's method computing sqrt(D) for one iteration, starting with the guess old=x/y. that is, new = (old + D/old)/2 (the Babylonian method is equivalent to Newton's method). the components of the resulting fraction newx/newy will be a solution to the regular Pell equation: newx^2 - D*newy^2 = +1.
solving the Pell equation with continued fractions is very straightforward. compute the continued fraction of sqrt(D), try all truncations in order of length. for each truncation, convert to a regular fraction x/y and check if x and y solve the regular Pell equation. there is no need to systematically modify the last term before the truncation point as you do when using continued fractions to find the sequence of best fractions that approximate a given number.
as an optimization, also check whether each (x,y) solves the negative Pell equation, then use the Newton's method trick above to convert the negative equation solution to a regular equation solution. the smallest negative solution (if one exists) will convert to the smallest regular solution.
the following sequence of D require increasing numbers of continued fraction terms to solve. the Newton's method optimization is included, so we are counting the number of continued fraction terms needed to reach the first regular or negative solution. these D represent challenges of increasing difficulty. the famous value 61 is not on the list. 61 requires only 11 continued fraction terms to find a negative equation solution (x=29718, y=3805). it is eclipsed by the smaller 46 which requires 12 terms for its smallest solution (x=24335, y=3588), which happens to solve the regular equation.
the sequence below is a subsequence of the sequence above. for these record-setting D values, the solution is to the negative equation, so you need to do one Newton's method iteration afterward, so the smallest regular solution unusually large.
some disorganized Haskell source code is here. it includes numerical testing of the conjectures stated above.
we used the quadratic-irrational package to compute the continued fraction representation of square roots. we used the continued-fraction package to convert continued fractions to Rational. below is code to convert a quadratic irrational into an infinite list of continued fraction terms, using the cycle function. Haskell handles infinite data structures elegantly.
having constructed an infinite list, use the inits function to construct all truncations, then for each truncation, test whether it solves the Pell equation. (but our code does not use inits because we were also testing whether the final term needs to be modified.)
2021-04-23: WHO WOULD WIN THIS FIGHT: Ragnarr Loðbrók or Miyamoto Musashi? Today's #haskell problem gives a bean-counter's approach, ... LITERALLY! Today's #haskell solutions shows that the winner of the Viking-Samurai deth-match is: ViewPatterns! ... obviously. �
2021-04-22: In today's #haskell exercise we try to find the opposite of Kung Fu Fighting, and we fail (because there is no opposite of awesome ... besides fail. GEDDIT?) So we try to find the dual of Kung Fu Fighting. Easy, no? "Your WuShu Praying Mantis-style is strong, but can it match my Tiger Claw-style?" Narrator: "We find out the thrilling conclusion in today's #haskell solution!"
2021-04-09: We get all officious, ... sry: 'OFFICIAL' ... with today's #haskell problem. Today's #haskell solution shows us that you haven't met real evil until your TPS-report cover letter is ... Lorem Ipsum.
2021-04-08, Thursday: In today's #haskell problem we optionally parse a string into some value of some type. Some how. Today's #haskell solution shows us the sum of the numbers in the string was 42 (... no, it wasn't).
2021-04-05, Monday: Today's #haskell exercise is to make a vector that is Foldable and Traversable. Today's #haskell solution gives us Vectors as Foldable and Traversable. YUS!
2021-04-01, Holy Thursday: In today's #haskell exercise we learn the WHOPR beat the Big Mac. We also learn how to make a safe call to an HTTPS endpoint, but that's not important as the WHOPR beating the Big Mac. "Would you like to play a game?" Today's #haskell solution says: "Yes, (safely, please)."
I've spent a considerable amount of coding time getting into the weeds of path parsing and generation in web applications. First with Yesod in Haskell, and more recently with a side project for routetypes in Rust. (Side note: I'll likely do some blogging and/or videos about that project in the future, stay tuned.) My recent work reminded me of a bunch of the pain points involved here. And as so often happens, I was complaining to my wife about these pain points, and decided to write a blog post about it.
First off, there are plenty of pain points I'm not going to address. For example, the insane world of percent encoding, and the different rules for what part of the URL you're in, is a constant source of misery and mistakes. Little things like required leading forward slashes, or whether query string parameters should differentiate between "no value provided" (e.g. ?foo) versus "empty value provided" (e.g. ?foo=). But I'll restrict myself to just one aspect: roundtripping path segments and rendered paths.
What's a path?
Let's take this blog post's URL: https://www.fpcomplete.com/blog/pains-path-parsing/. We can break it up into four logical pieces:
https is the scheme
:// is a required part of the URL syntax
www.fpcomplete.com is the authority. You may be wondering: isn't it just the domain name? Well, yes. But the authority may contain additional information too, like port number, username, password
/blog/pains-path-parsing/ is the path, including the leading and trailing forward slashes
This URL doesn't include them, but URLs may also include query strings, like ?source=rss, and fragments, like #what-s-a-path. But we just care about that path component.
The first way to think of a path is as a string. And by string, I mean a sequence of characters. And by sequence of characters, I really mean Unicode code points. (See how ridiculously pedantic I'm getting? Yeah, that's important.) But that's not true at all. To demonstrate, here's some Rust code that uses Hebrew letters in the path:
fn main() {
let uri = http::Uri::builder().path_and_query("/hello/מיכאל/").build();
println!("{:?}", uri);
}
And while that looks nice and simple, it fails spectacularly with the error message:
Err(http::Error(InvalidUri(InvalidUriChar)))
In reality, according to the RFC, paths are made up of a limited set of ASCII characters, represented as octets (raw bytes). And we somehow have to use percent encoding to represent other characters.
But before we can really talk about encoding and representing, we have to ask another orthogonal question.
What do paths represent?
While a path is technically a sequence of a reserved number of ASCII octets, that's not how our applications treat them. Instead, we want to be able to talk about the full range of Unicode code points. But it's more than just that. We want to be able to talk about groupings of sequences. We call these segments typically. The raw path /hello/world can be thought of as the segments ["hello", "world"]. I would call this parsing the path. And, in reverse, we can render those segments back into the original raw path.
With these kinds of parse/render pairs, it's always nice to have complete roundtripping abilities. In other words, parse(render(x)) == x and render(parse(x)) == x. Generally these rules fail for a variety of reasons, such as:
Multiple valid representations. For example, with the percent encoding we'll mention below, %2a and %2A mean the same thing.
Often unimportant whitespace details get lost during parsing. This applies to formats like JSON, where [true, false] and [ true, false ] have the same meaning.
Parsing can fail, so that it's invalid to call render on parse(x).
Because of this, we often end up reducing our goals to something like: for all x, parse(render(x)) is successful, and produces output identical to x.
In path parsing, we definitely have problem (1) above (multiple valid representations). But by using this simplified goal, we no longer worry about that problem. Paths in URLs also don't have unimportant whitespace details (every octet has meaning), so (2) isn't a problem to be concerned with. Even if it was, our parse(render(x)) step would end up "fixing" it.
The final point is interesting, and is going to be crucial to our complete solution. What exactly does it mean for path parsing to fail? I can think of two ideas in basic path parsing:
It includes an octet outside of the allowed range
It includes a percent encoding which is invalid, e.g. %@@
Let's assume for the rest of this post, however, that those have been dealt with at a previous step, and we know for a fact that those error conditions will not occur. Are there any other ways for parsing to fail? In a basic sense: no. In a more sophisticated parsing: absolutely.
Basic rendering
The basic rendering steps are fairly straightforward:
Perform percent encoding on each segment
Interpolate the segments with a slash separator
Prepend a slash to the entire string
To allow roundtripping, we need to ensure that each input to the render function generates a unique output. Unfortunately, with these basic rendering steps, we immediately run into an error:
In other words, both [] and [""] encode to the same raw path, /. This may seem like a trivial corner case not worth addressing. In fact, even more generally, empty path segments seem like a corner case. One possibility would be to say "segments must be non-zero length". Then there's no potential [""] input to worry about.
When this topic came up in Yesod, we decided to approach this differently. We actually did have some people who had use cases for empty path segments. We'll get back to this in normalized rendering.
Percent encoding
I mentioned originally the annoyances of percent encoding character sets. I'm still not going to go deeply into details of it. But we do need to discuss it at a surface level. In the steps above, let's ask two related questions:
Why did we percent encode before interpolating?
Do we percent encode forward slashes?
Let's try percent encoding after interpolating. And let's say we decide not to percent encode forward slashes. Then render(["foo/bar"]) would turn into /foo/bar, which is identical to render(["foo", "bar"]). That's not what we want. And if we decide we're going to percent encode after interpolating and that we will percent encode forward slashes, both inputs result in /foo%2Fbar as output. Neither of those is any good.
OK, going back to percent encoding before interpolating, let's say that we don't percent encode forward slashes. Then both ["foo/bar"] and ["foo", "bar"] will turn into /foo/bar, again bad. So by process of elimination, we're left with percent encoding before interpolating, and escaping the forward slashes in segments. With this configuration, we're left with render(["foo/bar"]) == "/foo%2Fbar" and render(["foo", "bar"]) == "/foo/bar". Not only is this unique output (our goal here), but it also intuitively feels right, at least to me.
Unicode codepoint handling
One detail we've glossed over here is Unicode, and the difference between codepoints and octets. It's time to rectify that. Percent encoding is a process that works on bytes, not characters. I can percent encode / into %2F, but only because I'm assuming an ASCII representation of that character. By contrast, let's go back to my favorite non-Latin alphabet example, Hebrew. How do you represent the Hebrew letter Alef א with percent encoding? The answer is that you can't, at least not directly. Instead, we need to represent that Unicode codepoint (U+05D0) as bytes. And the most universally accepted way to do that is to use UTF-8. So our process is something like this:
let segment: &[char] = "א";
let segment_bytes: &[u8] = encode_utf8(segment); // b"\xD7\x90"
let encoded: &[u8] = percent_encode(segment_bytes); // b"%D7%90"
OK, awesome, we now have a way to take a sequence of non-empty Unicode strings and generate a unique path representation of that. What's next?
Basic parsing
How do we go backwards? Easy: we reverse each of the steps above. Let's see the render steps again:
Percent encode each segment, consisting of:
UTF-8 encode the codepoints into bytes
Percent encode all relevant octets, including the forward slash
Interpolate all of the segments together, separated by a forward slash
Technically, the "forward slash" here is the forward slash octet\x2F. But because everyone basically assumes ASCII/UTF-8 encoding, we can typically be a little loose in our terminology.
Prepend a forward slash (octet).
Basic parsing is exactly the same steps in reverse:
Strip off the forward slash.
Arguably, if a forward slash is missing, you could consider this a parse error. Most parsers simply ignore it instead.
Split the raw path on each occurrence of a forward slash. We'll discuss some subtleties about this next.
Percent decode each segment, consisting of:
Look for any % signs, and grab the next two hexadecimal digits. In theory, you could treat an incorrect or missing digit as a parse error. In practice, many people end up using some kind of fallback.
Take the percent decoded octets and UTF-8 decode them. Again, in theory, you could treat invalid UTF-8 data as a parse error, but many people simply use the Unicode replacement character.
If implemented correctly, this should result in the goal we mentioned above: encoding and decoding a specific input will always give back the original value (ignoring the empty segment case, which we still haven't addressed). The one really tricky thing is making sure that our split and interpolate operations mirror each other correctly. There are actually many different ways of splitting lists and strings. Fortunately for my Rust interpolation, the standard split method on str happens to implement exactly the behavior we want. You can check out the method's documentation for details (helpful even for non-Rustaceans!). Pay particular attention to the comments about contiguous separators, and think about how ["foo", "", "", "bar"] would end up being interpolated and then parsed.
OK, we're all done, right? Wrong!
Normalization
I bet you thought I forgot about the empty segments. (Actually, given how many times I called them out, I bet you didn't think that.) Before, we saw exactly one problem with empty segments: the weird case of [""]. I want to first establish that empty segments are a much bigger problem than that.
I gave a link above to a GitHub repository: https://github.com/snoyberg/routetype-rs. Let's change that URL ever so slightly, and add an extra forward slash in between snoyberg and routetype-rs: https://github.com/snoyberg//routetype-rs. Amazingly, you get the same page for both URLs. Isn't that weird?
No, not really. Extra forward slashes are often times ignored by web servers. "I know what you meant, and you didn't mean an empty path segment." This isn't just a "feature" of webservers. The same concept applies on my Linux command line:
I've got two problems with the behavior GitHub is demonstrating above:
What if I'm writing some web application and I really, truly want to be able to embed a meaningful empty segment in the path?
Doesn't it feel wrong, and maybe even hurt SEO, to have two different URLs that resolve to the same content?
In Yesod, we addressed the second issue with a class method called cleanPath, that analyzes the segments of an incoming path and sees if there's a more canonical representation of them. For the case above, https://github.com/snoyberg//routetype-rs would produce the segments ["snoyberg", "", "routetype-rs"], and cleanPath would decide that a more canonical representation would be ["snoyberg", "routetype-rs"]. Then, Yesod would take the canonical representation and generate a redirect. In other words, if GitHub was written in Yesod, my request to https://github.com/snoyberg//routetype-rs would result in a redirect to https://github.com/snoyberg/routetype-rs.
Way back in 2012, this led to a problem, however. Someone actually had empty path segments, and Yesod was automatically redirecting away from the generated URLs. We came up with a solution back then that I'm still very fond of: dash prefixing. See the linked issue for the details, but the way it works is:
When encoding, if a segment consists entirely of dashes, add one more dash to it.
By our definition of "consists entirely of dashes," the empty string counts too. So dashPrefix "" == "-", and dashPrefix "---" == "----".
When decoding:
Perform the split operation above.
Next, perform the clean path check, and generate a redirect if there are any empty path segments.
Once we know that there are no empty path segments, then undo dash prefixing. If a segment consists of only dashes, remove one of the dashes.
If you work this through enough, you can see that with this addition, every possible sequence of segments—even empty segments—results in a unique raw path after rendering. And every incoming raw path can either be parsed to a necessary redirect (if there are empty segments) or to a sequence of segments. And finally, each sequence of segments will successfully roundtrip back to the original sequence when parsing and rendering.
I call this normalized parsing and rendering, since it is normalizing each incoming path to a single, canonical representation, at least as far as empty path segments are concerned. I suppose if someone wanted to be truly pedantic, they could also try to address variations in percent encoding behavior or invalid UTF-8 sequences. But I'd consider the former a non-semantic difference, and the latter garbage-in-garbage-out.
Trailing slashes
There's one final point to bring up. What exactly causes an empty path segment to occur when parsing? One example is contiguous slashes, like our snoyberg//routetype-rs example above. But there's a far more interesting and prevalent case: the trailing slash. Many web servers use trailing slashes, likely originating from the common pattern of having index.html files and accessing a page based on the containing directory name. In fact, this blog post is hosted on a statically generating site which uses that technique, which is why the URL has a trailing slash. And if you perform basic parsing on our path here, you'd get:
Whether to include trailing slashes in URLs has been an old argument on the internet. Personally, because I consider the parsing-into-segments concept to be central to path parsing, I prefer excluding the trailing slash. And in fact, Yesod's default (and, at least for now, routetype-rs's default) is to treat such a URL as non-canonical and redirect away from it. I felt even more strongly about that when I realized lots of frameworks have special handling for "final segments with filename extensions." For example, /blog/bananas/ is good with a trailing slash, but /images/bananas.png should not have a trailing slash.
However, since so many people like having trailing slashes, Yesod is configurable on this point, which is why cleanPath is a typeclass method that can be overridden. To each their own I suppose.
Conclusion
I hope this blog post gave a little more insight into the wild world of the web and how something as seemingly innocuous as paths actually hides some depth. If you're interested in learning more about the routetype-rs project, please let me know, and I'll try to prioritize some follow ups on it.
You may be interested in more Rust or Haskell from FP Complete. Also, check out our blog for a wide range of technical content.
At Facebook, we have an internal convention for tooling called "rage". When something goes wrong and you want to report a bug, the tool developer will typically ask you to give them a rage. For a command line tool, this can be done by running a rage subcommand, which will ask about which previous CLI invocation you'd like to report, and then giving you a bundle of logs to send to the developer.
A rage has an important property, compared to a conventional log level flag like -v: rage recording is always on. In other words, it is like traditional server application logs, but applied to client software. Logging is always turned on, and the rage subcommand makes it easy for a user to send only the relevant portion of logs (e.g., the logs associated with the command line invocation that is on).
For some reason, rage functionality is not that common in open source tools. I can imagine any number of reasons why this might be the case:
Adding proper logging is like flossing--annoying to do at the time even when it can save you a lot of pain later.
Even if you have logging, you still need to add infrastructure to save the logs somewhere and let users retrieve them afterwards.
It's something of an art to write logs that are useful enough so that developer can diagnose the problem simply by "reading the tea leaves", but not so detailed that they slow down normal execution of the program. And don't forget, you better not expose private information!
Most programs are simple, and you can just fall back on the old standby of asking the user to submit reproduction instructions in their bug report.
Still, in the same way most sysadmins view logging as an invaluable tool for debugging server issues, I think rage reporting is an invaluable tool for debugging client issues. In ghstack, it didn't take very many lines of code to implement rage reporting: ghstack.logs (for writing the logs to the rage directory) and ghstack.rage (for reading it out). But it has greatly reduced my support load for the project; given a rage, I can typically figure out the root cause of a bug without setting up a reproducer first.
Many major programming languages these days support some lightweight concurrency primitives. The most recent popular ones are Goroutines in Go, Coroutines in Kotlin and Async in Rust. Let’s explore some of these concepts in detail by implementing a programming language with support for coroutines and Go-style channels.
Lightweight concurrency has been a popular topic among programmers and programming language designers alike in recent times. Many languages created in the last decade have support for them either natively or using libraries. Some example are:
These examples differ in their implementation details but all of them enable programmers to run millions of tasks concurrently. This capability of being able to do multiple tasks at the same time is called Multitasking.
Coroutines@2 are computations that support cooperative multitasking1. Unlike ordinary Subroutines that execute from start to end and do not hold any state between invocations, coroutines can exit in the middle by calling other coroutines and may later resume at the same point. They also hold state between invocations. They do so by yielding the control of the current running thread so that some other coroutine can be run on the same thread2.
Coroutine implementations often come with support for Channels for inter-coroutine communication. One coroutine can send a message over a channel and another coroutine can receive the message from the same channel. Coroutines and channels together are an implementation of Communicating Sequential Processes (CSP)@5 a formal language for describing patterns of interaction in concurrent systems.
In this series of posts, we implement Co, a small interpreted dynamically-typed imperative programming language with support for coroutines and channels. Haskell is our choice of language to implement the Co interpreter.
Introducing Co
Co has these basic features that are found in many programming languages:
Null, boolean, string and integer literals and values.
Addition and subtraction arithmetic operations.
String concatenation operation.
Equality and inequality checks on booleans, strings and numbers.
Less-than and greater-than comparison operations on numbers.
Function declarations and calls.
First class functions.
Variable declarations, usage and assignments.
if and while statements.
It also has these special features:
yield statement to yield the current thread of computation (ToC).
spawn statement to start a new ToC.
First class channels with operators to send and receive values over them.
Let’s see some example code in Co for illustration:
// Fibonacci numbers // using a while loopvar a =0;var b =1;var j =0;while (j <6) {print(a);var temp = a; a = b; b = temp + b; j = j +1;}// Fibonacci numbers // using recursive function callfunctionfib(n, f) {if (n <2) {return n; }returnf(n -2, f) +f(n -1, f);}var i =0;while (i <6) {print(fib(i, fib)); i = i +1;}
As you may have noticed, Co’s syntax is heavily inspired by JavaScript. The code example above computes and prints the first six Fibonacci numbers in two different ways and demonstrates a number of features of Co, including variable declarations and assignments, while loops, if conditions, and function declarations and calls3.
We can save the code in a file and run it with the Co interpreter4 to print this (correct) output:
0
1
1
2
3
5
0
1
1
2
3
5
The next example show the usage of coroutines in Co:
functionprintNums(start, end) {var i = start;while (i < end +1) {print(i);yield; i = i +1; }}spawn printNums(1,4);printNums(11,15);
Running this code with the interpreter prints this output:
1
11
2
12
3
13
4
14
15
The printNum function prints numbers between the start and end arguments, but yields the ToC after each print. Notice how the prints are interleaved. This is because the two calls to the function printNums run concurrently in two separate coroutines.
The next example show the usage of channels in Co:
This is the popular Ping-pongbenchmark for communication between ToCs. Here we use channels and coroutines for the same. Running this code prints this output:
Interpretation: a tree-walking interpreter walks the AST, executes the instructions and produces the output.
Stages of the Co interpreter
<noscript>Stages of the Co interpreter</noscript>
In this post, we implement the parser for Co. In the second part, we create a first cut of the interpreter that supports the basic features of Co. In the third part, we extend the interpreter to add support for coroutines and channels.
The complete code for the parser is here. You can load it in GHCi using stack (by running stack co-interpreter.hs), and follow along while reading this article.
Let’s start with listing the extensions and imports needed5.
Since Co is an imperative programming language, it is naturally Statement oriented. A statement describes how an action is to be executed. A Co program is a list of top-level statements.
Statements have internal components called Expressions. Expressions evaluate to values at program run time. Let’s look at them first.
Expressions
We represent Co expressions as a Haskell Algebraic data type (ADT) with one constructor per expression type:
The boolean literals, true and false. Evaluate to their counterpart boolean values.
LStrString
A string literal like "towel". Evaluates to a string.
LNumInteger
An integer literal like 1 or -5. Evaluates to an integer.
VariableIdentifier
A variable named by an identifier like a1 or sender. Evaluates to the variable’s value at the point in the execution of code. An Identifier is a string of alphanumeric characters, starting with an alpha character.
BinaryOpExprExpr
A binary operation on two expressions. Example: 1+41 or 2=="2". Supported binary operations are defined by the BinOp enum: addition/concatenation, subtraction, equality and inequality checks, and less-than and greater-than comparisons.
CallIdentifier [Expr]
Calls a function named by an identifier, with the given argument expressions. Example: calcDistance(start, end).
ReceiveExpr
Receives a value from a channel. Examples:
// receive a value from the channel and prints itprint(<- someChannel);// receive a value from the channel and assigns itvar x =<- someChannel;
Next, we see how statements are represented in the AST.
Statements
We represent the Co statements as a Haskell ADT with one constructor per statement type:
A statement with just an expression. The expression’s value is thrown away after executing the statement. Example: 1+2;
VarStmtIdentifierExpr
Defines a new variable named by an identifier and sets it to an expression’s value. In Co, variables must be initialized when being defined. Example: var a =5;
AssignStmtIdentifierExpr
Sets an already defined variable named by an identifier to an expression’s value. Example: a =55+"hello";
IfStmtExpr [Stmt]
Executes a list of statements if the condition expression evaluates to a truthy value. In Co, null and false values are non-truthy, and every other value is truthy. Also note that, there are no else branches for if statements in Co. Example:
if (a ==1) {print("hello");}
WhileStmtExpr [Stmt]
Executes a list of statements repeatedly until the condition expression evaluates to a truthy value. Example:
var n =0;while (n <5) {print(n); n = n +1;}
FunctionStmtIdentifier [Identifier] [Stmt]
Defines a function with a name, a list of parameter names, and a list of body statements. Example:
Returns from a function, optionally returning an expression’s value. Example:
functionsquare(x) {return x * x;}
YieldStmt
Suspends the currently executing ToC so that some other ToC may run. The current ToC resumes later from statement next to the yield statement. Example:
functionprintNums(start, end) {var i = start;while (i < end +1) {print(i);yield; i = i +1; }}
SpawnStmtExpr
Starts a new ToC in which the given expression is evaluated, which runs concurrently with all other running ToCs. Example:
spawn printNums(1,4);printNums(11,15);
SendStmtExprIdentifier
Sends an expressions’s value to a channel named by an identifier. Example: 1+square(3) -> someChannel;
The Co AST is minimal but it serves our purpose. Next, let’s figure out how to actually parse source code to AST.
Parsing
Parsing is the process of taking textual input data and converting it to a data structure—often a hierarchal structure like AST—while checking the input for correct syntax. There are many ways of writing parsers6, Parser Combinators@10 being one of them. Parser combinators libraries are popular in Haskell because of their ease of use and succinctness. We are going to use one such library, Megaparsec, to create a parser for Co.
Let’s start with writing some basic parsers for the Co syntax using the Megaparsec parsers.
In the above code, Parser is a concise type-alias for our parser type7.
The sc parser is the space consumer parser. It dictates what’s ignored while parsing the sources code. For Co, we consider the Unicode space character and the control characters—tab, newline, carriage return, form feed, and vertical tab—as whitespaces. We use the spaceChar parser to configure that. sc also lets us configure how to ignore line comments and block comments while parsing.
The lexeme combinator is for parsing Lexemes while ignoring whitespaces and comments. It is implemented using the lexeme combinator from Megaparsec.
The symbol combinator is for parsing symbols, that is, verbatim strings like if and ;. It is implemented using the symbol combinator.
The parens and braces combinators are for parsing code surrounded by parentheses ( and ) and braces { and } respectively.
The semi parser matches a semicolon ;. The identifier parser is for parsing identifiers in Co. The stringLiteral parser matches a string literal like "polyphonic ringtone". integer is the parser for Co integers.
Let’s also write some functions to run the parsers and pretty-print the output in GHCi:
runParser ::Parser a ->String->EitherString arunParser parser code =docase parse parser "" code ofLeft err ->Left$ errorBundlePretty errRight prog ->Right progpPrint :: (MonadIO m, Show a) => a -> m ()pPrint = pPrintOpt CheckColorTty$ defaultOutputOptionsNoColor { outputOptionsIndentAmount =2, outputOptionsCompact =True, outputOptionsCompactParens =True }
That completes our basic setup for parsing. Let’s try them out in GHCi now:
They work as expected. Next, off to parsing Co expressions.
Parsing Expressions
Parsing Co expressions to AST requires us to handle the Associativity and Precedence of the operators. Fortunately, Megaparsec makes it easy with the makeExprParser combinator. makeExprParser takes a parser to parse the terms and a table of operators, and creates the expression parser for us.
Terms are parts of expressions which cannot be broken down further into sub-expressions. Examples in Co are literals, variables, groupings and function calls.
The table of operators is a list of OperatorParserExpr lists ordered in descending precedence. All operators in one list have the same precedence but may have different associativity.
This is a lot to take in but looking at the code makes it clear. First, the operator table:
operators :: [[OperatorParserExpr]]operators = [ [Prefix$Receive<$ symbol "<-"], [ binary Plus$ symbol "+", binary Minus$ try (symbol "-"<* notFollowedBy (char '>')) ], [ binary LessThan$ symbol "<", binary GreaterThan$ symbol ">" ], [ binary Equals$ symbol "==", binary NotEquals$ symbol "!=" ] ]where binary op symP =InfixL$Binary op <$ symP
The prefix operator <-, for receiving values from channels, is of highest precedence and hence the first in the table. Next are the binary + and - operators which are of same precedence. After that come the comparison operators < and >. Finally, we have the lowest precedence operators, the equality and inequality checks = and !=.
Each operator also contains the parser to parse the operator symbol in the source code. All of them are self-explanatory except the parser for the - operator. The - parser is slightly complicated because we want it to not parse ->, the channel send operator, the first character of which is same as the symbol for the - operator.
Moving on to the term parser next:
term ::ParserExprterm =LNull<$ symbol "null"<|>LBoolTrue<$ symbol "true"<|>LBoolFalse<$ symbol "false"<|>LStr<$> stringLiteral<|>LNum<$> integer<|> try (Call<$> identifier <*> parens (sepBy expr (char ','*> sc)))<|>Variable<$> identifier<|> parens expr
The term parser is a combination of smaller parsers—one for each type of terms in Co—combined together using the Alternative instance of the parsers. It matches each parser one by one from the top until the match succeeds8. First, it tries to match for literals null, true, and false, failing which it matches for string and integer literals. Then it matches for function calls, variables, and expressions grouped in parentheses—in that order.
That’s it! We finally write the expression parser:
expr ::ParserExprexpr = makeExprParser term operators
Most of the statements start with keywords like if, var, function, etc, so our parsers look for the keywords first using the symbol parser. If none of the keyword-starting statements match then we try9 matching for assignments, channel send, and expression statements, in that order. The stmt parser uses the expr parser to parse expressions within statements. It also uses the combinators many, optional, try and sepBy from Megaparsec.
Finally, we put everything together to create the parser for Co:
program ::ParserProgramprogram = sc *> many stmt <* eof
The Co parser matches multiple top-level statements, starting with optional whitespace and ending with end-of-file.
It’s demo time. In GHCi, we parse a file and pretty-print the AST:
I’ve gone ahead and created a side-by-side view of the source code and corresponding AST for all three input files, aligned neatly for your pleasure of comparison. If you so wish, feast your eyes on them and check for yourself that everything works correctly.
Source code and AST for fib.co
// Fibonacci numbers // using a while loopvar a =0;var b =1;var j =0;while (j <6) {print(a);var temp = a; a = b; b = temp + b; j = j +1;}// Fibonacci numbers // using recursive function callfunctionfib(n, f) {if (n <2) {return n; }returnf(n -2, f) +f(n -1, f);}var i =0;while (i <6) {print(fib(i, fib)); i = i +1;}
That’s all for now. In exactly 102 lines of code, we have implemented the complete parser for Co. In the next part, we’ll implement a tree-walking interpreter for the Co AST that supports the basic features.
Knuth, Donald E. “Coroutines.” In The Art of Computer Programming: Volume 1: Fundamental Algorithms, 3rd ed., 193–200. Addison Wesley, 1997.
Watson, Des. “Approaches to Syntax Analysis.” In A Practical Approach to Compiler Construction, 75–93. Springer International Publishing, 2017. https://doi.org/10.1007/978-3-319-52789-5_4.
Coroutines are often conflated with Fibers. Some implementations of coroutines call themselves fibers instead. Quoting Wikipedia:
Fibers describe essentially the same concept as coroutines. The distinction, if there is any, is that coroutines are a language-level construct, a form of control flow, while fibers are a systems-level construct, viewed as threads that happen to not run in parallel. It is contentious which of the two concepts has priority: fibers may be viewed as an implementation of coroutines, or as a substrate on which to implement coroutines.
Green threads are another popular solution for lightweight concurrency. Unlike coroutines, green threads are preemtable. Unlike Threads, they are scheduled by the language runtime instead of the operating system.↩︎
In the interest of simplicity of implementation, Co does not support recursive functions. However, it does support first-class functions. So for the fib function to call itself, we pass it itself as an argument.↩︎
I explore parsing in more depth by writing a JSON parser from scratch in Haskell in one of my previous posts.↩︎
I followed this (outdated) Megaparsec tutorial for writing a parser for an imperative language. This tutorial is also helpful in learning the intricacies of Megaparsec.↩︎
Notice how some of the statement type parsers have try in front of them while most don’t. The try parser combinator backtracks to the start of its input when its argument parser fails, but so does the <|> operator. What’s the difference then?
The difference is that for parsers, the <|> operator can work only with a known look ahead, while the try combinator can work with arbitrary look ahead. Since the symbol parser knows the exact symbol to look for, it requires a known look ahead and, hence, works with <|>. However, the identifier and expr parsers don’t know their look aheads beforehand and, hence, need to be wrapped with try.
However, as commented by Gilmi, this kind of use of try may cause confusing error messages. We use it here only for the sake of simplicity.↩︎
Basile Henry @basile_henry: Slightly less polymorphic: sequence' = traverse @[] (sequence @((,) _) @IO)
Basile Henry @basile_henry: I think it's just traverse sequence ;)
Jérôme Avoustin @JeromeAvoustin: there surely is a shorter version, but I could come up with... fmap join . sequence . (fmap . fmap) sequence . fmap sequence
2021-04-16, Friday:
You have a monad, or applicative, and you wish to execute the action of the latter but return the result of the former. The simplest representation for me is:
pass :: IO a -> b -> IO b
so:
return 5 >>= pass (putStrLn "Hi, there!")
would return IO 5
GO!
D Oisín Kidney @oisdk flip (<$)
ⓘ_jack @Iceland_jack ($>)
2021-04-12, Monday:
A function that takes the result of another function then uses that result and the original pair of arguments to compute the result:
f :: a -> a -> b g :: b -> a -> a -> c
so:
(\x y -> g (f x y) x y)
curry away the x and y arguments.
2021-04-07, Wednesday: you have (Maybe a, Maybe b) you want Maybe (a, b)
If either (Maybe a) or (Maybe b) is Nothing then the answer is Nothing.
If both (Maybe a) and (Maybe b) are (Just ...) then the answer is Just (a, b)
In this article, we’ll develop a Haskell library for continued fractions. Continued fractions are a different representation for real numbers, besides the fractions and decimals we all learned about in grade school. In the process, we’ll build correct and performant software using ideas that are central to the Haskell programming language community: equational reasoning, property testing, and term rewriting.
I posted an early version of this article to Reddit, and I’m grateful to those who responded with insight and questions that helped me complete this journey, especially iggybibi and lpsmith.
Introduction to continued fractions
Let’s start with a challenge. Assume I know how to write down an integer. Now, how can I write down a real number? In school, we’ve all learned a few answers to this question. The big two are fractions and decimals.
Fractions have simple forms for rational numbers. Just write two integers, for the numerator and denominator. That’s great! But alas, they are no good at all for irrational numbers like π, or the square root of 2. I could, of course, write a fraction that’s close to the irrational number, such as 22/7 as an approximation of π. But this is only good up to a certain limited precision. To get a closer approximation (say, 355/113), I have to throw it away and start over. Sure, I could write an infinite sequence of fractions which converge to the rational number, but every finite prefix of such a sequence is redundant and wasted.
Irrationals are just the limiting case of rational numbers, and very precise rational numbers suffer from this, too. It’s not easy to glance at a fraction like 103993/33102 and get a sense of its value. (Answer: this is another approximation of π.) Or, can you quickly tell whether 5/29 is larger or smaller than 23/128?
Instead, we can turn to decimals. If you’re like me, you don’t think about the mechanics of decimal numbers very often, and conventional notation like 3.14159… really hides what’s going on. I’d like to offer a different perspective by writing it like this:
Here, d₀ is the integer part, and d₁, d₂, d₃, d₄, etc. are called the decimal digits. Notice the symmetry here: after the integer part, a decimal has inside of it another decimal representing its tenths.
This recursive structure leads to the main property that we care about with decimals: we can truncate them. If I have a hundred-digit decimal number, and don’t need that much accuracy, I can simply chop off what I don’t need to get a simpler approximation. This is the same property that lets me write a decimal representation of an irrational number. As long as we accept that infinite sequences exist — and as Haskell programmers, infinite data structures don’t scare us…. much — there is a unique infinite sequence of digits that represents any irrational number. The sequence must be infinite is to be expected, since there are uncountably many irrationals, and any finite representation has only countable values. (Unfortunately, this uniqueness isn’t quite true of rational numbers, since 0.4999… and 0.5 are the same number, for example!)
As nice as that is, though, there are also some significant disadvantages to decimals as a representation:
Although a few rational numbers have simple decimal representations, almost all rational numbers actually need infinite repeating decimals. Specifically, a rational number whose denominator has any prime factor other than 2 or 5 repeats indefinitely.
Why 10? This is a question for historians, anthropologists, perhaps biologists, but has no answer within mathematics. It is a choice that matters, too! In a different base, a different subset of the rational numbers will have finite decimals. It’s the choice of base 10 that makes it harder to compute with 1/3. Having to make an arbitrary choice about something that matters so much is unfortunate.
We can make up some ground on these disadvantages with a third representation: continued fractions. A continued fraction is an expression of this form:
Just like with decimals, this gives a representation of a real number as a (possibly finite, or possibly infinite) sequence [n₀, n₁, n₂, n₃, n₄, …]. Now we call the integer parts terms, instead of digits. The first term, n₀, is the integer part. The difference is that when we have a remainder to write, we take its reciprocal and continue the sequence with that.
Note:
Continued fractions represent all rational numbers as finite sequences of terms, while still accounting for all irrationals using infinite sequences.
Continued fractions do not depend on an arbitrary choice of base. The reciprocal is the same regardless of how we choose to write numbers.
Continued fractions, like decimals, can be truncated to produce a rational approximation. Unlike decimals, though, the approximations are optimal in the sense that they are as close as possible to the original number without needing a larger denominator.
With this motivation in mind, let’s see what we can do about using continued fractions to represent real numbers.
Part 1: Types and Values
Time to write some code. As a declarative language that promotes equational reasoning, Haskell is a joy to use for problems like this one.
To start, we want a new type for continued fractions. A continued fraction is a (finite or infinite) sequence of terms, so we could simply represent it with a Haskell list. However, I find it more readable to define a new type so that we can choose constructors with suggestive names. (It helps with this decision that standard list combinators like map or ++ don’t have obvious interpretations for continued fractions.) I’ve defined the type like this.
data CFrac where (:+/) :: Integer -> CFrac -> CFrac Inf :: CFrac
deriving instance Eq CFrac deriving instance Show CFrac
infixr 5 :+/
Here, x :+/ y means x + 1/y, where x is an integer, and y is another continued fraction (the reciprocal of the remainder). This is the basic building block of continued fractions, as we saw above.
The other constructor may be surprising, though. I’ve defined a special continued fraction representing infinity! Why? We’ll use it for terminating fractions. When there is no remainder, we have x = x + 0 = x + 1/∞, so Inf is the reciprocal of the remainder for a continued fraction that represents an integer. That we can represent ∞ itself as a top-level continued fraction wasn’t the goal, but it doesn’t seem worth dodging. Noticing that an empty list of terms acts like infinity is actually a great intuition for things to come.
There is a simple one-to-one correspondence between continued fractions and their lists of terms, as follows:
The :+/ operator corresponds to the list cons operator, :.
The Inf value corresponds to the empty list, [].
We can define conversions to witness this correspondence between CFrac and term lists.
terms :: CFrac -> [Integer] terms Inf = [] terms (n :+/ x) = n : terms x
The real fun of continued fractions, though, is that we can represent irrational numbers precisely, as well!
Rational numbers are precisely the solutions to linear equations. The simplest irrational numbers to write as continued fractions are the quadratic irrationals: numbers that are not rational, but are solutions to quadratic equations, and these are precisely the continued fractions with infinitely repeating terms. We can write a function to build these:
cycleTerms :: [Integer] -> CFrac cycleTerms ns = fix (go ns) where go [] x = x go (t : ts) x = t :+/ go ts x
And then we can write a quick catalog of easy continued fractions, including some small square roots and the golden ratio.
sqrt2 :: CFrac sqrt2 = 1 :+/ cycleTerms [2]
sqrt3 :: CFrac sqrt3 = 1 :+/ cycleTerms [1, 2]
sqrt5 :: CFrac sqrt5 = 2 :+/ cycleTerms [4]
phi :: CFrac phi = cycleTerms [1]
It’s really worth pausing to think how remarkable it is that these fundamental quantities, whose decimal representations have no obvious patterns at all, are so simple as continued fractions!
And it doesn’t end there. Euler’s constant e, even though it’s a transcendental number, also has a simple pattern as a continued fraction.
This really strengthens the claim I made earlier, that being based on the reciprocal instead of multiplying by an arbitrary base makes continued fractions somehow less arbitrary than decimals.
I wish I could tell you that π has a similar nice pattern, but alas, it does not. π has a continued fraction just as random as its representation in other notations, and just as hard to compute to arbitrary precision.
It will be interesting later, though, to look at its first few terms. For that, it’s good enough to use the Double approximation to π from Haskell’s base library. It’s not really π; in fact, it’s a rational number! But it‘s close enough for our purposes. Computing an exact value of π needs some more machinery, which we will develop in the next section.
We’ll now return to a more theoretical concern. We want each real number to have a unique continued fraction representation. In fact, I snuck something by you earlier when I derived an Eq instance for CFrac, because that instance is only valid when the type has unique representations. However, this isn’t quite true in general. Here are some examples of continued fractions that are written differently, but have the same value:
Case (1) deals with negative numbers. These will actually cause quite a few problems — not just now, but in convergence properties of algorithms later, too. I have been quietly assuming up to this point that all the numbers we’re dealing with are non-negative. Let’s make that assumption explicit, and require that all continued fraction terms are positive. That takes care of case (1).
This might seem like a terrible restriction. In the end, though, we can recover negative numbers in the same way humans have done so for centuries: by keeping track of a sign, separate from the absolute value. A signed continued fraction type, then, would wrap CFrac and include a Bool field for whether it’s negative. This is entirely straight-forward, and I leave it as an exercise for the interested reader.
Case (2) involves a term with a value of zero. It’s normal for the integer part of a continued fraction to be zero. After that, though, the remaining terms should never be zero, because they are the reciprocals of numbers less than one. So we will impose a second rule that only the first term of a continued fraction may be zero.
Case (3) involves a rational continued fraction with 1 as its final term. It turns out that even after solving cases (1) and (2), every rational number has two distinct continued fractions: one that has a 1 that is not the integer part as its last term, and one that doesn’t. This is analogous to the fact that terminating decimals have two decimal representations, one ending in an infinite sequence of 0s, and the other in an infinite sequence of 9s. In this case, the trailing 1 is longer than it needs to be, so we’ll disallow it, making a third rule that a term sequence can never end in 1, unless that 1 is the integer part.
Subject to these three rules, we get the canonical forms we wanted. It’s unfortunate that non-canonical forms are representable in the CFrac type (i.e., we have failed to make illegal data unrepresentable), but we can do the next best thing: check that our computations all produce canonical values. To do so, let’s write some code to check that a CFrac obeys these rules.
One very powerful idea that originated in the Haskell community is property testing. We state a property that we want to verify about our code, and allow a testing framework like to QuickCheck to make up examples and test them. Now is a great time to try this out. Here I’ve defined a few properties around canonical forms. The first is a trivial property that just asserts that we make the right decision about the specific examples above. The second, less trivial, guarantees that our cfFromRational function, the only real semantic function we’ve written, produces canonical forms.
(We could try to check that the irrational values are canonical, as well, but alas, we run into non-termination since they are infinite. This will be a theme: we’ll have to be satisfied with checking rational test cases, and relying on the fact that any bugs with irrational values will manifest in some nearby rational value.)
We can now run our code for this section. Try it yourself at https://code.world/haskell#PVxpksYe_YAcGw3CNdvEFzw. In addition to running the tests, we’ll print out our example continued fractions so we can see them.
In the previous section, we converted from rational numbers into continued fractions. We now turn to the inverse conversion: from a continued fraction to a rational number.
Not all continued fractions correspond to rational numbers. However, one of the key benefits of the continued fraction representation is that its truncated term sequences represent particularly efficient rational approximations. These rational approximations are called the convergents. It’s not hard to write a simple recursive function to compute the convergents.
We can also use QuickCheck to write a property test, verifying that a rational number is its own final convergent. This gives a sanity check on our implementation.
prop_naiveConvergents_end_at_Rational :: NonNegative Rational -> Property prop_naiveConvergents_end_at_Rational (NonNegative r) = last (naiveConvergents (cfFromRational r)) === r
You may guess from my choice of name that I am not happy with this implementation. The problem with naiveConvergents is that it recursively maps a lambda over the entire tail of the list. As it recurses deeper into the list, we build up a bunch of nested mapped lambdas, and end up evaluating O(n²) of these lambdas to produce only n convergents.
Mobius transformations
Let’s fix that quadratic slowdown. A more efficient implementation can be obtained by defunctionalizing. The lambdas have the form \x -> fromInteger n + 1 / x. To avoid quadratic work, we need a representation that lets us compose functions of this form into something that can be applied in constant time. What works is mobius transformations.
A mobius transformation is a function of the form:
For our purposes, the coefficients a, b, c, and d are always integers. Instead of an opaque lambda, we represent a mobius transformation by its four coefficients.
data Mobius where Mobius :: Integer -> Integer -> Integer -> Integer -> Mobius
deriving instance Eq Mobius deriving instance Show Mobius
We seek to rewrite our computation of convergents using mobius transformations. First of all, we will restructure it as a left fold, which exposes the mobius transformation itself as an accumulator. We’ll want these building blocks:
An identity mobius transformation, to initialize the accumulator.
A composition to combine two mobius transformations into a single one.
In other words, mobius transformations should form a monoid!
There are axioms for monoids: mempty should act like an identity, and <> should be associative. We can test these with property tests.
This is as good a time as any to set up a QuickCheck generator for mobius transformations. To keep things sane, we want to always want to choose mobius transformations that have non-zero denominators, since otherwise the function is undefined on all inputs. We will also want non-negative coefficients in the denominator, since this ensures the transformation is defined (and monotonic, which will matter later on) for all non-negative input values.
instance Arbitrary Mobius where arbitrary = suchThat ( Mobius <$> arbitrary <*> arbitrary <*> (getNonNegative <$> arbitrary) <*> (getNonNegative <$> arbitrary) ) (\(Mobius _ _ c d) -> max c d > 0)
shrink (Mobius a b c d) = [ Mobius a' b' c' d' | (a', b', NonNegative c', NonNegative d') <- shrink (a, b, NonNegative c, NonNegative d), max c' d' > 0 ]
With the generator in place, the tests are trivial to write:
prop_Mobius_id :: Mobius -> Property prop_Mobius_id m = mempty <> m === m .&&. m <> mempty === m
Now we return to computing convergents. Here is the improved implementation.
convergents :: CFrac -> [Rational] convergents = go mempty where go m Inf = [] go m (n :+/ x) = mobiusLimit m' : go m' x where m' = m <> Mobius n 1 1 0
mobiusLimit (Mobius a _ c _) = a % c
In the expression, go m x, m is a mobius transformation that turns a convergent of x into a convergent for the entire continued fraction. When x is the entire continued fraction, m is the identity function. As we recurse into the continued fraction, we compose transformations onto m so that this remains true. The lambda \x -> fromInteger n + 1 / x from the naive implementation is now defunctionalized into Mobius n 1 1 0.
The remaining task is to compute a truncated value of the continued fraction at each step. Recall that a terminating continued fraction is one which has an infinite term. To determine this truncated value, then, we want to consider the limit of the accumulated mobius transformation as its input tends to infinity. This is:
and that completes the implementation. A quick test helps us to be confident in the result.
Let’s also consider the conversion from continued fractions to decimals. This time, both representations can be approximated by just truncating, so instead of producing a sequence of approximations as we did with rational numbers, we will just a single decimal that lazily adds to the representation with more precision.
Mobius transformations are still a good tool for this job, but we need to refine our earlier observations. Consider the general form of a mobius transformation again:
It’s worth taking the time now to play around with different choices for a, b, c, and d, and what they do to the function. You can do so with the Desmos link below:
We previously noted one of these bounds, and now add the other:
As long as c and d are positive (remember when we agreed to keep them so?), f is monotonic on the entire interval [0, ∞), so f is bounded by these two fractions. In particular, if a/c and b/d have the same integer part, then we know this is the integer part of the result of f, regardless of the (always non-negative) value of x. We can express this insight as a function:
mobiusIntPart :: Mobius -> Maybe Integer mobiusIntPart (Mobius a b c d) | c /= 0 && d /= 0 && n1 == n2 = Just n1 | otherwise = Nothing where n1 = a `div` c n2 = b `div` d
We can now proceed in a manner similar to the computation of convergents. We maintain a mobius transformation which maps the remaining continued fraction to the remaining decimal. Initially, that is the identity. But this time, instead of blindly emitting an approximation at each step, we make a choice:
If the mobius transformation is bounded to guarantee the integer part of its result, then we can produce that decimal digit. We then update the transformation m to yield the remaining decimal places after that. This will tend to widen its bounds.
Otherwise, we will pop off the integer part of the continued fraction, and update the transformation to expect the remaining continued fraction terms after that. This will tend to narrow its bounds, so we’ll be closer to producing a new decimal digit.
If we ever end up with the zero transformation (that is, a and b are both zero), then all remaining decimal places will be zero, so we can stop. If we encounter an infinite term of input, though, we still need to continue, but we can update b and d to match a and c, narrowing those bounds to a single point to indicate that we now know the exact value of the input.
Here is the implementation:
cfToBase :: Integer -> CFrac -> [Integer] cfToBase base = go mempty where go (Mobius 0 0 _ _) _ = [] go m x | Just n <- mobiusIntPart m, let m' = Mobius base (-base * n) 0 1 <> m = n : go m' x go m (n :+/ x) = go (m <> Mobius n 1 1 0) x go (Mobius a _ c _) Inf = go (Mobius a a c c) Inf
cfToDecimal :: CFrac -> String cfToDecimal Inf = "Inf" cfToDecimal x = case cfToBase 10 x of [] -> "0.0" [z] -> show z ++ ".0" (z : digits) -> show z ++ "." ++ concatMap show digits
To test this, we will compare the result to standard Haskell output. However, Haskell’s typical Show instances for fractional types are too complex, producing output like 1.2e6 that we don’t want to match. The Numeric module has the simpler functionality we want. There are some discrepancies due to rounding error, as well, so the test will ignore differences in the last digit of output from the built-in types.
prop_cfToDecimal_matchesRational :: NonNegative Rational -> Property prop_cfToDecimal_matchesRational (NonNegative r) = take n decFromRat === take n decFromCF where decFromRat = showFFloat Nothing (realToFrac r :: Double) "" decFromCF = cfToDecimal (cfFromRational r) n = max 10 (length decFromRat - 1)
Generalized Continued Fractions
Earlier, we settled for only an approximation of π, and I mentioned that π doesn’t have a nice pattern as a continued fraction. That’s true, but it’s not the whole story. If we relax the definition of a continued fraction just a bit, there areseveral known expressions for π as a generalized continued fraction. The key is to allow numerators other than 1.
Here’s one that’s fairly nice:
Generalized continued fractions don’t have many of the same nice properties that standard continued fractions do. They are not unique, and sometimes converge very slowly (or not at all!), yielding poor rational approximations when truncated. But we can compute with them using mobius transformations, using the same algorithms from above. In particular, we can use the same tools as above to convert from a generalized continued fraction to a standard one.
First, we’ll define a type for generalized continued fractions.
data GCFrac where (:+#/) :: (Integer, Integer) -> GCFrac -> GCFrac GInf :: GCFrac
deriving instance Show GCFrac
This time I haven’t defined an Eq instance, because representations are not unique. Converting from a standard to a generalized continued fraction is trivial.
The conversion in the other direction, though, requires a mobius-like algorithm.
gcfToCFrac :: GCFrac -> CFrac gcfToCFrac = go mempty where go (Mobius a _ c _) GInf = cfFromFrac a c go m gcf@((int, numer) :+#/ denom) | Just n <- mobiusIntPart m = n :+/ go (Mobius 0 1 1 (- n) <> m) gcf | otherwise = go (m <> Mobius int numer 1 0) denom
We can write a property test to verify that, at least, this gives a correct round trip from continued fractions to generalized, and back to standard again.
prop_GCFrac_roundtrip :: NonNegative Rational -> Property prop_GCFrac_roundtrip (NonNegative r) = gcfToCFrac (cfToGCFrac x) === x where x = cfFromRational r
Here’s the definition of π above, expressed in code, and converted to a continued fraction. I’ve written a test to verify that it mostly agrees with the approximate value we defined earlier, just as a sanity check on the implementation.
gcfPi = (0, 4) :+#/ go 1 where go i = (2 * i - 1, i ^ 2) :+#/ go (i + 1)
exactPi = gcfToCFrac gcfPi
prop_correct_pi :: Property prop_correct_pi = take 17 (cfToDecimal approxPi) === take 17 (cfToDecimal exactPi)
Results
We now have built enough to get some interesting results. For example, we have exact representations several irrational numbers, and we can use those to obtain both rational approximations and long decimal representations of each.
In addition to running our tests, we will print the continued fraction terms, convergents, and decimal representation of each of our test values. Check out the full code at https://code.world/haskell#PemataxO6pmYz99dOpaeT9g.
Most of these are exact irrational values, so we can only look at a finite prefix of the convergents, which go on forever getting more and more accurate (at the expense of larger denominators). I’ve chopped them off at 50 characters, so that they comfortably fit on the screen, but you can change that to see as much of the value as you like.
With this data in front of us, we can learn more about continued fractions and how they behave.
Did you notice that the convergents of π jump from 355/113 all the way to 103993/33102? That’s because 355/113 is a remarkably good approximation of π. You have to jump to a much larger denominator to get a better approximation. You can observe this same fact directly from the continued fraction terms. The fifth term is 292, which is unusually large. 292 is the next term after the truncation that yields the convergent 355/113. Large terms like that in a continued fraction indicate unusually good rational approximations.
Conversely, what if there are no large continued fraction terms? More concretely, what can we say about the number that has all 1s as its continued fraction? That’s the golden ratio! What this means is that there are no particularly good rational approximations to the golden ratio. And you can see that, and hear it!
Flowers tend to grow new petals or seeds at an angle determined by the golden ratio, specifically because there are no small intervals (i.e., small denominators) after which they will overlap.
Musical notes sound harmonic when their frequencies are close to simple rational numbers. The least harmonic frequencies are those related by the golden ratio, and you can hear it!
(By the way, did you notice the fibonacci numbers in the convergents of the golden ratio? That’s a whole other topic I won’t get into here.)
Part 3: Arithmetic
Now that we’ve got a nice type and some conversions, let’s turn our attention to computing with continued fractions.
There are a few computations that are easy, so let’s warm up with those.
Computing the reciprocal of a continued fraction turns out to be almost trivial, because continued fractions are built out of reciprocals! All we need to do for this one is shift all the terms by one… in either direction, really. Since our normal form only allows zero as the first term, we’ll make the decision based on whether the whole part is already zero. We also need a special case for 1 to prevent building a non-normal representation.
cfRecip :: CFrac -> CFrac cfRecip (0 :+/ x) = x cfRecip (1 :+/ Inf) = 1 :+/ Inf cfRecip x = 0 :+/ x
We will test a few expected properties:
The reciprocal is self-inverse.
The result matches taking a reciprocal in the rationals.
The reciprocal always gives an answer in normal form.
Comparing two continued fractions is also not too difficult. It’s similar to comparing decimals, in that you look down the sequence for the first position where they differ, and that determines the result. However, since we take reciprocals at each term of a continued fraction, we’ll need to flip the result at each term.
cfCompare :: CFrac -> CFrac -> Ordering cfCompare Inf Inf = EQ cfCompare _ Inf = LT cfCompare Inf _ = GT cfCompare (a :+/ a') (b :+/ b') = case compare a b of EQ -> cfCompare b' a' r -> r
Let’s write a QuickCheck property to verify that the result matches the Ord instance for Rational, and then define an Ord instance of our own.
To make any progress on the rest of basic arithmetic, we must return to our trusty old hammer: the mobius transformation. Recall that we’ve used the mobius transformation in two ways so far:
To produce convergents, we produced mobius transformations from terms, composed them together, and also produced a new approximation at each step.
To produce decimals, we looked at the bounds on the result of a mobius transformation, and proceeded in one of two ways: produce a bit of output, and then add a new mobius transformation to the output side; or consume a term of input, and then add a new mobius transformation to the input side.
We will now implement mobius transformations that act on continued fractions and produce new continued fractions. The strategy is the same as it was when producing decimals, but modified to produce continued fraction terms as output instead of decimal digits.
That leads to this implementation.
cfMobius :: Mobius -> CFrac -> CFrac cfMobius (Mobius a _ c _) Inf = cfFromFrac a c cfMobius (Mobius _ _ 0 0) _ = Inf cfMobius m x | Just n <- mobiusIntPart m = n :+/ cfMobius (Mobius 0 1 1 (- n) <> m) x cfMobius m (n :+/ x) = cfMobius (m <> Mobius n 1 1 0) x
There are two properties to test here. The first is that computations on continued fractions match the same computations on rational numbers. To implement this test, we’ll need an implementation of mobius transformations on rational numbers. Then we’ll test that cfMobius gives results in their canonical form. For both tests, we don’t care about transformations whose rational results are negative, as we will just agree not to evaluate them.
mobius :: (Eq a, Fractional a) => Mobius -> a -> Maybe a mobius (Mobius a b c d) x | q == 0 = Nothing | otherwise = Just (p / q) where p = fromInteger a * x + fromInteger b q = fromInteger c * x + fromInteger d
prop_cfMobius_matches_Rational :: Mobius -> NonNegative Rational -> Property prop_cfMobius_matches_Rational m (NonNegative r) = case mobius m r of Just x | x >= 0 -> cfMobius m (cfFromRational r) === cfFromRational x _ -> discard
prop_cfMobius_isCanonical :: Mobius -> NonNegative Rational -> Property prop_cfMobius_isCanonical m (NonNegative r) = case mobius m r of Just rat | rat >= 0 -> let x = cfMobius m (cfFromRational r) in counterexample (show x) (isCanonical x) _ -> discard
The punchline here is that, using an appropriate mobius transformation, we can perform any of the four arithmetic functions — addition, subtraction, multiplication, or division — involving one continuous fraction and one rational number.
Binary operations on continued fractions
This was all sort of a warm-up, though, for the big guns: binary operations on two continued fractions, yielding a new continued fraction. Here, for the first time, our trusted mobius transformations fail us. Indeed, this remained an unsolved problem until 1972, when Bill Gosper devised a solution. That solution uses functions of the form:
Again, for our purposes, all coefficients will be integers, and the coefficients in the denominator will always be positive. Since these are the generalization of mobius transformations to binary operations, we can call them the bimobius transformations.
deriving instance Eq Bimobius deriving instance Show Bimobius
The rest of the implementation nearly writes itself! We simply want to follow the same old strategy. We keep a bimobius transformation that computes the rest of the output, given the rest of the inputs. At each step, we emit more output if possible, and otherwise consume a term from one of two inputs. As soon as either input is exhausted (if, indeed that happens), the remaining calculation reduces to a simple mobius transformation of the remaining argument.
First, we need an analogue of mobiusIntPart, which was how we determined whether an output was available.
bimobiusIntPart :: Bimobius -> Maybe Integer bimobiusIntPart (BM a b c d e f g h) | e /= 0 && f /= 0 && g /= 0 && h /= 0 && n2 == n1 && n3 == n1 && n4 == n1 = Just n1 | otherwise = Nothing where n1 = a `div` e n2 = b `div` f n3 = c `div` g n4 = d `div` h
The other building block we used to implement mobius transformations was composition with the Monoid instance. Unfortunately, that’s not so simple, since bimobius transformations have two inputs. There are now three composition forms. First, we can compose the mobius transformation on the left, consuming the output of the bimobius:
(<>||) :: Mobius -> Bimobius -> Bimobius -- (mob <>|| bimob) x y = mob (bimob x y) Mobius a1 b1 c1 d1 <>|| BM a2 b2 c2 d2 e2 f2 g2 h2 = BM a b c d e f g h where a = a1 * a2 + b1 * e2 b = a1 * b2 + b1 * f2 c = a1 * c2 + b1 * g2 d = a1 * d2 + b1 * h2 e = c1 * a2 + d1 * e2 f = c1 * b2 + d1 * f2 g = c1 * c2 + d1 * g2 h = c1 * d2 + d1 * h2
We can also compose it on the right side, transforming either the first or second argument of the bimobius.
(||<) :: Bimobius -> Mobius -> Bimobius -- (bimob ||< mob) x y = bimob (mob x) y BM a1 b1 c1 d1 e1 f1 g1 h1 ||< Mobius a2 b2 c2 d2 = BM a b c d e f g h where a = a1 * a2 + c1 * c2 b = b1 * a2 + d1 * c2 c = a1 * b2 + c1 * d2 d = b1 * b2 + d1 * d2 e = e1 * a2 + g1 * c2 f = f1 * a2 + h1 * c2 g = e1 * b2 + g1 * d2 h = f1 * b2 + h1 * d2
(||>) :: Bimobius -> Mobius -> Bimobius -- (bimob ||> mob) x y = bimob x (mob y) BM a1 b1 c1 d1 e1 f1 g1 h1 ||> Mobius a2 b2 c2 d2 = BM a b c d e f g h where a = a1 * a2 + b1 * c2 b = a1 * b2 + b1 * d2 c = c1 * a2 + d1 * c2 d = c1 * b2 + d1 * d2 e = e1 * a2 + f1 * c2 f = e1 * b2 + f1 * d2 g = g1 * a2 + h1 * c2 h = g1 * b2 + h1 * d2
That’s a truly dizzying bunch of coefficients! I’m definitely not satisfied without property tests to ensure they match the intended meanings. To do that, we need two pieces of test code.
First, we want to generate random bimobius transformations with an Arbitrary instance. As with mobius transformations, the instance will guarantee that the denominators are non-zero and have non-negative coefficients.
shrink (BM a b c d e f g h) = [ BM a' b' c' d' e' f' g' h' | ( a', b', c', d', NonNegative e', NonNegative f', NonNegative g', NonNegative h' ) <- shrink ( a, b, c, d, NonNegative e, NonNegative f, NonNegative g, NonNegative h ), maximum [e', f', g', h'] > 0 ]
And second, we want a simple and obviously correct base implementation of the bimobius transformation to compare with.
bimobius :: (Eq a, Fractional a) => Bimobius -> a -> a -> Maybe a bimobius (BM a b c d e f g h) x y | q == 0 = Nothing | otherwise = Just (p / q) where p = fromInteger a * x * y + fromInteger b * x + fromInteger c * y + fromInteger d q = fromInteger e * x * y + fromInteger f * x + fromInteger g * y + fromInteger h
With this in mind, we’ll check each of the three composition operators performs as expected on rational number, at least. There’s one slight wrinkle: when composing the two functions manually leads to an undefined result, the composition operator sometimes cancels out the singularity and produces an answer. I’m willing to live with that, so I discard cases where the original result is undefined.
prop_mob_o_bimob :: Mobius -> Bimobius -> Rational -> Rational -> Property prop_mob_o_bimob mob bimob r1 r2 = case mobius mob =<< bimobius bimob r1 r2 of Just ans -> bimobius (mob <>|| bimob) r1 r2 === Just ans Nothing -> discard
prop_bimob_o_leftMob :: Bimobius -> Mobius -> Rational -> Rational -> Property prop_bimob_o_leftMob bimob mob r1 r2 = case (\x -> bimobius bimob x r2) =<< mobius mob r1 of Just ans -> bimobius (bimob ||< mob) r1 r2 === Just ans Nothing -> discard
prop_bimob_o_rightMob :: Bimobius -> Mobius -> Rational -> Rational -> Property prop_bimob_o_rightMob bimob mob r1 r2 = case (\y -> bimobius bimob r1 y) =<< mobius mob r2 of Just ans -> bimobius (bimob ||> mob) r1 r2 === Just ans Nothing -> discard
Now to the task at hand: implementing bimobius transformations on continued fractions.
cfBimobius :: Bimobius -> CFrac -> CFrac -> CFrac cfBimobius (BM a b _ _ e f _ _) Inf y = cfMobius (Mobius a b e f) y cfBimobius (BM a _ c _ e _ g _) x Inf = cfMobius (Mobius a c e g) x cfBimobius (BM _ _ _ _ 0 0 0 0) _ _ = Inf cfBimobius bm x y | Just n <- bimobiusIntPart bm = let bm' = Mobius 0 1 1 (- n) <>|| bm in n :+/ cfBimobius bm' x y cfBimobius bm@(BM a b c d e f g h) x@(x0 :+/ x') y@(y0 :+/ y') | g == 0 && h == 0 = consumeX | h == 0 || h == 0 = consumeY | abs (g * (h * b - f * d)) > abs (f * (h * c - g * d)) = consumeX | otherwise = consumeY where consumeX = cfBimobius (bm ||< Mobius x0 1 1 0) x' y consumeY = cfBimobius (bm ||> Mobius y0 1 1 0) x y'
There are a few easy cases at the start. If either argument is infinite, then the infinite terms dominate, and the bimobius reduces to a unary mobius transformation of the remaining argument. On the other hand, if the denominator is zero, the result is infinite.
Next is the output case. This follows the same logic as the mobius transformations above:
But since m and m’ are now bimobius transformations, we just use one of the special composition forms defined earlier.
The remaining case is where we don’t have a new term to output, and must expand a term of the input, using the now-familiar equation, but composing with the ||< or ||> operators, instead.
There’s a new wrinkle here, though! Which of the two inputs should we expand? It seems best to make the choice that narrows the bounds the most, but I’ll be honest: I don’t actually understand the full logic behind this choice. I’ve simply copied it from Rosetta Code, where it’s not explained in any detail. We will rely on testing for confidence that it works.
Speaking of testing, we definitely need a few tests to ensure this works as intended. We’ll test the output against the Rational version of the function, and that the result is always in canonical form.
prop_cfBimobius_matches_Rational :: NonNegative Rational -> NonNegative Rational -> Bimobius -> Property prop_cfBimobius_matches_Rational (NonNegative r1) (NonNegative r2) bm = case bimobius bm r1 r2 of Just x | x >= 0 -> cfFromRational x === cfBimobius bm (cfFromRational r1) (cfFromRational r2) _ -> discard
prop_cfBimobius_isCanonical :: NonNegative Rational -> NonNegative Rational -> Bimobius -> Bool prop_cfBimobius_isCanonical (NonNegative r1) (NonNegative r2) bm = case bimobius bm r1 r2 of Just x | x >= 0 -> isCanonical (cfBimobius bm (cfFromRational r1) (cfFromRational r2)) _ -> discard
If you like, it’s interesting to compare the implementations of continued fraction arithmetic here with some other sources, such as:
These sources are great, but I think the code above is nicer. The reason is that with lazy evaluation instead of mutation, you can read the code more mathematically as a set of equations, and see what’s going on more clearly. This is quite different from the imperative implementations, which modify state in hidden places and require a lot of mental bookkeeping to keep track of it all. As I said, Haskell is a joy for this kind of programming.
Tying it all together
We now have all the tools we need to write Num and Fractional instances for CFrac. That’s a great way to wrap this all together, so that clients of this code don’t need to worry about mobius and bimobius transformations at all.
checkNonNegative :: CFrac -> CFrac checkNonNegative Inf = Inf checkNonNegative x@(x0 :+/ x') | x < 0 = error "checkNonNegative: CFrac is negative" | x > 0 = x | otherwise = x0 :+/ checkNonNegative x'
instance Num CFrac where fromInteger n | n >= 0 = n :+/ Inf | otherwise = error "fromInteger: CFrac cannot be negative"
(+) = cfBimobius (BM 0 1 1 0 0 0 0 1)
x - y = checkNonNegative (cfBimobius (BM 0 1 (-1) 0 0 0 0 1) x y)
instance Fractional CFrac where fromRational x | x < 0 = error "fromRational: CFrac cannot be negative" | otherwise = cfFromRational x
recip = cfRecip
(/) = cfBimobius (BM 0 1 0 0 0 0 1 0)
You can find the complete code and tests for this section at https://code.world/haskell#P_T4CkfWGu3bgSwFV0R04Sg. In addition to running the tests, I’ve included a few computations, demonstrating for example that:
Part 4: Going faster
Everything we’ve done so far is correct, but there’s one way that it’s not terribly satisfying. It can do a lot of unnecessary work. In this part, we’ll try to fix this performance bug.
Consider, for example, a computation like x + 0.7. We know (and, indeed, Haskell also knows!) that 0.7 is a rational number. We should, then, be able to work this out by applying cfMobius to a simple mobius transformation.
But that’s not what GHC does. Instead, GHC reasons like this:
We assume that x is a CFrac. But + requires that both arguments have the same type, so 0.7 must also be a CFrac.
Since 0.7 is a literal, GHC will implicitly add a fromRational, applied to the rational form 7/10, which (using the Fractional instance defined in the last part) uses cfFromRational to convert 0.7 into the continued fraction 0 :+/ 1 :+/ 2 :+/ 3 :+/ Inf.
Now, since we have two CFrac values, we must use the heavyweight cfBimobius to compute the answer, choosing between the two streams at every step. In the process, cfBimobius must effectively undo the conversion of 7/10 into continued fraction form, doubling the unnecessary work.
Rewriting expressions
We can fix this with GHC’s rewrite rules. They can avoid this unnecessary conversion, and reduce constant factors at the same time, by rewriting expressions to use cfMobius instead of cfBimobius when one argument need not be a continued fraction. Similarly, when cfMobius is applied to a rational argument, it can be replaced by a simpler call to cfFromFrac after just evaluating the mobius transformation on the Rational directly. Integers are another special case: less expensive because their continued fraction representations are trivial, but but we might as well rewrite them, too.
{-# RULES "cfrac/cfMobiusInt" forall a b c d n. cfMobius (Mobius a b c d) (n :+/ Inf) = cfFromFrac (a * n + b) (c * n + d) "cfrac/cfMobiusRat" forall a b c d p q. cfMobius (Mobius a b c d) (cfFromFrac p q) = cfFromFrac (a * p + b * q) (c * p + d * q) "cfrac/cfBimobiusInt1" forall a b c d e f g h n y. cfBimobius (BM a b c d e f g h) (n :+/ Inf) y = cfMobius (Mobius (a * n + c) (b * n + d) (e * n + g) (f * n + h)) y "cfrac/cfBimobiusRat1" forall a b c d e f g h p q y. cfBimobius (BM a b c d e f g h) (cfFromFrac p q) y = cfMobius ( Mobius (a * p + c * q) (b * p + d * q) (e * p + g * q) (f * p + h * q) ) y "cfrac/cfBimobiusInt2" forall a b c d e f g h n x. cfBimobius (BM a b c d e f g h) x (n :+/ Inf) = cfMobius (Mobius (a * n + b) (c * n + d) (e * n + f) (g * n + h)) x "cfrac/cfBimobiusRat2" forall a b c d e f g h p q x. cfBimobius (BM a b c d e f g h) x (cfFromFrac p q) = cfMobius ( Mobius (a * p + b * q) (c * p + d * q) (e * p + f * q) (g * p + h * q) ) x #-}
There’s another opportunity for rewriting, and we’ve actually already used it! When mobius or bimobius transformations are composed together, we can do the composition in advance to produce a single transformation. Especially since these transformations usually have simple integer arguments that GHC knows how to optimize, this can unlock a lot of other arithmetic optimizations.
Fortunately, we’ve already defined and tested all of the composition operators we need here.
{-# RULES "cfrac/mobius_o_mobius" forall m1 m2 x. cfMobius m1 (cfMobius m2 x) = cfMobius (m1 <> m2) x "cfrac/mobius_o_bimobius" forall m bm x y. cfMobius m (cfBimobius bm x y) = cfBimobius (m <>|| bm) x y "cfrac/bimobius_o_mobiusLeft" forall bm m x y. cfBimobius bm (cfMobius m x) y = cfBimobius (bm ||< m) x y "cfrac/bimobius_o_mobiusRight" forall bm m x y. cfBimobius bm x (cfMobius m y) = cfBimobius (bm ||> m) x y #-}
Inlining, and not
This all looks great… but it doesn’t work. The problem is that GHC isn’t willing to inline the right things at the right times to get to exactly the form we need for these rewrite rules to fire. To fix this, I had to go back through all of the previous code, being careful to annotate many of the key functions with {-# INLINE ... #-} pragmas.
There are two forms of this pragma. When a trivial function might just get in the way of a rewrite rule, it’s annotated with a simple INLINE pragma so that GHC can inline and hopefully eliminate the function.
But when a function actually appears on the left hand side of a rewrite rule, though, we need to step more carefully. Inlining that function can prevent the rewrite rule from firing. We could use NOINLINE for this, but I’ve instead picked the form of inline that includes a phase number. The phase number prevents inlining from occurring prior to that phase. It looks something like:
{-# INLINE [2] cfRecip #-}
Choosing a phase number seems to be something of a black art. I picked 2, and it worked, so… maybe try that one?
All these edits mean we have completely new versions of all of the code, and it’s in these files:
Rewrite rules are a bit of a black art, and it’s nice to at least get some confirmation that they are working. GHC can tell us what’s going on if we pass some flags. Here are some useful ones:
-ddump-rule-rewrites will print out a description of every rewrite rule that fires, including the before-and-after code.
-ddump-simpl will print out the final Core, which is GHC’s Haskell-like intermediate language, after the simplifier (which applies these rules) is done. Adding -dsuppress-all makes the result more readable by hiding metadata we don’t care about.
In CodeWorld, we can pass options to GHC by adding an OPTIONS_GHC pragma to the source code. If you looked at the Part 4 link above, you saw that I’ve done so. The output is long! But there are a few things we can notice.
First, we named our rules, so we can search for the rules by name to see if they have fired. They do! We see this:
I highlighted the timesInteger and plusInteger rules, as well. This is GHC taking the result of our rules, which involve a bunch of expressions like a1 * a2 + b1 * c2, and working out the math at compile time because it already knows the arguments. This is great! We want that to happen.
But what about the result? For this, we turn to the -ddump-simpl output. It’s a bit harder to read, but after enough digging, we can find this (after some cleanup):
This is essentially the expression take 50 (cfDecimal (cfMobius (Mobius 1 1 0 2) sqrt5)). Recall that what we originally wrote was (1 + sqrt5) / 2. We have succeeded in turning an arithmetic expression of CFrac values into a simple mobius transformation at compile time!
Conclusion
We finally reach the end of our journey. There are still some loose ends, as there always are. For example:
There’s still the thing with negative numbers, which I left as an exercise.
Arithmetic hangs when computing with irrational numbers that produce rational results. For example, try computing sqrt2 * sqrt2, and you’ll be staring at a blank screen while your CPU fan gets some exercise. This might seems like a simple bug, but it’s actually a fundamental limitation of our approach. An incorrect term arbitrarily far into the expression for sqrt2 could change the integer part of the result, so any correct algorithm must read the entire continued fraction before producing an answer. But the entire fraction is infinite! You could try to fix this by special-casing the cyclic continued fractions, but you can never catch all the cases.
There’s probably plenty more low-hanging fruit for performance. I haven’t verified the rewrite rules in earnest, really. Something like Joachim Breitner’s inspection testing idea could bring more rigor to this. (Sadly, libraries for inspection testing require Template Haskell, so you cannot try it within CodeWorld.)
Other representations for exact real arithmetic offer more advantages. David Lester, for example, has written a very concise exact real arithmetic library using Cauchy sequences with exponential convergence.
In the end, though, I hope you’ve found this part of the journey rewarding in its own right. We developed a non-trivial library in Haskell, derived the code using equational reasoning (even if it was a bit fuzzy), tested it with property testing, and optimized it with rewrite rules. All of these play very well with Haskell’s declarative programming style.
Two years ago I started working on a new formatter for Haskell source code.
My idea was to take advantage of the parser of GHC itself, which finally
became available as a library around that time. Soon Tweag supported the
initiative and the project became Ormolu. It was
announced in May, right before ZuriHac 2019. Many
people kindly helped me with it during the Hackathon. We went on to release
the first version in October 2019.
The use of the GHC parser and a solid approach to testing won Ormolu the
reputation of a dependable tool. More and more industrial users choose it as
their Haskell formatter. In summer 2020 a rather big company had decided
that they wanted to format their Haskell code with Ormolu, which resulted in
a three-month full-time contract. A new level of quality was reached.
I think it is cool to be paid to work full-time on a project like this. If
you concur, we have good news for you! Tweag currently has an opening for an
intern who would like to work on the formatter. This is a project for
someone who would enjoy iteratively improving a popular Haskell tool and
learn about GHC AST, its parser, and perhaps a bit of Nix.
There are various issues affecting Ormolu, and fixing these would have a
positive impact on the user experience. The internship would address them in
severity order:
Upgrading the GHC parser. Presently, Ormolu uses
ghc-lib-parser. GHC 9.0 fixes some long-standing
issues, and we can take advantage of that by
switching to ghc-lib-parser-9.0.1.xxx.
Please let us know if you are interested! Include a cover letter
describing your Haskell experience; familiarity with GHC AST is a plus. We
will collect applications till Tuesday, June 1, 2021. The internship can
start any time after the offer is made, subject to mutual availability.
Internships typically last 12 weeks, although the duration may be adjusted
if necessary. If you have any questions, feel free to email me.
The GHC developers are very happy to announce the availability of the second alpha release in the 9.2.1 series. Binary distributions, source distributions, and documentation are available from downloads.haskell.org.
GHC 9.2 will bring a number of exciting features including:
Many changes in the area of records, including the new RecordDotSyntax and NoFieldSelectors language extensions, as well as Support for DuplicateRecordFields with PatternSynonyms.
Introduction of the new GHC2021 language extension set, giving users convenient access to a larger set of language extensions which have been long considered stable.
Merge of ghc-exactprint into the GHC tree, providing infrastructure for source-to-source program rewriting out-of-the-box.
Introduction of a BoxedRepRuntimeRep, allowing for polymorphism over levity of boxed objects (#17526)
Implementation of the UnliftedDataTypes extension, allowing users to define types which do not admit lazy evaluation
The new -hi profiling mechanism which provides significantly improved insight into thunk leaks.
Support for the ghc-debug out-of-process heap inspection library
Support for profiling of pinned objects with the cost-centre profiler (#7275)
Introduction of Haddock documentation support in TemplateHaskell (#5467)
Proper support for impredicative types in the form of Quick-Look impredicativity.
A native code generator backend for AArch64.
This pre-release brings nearly 50 fixes relative to the first alpha, although the long-awaited ARM NCG backend hasn’t quite landed yet.
As always, do give this a try and open a ticket if you see anything amiss.
Nothing much to be proud of, but yesterday I found out that servant-docs was
marked broken in nixpkgs even though it builds just fine and this morning I
decided to do something about it.
I spend quite a bit of time thinking about what the end of history for programming might look like. By the “end of history” I mean the point beyond which programming paradigms would not evolve significantly.
I care about programming’s “destiny” because I prefer to work on open source projects that bring us closer to that final programming paradigm. In my experience, this sort of work has greater longevity, higher impact, and helps move the whole field of programming forward.
So what would the end of history look like for programming? Is it:
… already here?
Some people treat programming as a solved problem and view all new languages and paradigms as reskins of old languages or paradigms. From their point of view all that remains is to refine our craft by slowly optimizing things, weeding out bugs or addressing non-technical issues like people management or funding.
I personally do not subscribe to this philosophy because I believe, at a very minimum, that functional programming paradigms will slowly displace object-oriented and imperative programming paradigms (although functional programming might not necessarily be the final programming paradigm).
… artificial intelligence?
Maybe machines will translate our natural language instructions into code for us, relieving us of the burden of precisely communicating our intent. Or maybe some sufficiently smart AI-powered IDE will auto-complete most of our program for us.
Truthfully, I’m not certain what is the correct answer, but I’ll present my own guess for what the end of history for programming looks like.
Mathematical DSLs
My view is that the next logical step for programming is to split into two non-overlapping programming domains:
runtime building for …
… mathematical programming languages
Specifically, I expect programming languages to evolve to become more mathematical in nature where the programs users author to communicate their intent resemble pure mathematical expressions.
For example, consider the following mathematical specifications of the boolean logical “and” operator and the function composition operator:
True&& x = x x &&True= x False&&False=False (f . g)(x) = f(g(x))
These mathematical specifications are also executable Haskell code (albeit with extra parentheses to resemble mainstream function syntax). Haskell is one example of a programming language where the code aspires to resemble pure mathematical expressions and definitions.
Any language presenting such an idealized mathematical interface requires introducing a significant amount of complexity “under the hood” since the real world is messy. This is where runtime building comes into play to gloss over such ugly details.
In other words, I’m predicting that the end of history for programming is to become an interface bridging pure mathematical expressions to the real world.
The past and present
Let me give a few examples of where this trend towards mathematical userland code is already playing out:
Memory management
Manual memory management used to be a “userland” concern for most programming languages, but the general trend for new languages is automatic memory management (with the exception of Rust). Memory management used to be an explicit side effect that programmers had to care about and pushing memory management down into the runtime (via garbage collection or other methods) has made programming languages more pure, allowing them to get one step closer to idealized mathematical expressions.
Indeed, Rust is the exception that proves the rule, as Rust is widely viewed as better suited for building runtimes rather than being used for high-level specification of intent.
However, functional programming is not a free win. Supporting higher-order functions and closures efficiently isn’t easy (especially for compiled languages), which is why less sophisticated language implementations tend to be more imperative and less functional.
Evaluation order (especially laziness)
I’ve always felt that “lazy evaluation” doesn’t do a good job of selling the benefits of Haskell’s evaluation model. I prefer to think of Haskell as having “automatic evaluation management”1. In other words, the programmer specifies the expressions as a graph of dependent computations and the runtime figures out the most efficient order in which to reduce the graph.
This is yet another example of where we push something that used to be a userland concern (order of evaluation) into the runtime. Glossing over evaluation order frees us to specify things in a more mathematical style, because in mathematics the evaluation order is similarly irrelevant.
The present and future
A common pattern emerges when we study the above trends:
Push a userland concern into a runtime concern, which:
… makes programs more closely resemble pure mathematical expressions, and:
… significantly increases the complexity of the runtime.
You might wonder: what are some other userland concerns that might eventually get pushed into runtime concerns in the near future? Some examples I can think of are:
Package management
This one is so near in the future that it’s already happening (see: Nix and Dhall). Both languages provide built-in support for fetching code, rather than handling packages out-of-band using a standalone package management tool. This language-level support allows programs to embed external code as if it were a pure sub-expression, more closely approximating the mathematical ideal.
Error handling
This one requires more explanation: I view type systems as the logical conclusion of pushing error handling into the “runtime” (actually, into the type-checker, not the runtime, to be pedantic). Dhall is an example of a language that takes this idea to the extreme: Dhall has no userland support for raising or catching errors because all errors are type errors2.
Advances in dependently-typed and total functional programming languages get us closer to this ideal of pushing error handling into a runtime concern.
Logging
I’m actually surprised that language support for pervasively logging everything hasn’t been done already (or maybe it has happened and I missed it). It seems like a pretty mundane thing languages could implement, especially for application domains that are not performance-sensitive.
Many languages already support profiling, and it seems like it wouldn’t be a big leap to turn profiling support into logging support if the user is willing to spend their performance budget on logging.
Logging is one of those classic side effects that is an ugly detail that “ruins” code that would have otherwise been pure and mathematical.
Services
Service-oriented architectures are another thing that tends to get in the way of writing pure side-effect-free code.
I’m not exactly sure what a service-oriented language runtime would look like, but I don’t think current “serverless” solutions are what I have in mind. Something like AWS Lambda is still too low-level to promote code that is mathematical in nature. Like, if any part of the programming process involves using a separate tool to deploy or manage the serverless code then that is a significant departure from authoring pure mathematical expressions. There needs to be something like a “Nix or Dhall for serverless code”.
Conclusion
You might criticize my prediction by claiming that it’s not falsifiable. It seems like I could explain away any new development in programming as falling into the runtime building or mathematical expression category.
That’s why I’d like to stress a key pillar of the prediction: runtimes and mathematical expressions will become more sharply divided over time. This is the actual substance of the prediction and we can infer a few corrolaries from that prediction.
Currently, many mainstream programming paradigms and engineering organizations conflate the two responsibilities, so you end up with people authoring software projects that mix operational logic (runtime concerns) and “business logic” (mathematical intent).
What I predict will happen is that the field of engineering will begin to generate a sharp demand for people with experience in programming language theory or programming language engineering. These people will be responsible for building special-purpose languages and runtimes that abstract away as many operational concerns as possible to support pure mathematical domain-specific languages for their respective enterprise. These languages will in turn be used by a separate group of people whose aim is to translate human intent into mathematical expressions.
One consequence of this prediction is that you’ll begin to see a Cambrian explosion of programming languages in the near future. Naturally as language engineers push more operational concerns into the runtime they will need to more closely tailor the runtime to specific purposes or organizational needs rather than trying to author a general-purpose language. In other words, there will be a marked fragmentation of language runtimes (and type-checkers) as each new language adapts to their respective niche.
Despite runtime fragmentation, you will see the opposite trend in userland code: programs authored in these disparate languages will begin to more closely resemble one another as they become more mathematical in nature. In a sense, mathematical expressions will become the portable “lingua franca” of userland code, especially as non-mathematical concerns get pushed into each respective language’s runtime.
That is a prediction that is much easier to falsify.
Also, this interpretation is not that far from the truth as the Haskell standard only specifies a non-strict evaluation strategy. GHC is lazy, but Haskell the language standard does not require implementations to be lazy. For more details, see: Lazy vs. non-strict↩︎
Okay, this is a bit of an over-simplification because Dhall has support for Optional values and you can model errors using unions in Dhall, but they are not commonly used in this way and idiomatically most Dhall code in the wild uses the type-checker to catch errors. Dhall is a total functional programming language and the language goes to significant lengths to discourage runtime errors, like forbidding comparing Text values for equality. Also, the language is technically dependently typed and supports testing arbitrary code at type-checking time to catch errors statically.↩︎
I’m investigating a way to speed up persistent as well as make it more powerful, and one of the potential solutions involves persisting some global state across module boundaries.
I decided to investigate whether the “Global IORef Trick” would work for this.
Unfortunately, it doesn’t.
On reflection, it seems obvious: the interpreter for Template Haskell is a GHCi-like process that is loaded for each module.
Loading an interpreter for each module is part of why Template Haskell imposes a compile-time penalty - in my measurements, it’s something like ~100ms.
Not huge, but noticeable on large projects.
(I still generally find that DeriveGeneric and the related Generic code to be slower, but it’s a complex issue).
Anyway, let’s review the trick and obseve the behavior.
Global IORef Trick
This trick allows you to have an IORef (or MVar) that serves as a global reference.
You almost certainly do not need to do this, but it can be a convenient way to hide state and make your program deeply mysterious.
You must write the {-# NOINLINE globalRef #-} pragma.
Let’s say we give globalRef a more general type:
globalRef::IORef[a]
This means that we woudl be allowed to write and read whatever we want from this reference.
That’s bad!
We could do something like writeIORef globalRef [1,2,3], and then readIORef globalRef :: IO [String].
Boom, your program explodes.
Unless you want a dynamically typed reference for some reason - and even then, you’d better use Dynamic.
If you omit the NOINLINE pragma, then you’ll just get a fresh reference each time you use it.
GHC will see that any reference to globalRef can be inlined to unsafePerformIO (newIORef []), and it’ll happily perform that optimization.
But that means you won’t be sharing state through the reference.
This is a bad idea, don’t use it.
I hesitate to even explain it.
Testing the Trick
But, well, sometimes you try things out to see if they work.
In this case, they don’t, so it’s useful to document that.
We’re going to write a function trackString that remembers the strings that are passed previously, and defines a value that returns those.
{-# language QuasiQuotes #-}{-# language TemplateHaskell #-}moduleLibwhereimportData.IORefimportSystem.IO.UnsafeimportLanguage.Haskell.THimportLanguage.Haskell.TH.QuoteimportLanguage.Haskell.TH.SyntaxglobalRef::IORef[String]globalRef=unsafePerformIO$newIORef[]{-# NOINLINE globalRef #-}trackStrings::String->Q[Dec]trackStringsinput=dostrs<-runIO$readIORefglobalRef_<-runIO$atomicModifyIORefglobalRef(\i->(input:i,()))ty<-[t| [String] |]pure[SigD(mkNameinput)ty,ValD(VarP(mkNameinput))(NormalB(ListE$map(LitE.stringL)$strs))[]]
This works in a single module just fine.
{-# language TemplateHaskell #-}moduleTestwhereimportLibtrackStrings"what"trackStrings"good"trackStrings"nothx"test::IO()test=doprintwhatprintgoodprintnothx
If we evaluate test, we get the following output:
[]
["what"]
["good","what"]
This is exactly what we want.
Unfortunately, this is only module-local state.
Given this Main module, we get some disappointing output:
{-# language TemplateHaskell #-}moduleMainwhereimportLibimportTesttrackStrings"hello"trackStrings"world"trackStrings"goodbye"main::IO()main=dotestprinthelloprintworldprintgoodbye
Fortunately, we don’t even need to do anything awful like this.
The Q monad offers two methods, getQ and putQ that allow module-local state.
-- | Get state from the Q monad. Note that the state is -- local to the Haskell module in which the Template -- Haskell expression is executed.getQ::Typeablea=>Q(Maybea)-- | Replace the state in the Q monad. Note that the -- state is local to the Haskell module in which the -- Template Haskell expression is executed.putQ::Typeablea=>a->Q()
These use a Typeable dictionary, so you can store many kinds of state - one for each type!
This is a neat way to avoid the “polymorphic reference” problem I described above.
How to actually solve the problem?
If y’all dare me enough I might write a follow-up where I investigate using a compact region to persist state across modules, but I’m terrified of the potential complexity at play there.
I imagine it’d work fine for a single threaded compile, but there’d probably be contention on the file with parallel builds.
Hey, maybe I just need to spin up a redis server to manage the file locks…
Perhaps I can install nix at compile-time and call out to a nix-shell that installs Redis and runs the server.
Recently I've been complaining about unforced translation
errors.
([1]
[2])
Here's one I saw today:
The translation was given as:
“honk honk, your Uber has arrived”
“Oleg, what the fuck”
Now, the Russian text clearly says “beep-beep” (“бип-бип”), not “honk
honk”. I could understand translating this as "honk honk" if "beep
beep" were not a standard car sound in English. But English-speaking
cars do say “beep beep”, so why change the original?
(Also, a much smaller point: I have no objection to translating “Что
за херня” as “what the fuck”. But why translate “Что за херня, Олег?”
as “Oleg, what the fuck” instead of “What the fuck, Oleg”?)
[ Addendum 20210420: Katara suggested that perhaps the original translator
was simply unaware that Anglophone cars also “beep beep”. ]
Functional programming and blockchains are a match made in heaven! The immutable and reproducible nature of distributed ledgers is mirrored in the semantic foundation of functional programming. Moreover, the concurrent and distributed operation calls for a programming model that carefully controls shared mutable state and side effects. Finally, the high financial stakes often associated with blockchains suggest the need for high assurance software and formal methods.
Nevertheless, most existing blockchains favour an object-oriented, imperative approach in both their implementation as well as in the contract programming layer that provides user-defined custom functionality on top of the basic ledger. On the one hand, this might appear surprising, given that it is widely understood that this style of programming is particularly risky in concurrent and distributed systems. On the other hand, blockchains are still in their infancy and little research has been conducted into associated programming language technology.
In this talk, I explain the connection between blockchains and functional programming as well as highlight several areas where functional programming, type systems, and formal methods have the potential to advance the state of the art. Overall, I argue that blockchains are not just a well-suited application area for functional programming techniques, but that they also provide fertile ground for future research. I illustrate this with evidence from the research-driven development of the Cardano blockchain and its contract programming platform, Plutus. Cardano and Plutus are implemented in Haskell and Rust, and the development process includes semi-formal specifications together with the use of Agda, Coq, and Isabelle to formalise key components.
Our previous posts on computational pipelines, such as those introducing Funflow and Porcupine, show that Arrows are very useful for data science workflows.
They allow the construction of effectful and composable pipelines whose structure is known at compile time, which is not possible when using Monads.
However, Arrows may seem awkward to work with at first. For instance,
it’s not obvious how to use lenses to access record fields in Arrows.
My goal in this post is to show how lenses and other optics can be used in Arrow-based workflows.
Doing so is greatly simplified thanks to Profunctor
optics and some utilities that I helped add to the latest version of the lens library.
Optics on functions
We’re used to think of lenses in terms of getters and setters, but I’m
more interested today in the functions over and traverseOf.
-- We will use this prefix for the remaining of the post.-- VL stands for Van Laarhoven lenses.importqualified Control.Lens as VL-- Transform a pure function.over::VL.Lensstab->(a->b)->(s->t)-- Transform an effectful function.traverseOf::VL.Traversalstab->(a->mb)->(s->mt)
We would like to use similar functions on Arrow-based workflows,
something like
overArrow::VL.Lensstab->Taskab->Taskst
However, the type of lenses:
typeLensstab=forallf.Functorf=>(a->fb)->s->ft
doesn’t make it very obvious how to define overArrow.
On the other hand, Arrows come equipped with functions first and second:
which very much feel like specialised versions of overArrow for the
lenses
_1::VL.Lens(b,d)(c,d)bc_2::VL.Lens(d,b)(d,c)bc
so maybe there is a common framework that can take both of these into account?
The answer is yes, and the solution is lenses — but lenses of a different type.
Profunctor optics
There is an alternative and equivalent formulation of optics, called Profunctor optics, that works very well with Arrows.
Optics in the Profunctor framework have the following shape:
typeOpticpstab=pab->pst
with more precise optics such as Lens being obtained by imposing constraints to p coming from the different Profunctor classes.
In other words, an Optic is precisely a higher-order function acting on some profunctor.
Because every Arrow is also a Profunctor1, the shape
of an Optic is precisely what is needed to act on Arrows! Moreover, like the optics of the lens library, profunctor optics can be composed like regular functions, with (.).
The lens library now includes a module containing functions that convert between standard and profunctor optics, which makes using them very convenient.
In the following sections, we will go through the use and the intuition of the most common optics: Lens, Prism and Traversal.
But first, let’s import the compatibility module for profunctor optics:
-- PL for Profunctor Lensesimport Control.Lens.Profunctor as PL
Lenses
Standard lenses are all about products — view, for example, is used to deconstruct records:
view_fst::(a,b)->a
Therefore, it makes sense for Profunctor lenses to also talk about products.
Indeed, that is exactly what happens, through the Strong type class:
That is, a Strong profunctor is equipped with lenses to reach
inside products.
One can always convert a record into nested pairs and act on them using Strong — the Lens just makes this much more convenient.
But how do we build a Lens?
Besides writing them manually, we can also use all Lenses from lens:
PL.fromLens::VL.Lensstab->Lensstab
which means we can still use all the lenses we know and love.
For example, one can apply a task to a tuple of arbitrary size:
PL.fromLens_1::Taskab->Task(a,x,y)(b,x,y)
Summarizing, a Strong profunctor is one we can apply lenses to.
Since every Arrow is also a Strong profunctor, one can use Lenses with them.
Prisms
Standard prisms are all about sums — preview, for example, is used to deconstruct sum-types:
view_Left::Eitherab->Maybea
Therefore, it makes sense for Profunctor prisms to also talk about sums.
Indeed, that is exactly what happens, through the Choice type class:
That is, a Choice profunctor is equipped with prisms to discriminate sums.
One can always convert a sum into nested Eithers and act on them using Choice — the Prism just makes this much more convenient.
But how do we build a Prism?
We can also use any prisms from lens with a simple conversion:
PL.fromPrism::VL.Prismstab->Prismstab
For example, one can execute a task conditionally, depending on the existence of the input:
Summarizing, a Choice profunctor is one we can apply prisms to.
Since every ArrowChoice can be a Choice profunctor, one can uses prisms with them.
Traversals
Standard traversals are all about Traversable structures — mapMOf, for example, is used to execute effectful functions:
mapMOftraversereadFile::[FilePath]->IO[String]
Therefore, it makes sense for Profunctor traversals to also talk about these traversable structures.
Indeed, that is exactly what happens, through the Traversing type class:
With profunctor optics, a Traversal is defined as follows:
typeTraversalstab=forallp.Traversingp=>pab->pst
There is no associated Arrow class that corresponds to this class, but many Arrows, such as Kleisli, satisfy it.
We can rewrite the type of this functions as:
traverse' ::Traversablef=>Traversal(fa)(fb)ab
That is, a Traversing profunctor can be lifted through Traversable
functors.
But how do we build a Traversal?
We can also use any Traversal from lens with a simple conversion:
PL.fromTraversal::VL.Traversalstab->Traversalstab
For example, one can have a task and apply it to a list of inputs:
PL.fromTraversaltraverse::Actionab->Action[a][b]
Conclusion
Using Arrows does not stop us from taking advantage of the Haskell ecosystem.
In particular, optics interact very naturally with Arrows, both in their classical and profunctor formulations.
For the moment, the ecosystem is still lacking a standard library for Profunctor optics, but this is not a show stopper — the lens library itself has most of the tools we need.
So the next time you are trying out Funflow or Porcupine, don’t shy away from using lens!
The fact that these hierarchies are separated is due to historical reasons.
A couple of days ago
I discussed the epithet “soup-guzzling pie-muncher”,
which in the original Medieval Italian was brodaiuolo manicator di
torte. I had compained that where most translations rendered the
delightful word brodaiuolo as something like “soup-guzzler” or
“broth-swiller”, Richard Aldington used the much less vivid “glutton”.
A form of the word brodaiuolo appears in one other place in the Decameron, in
the sixth story on the first day,
also told by Emilia, who as you remember has nothing good to say about
the clergy:
… lo 'nquisitore sentendo trafiggere la lor brodaiuola ipocrisia
tutto si turbò…
J. M. Rigg (1903), who had elsewhere translated brodaiuolo as
“broth-guzzling”, this time went with “gluttony”:
…the inquisitor, feeling that their gluttony and hypocrisy had
received a home-thrust…
G. H. McWilliam (1972) does at least imply the broth:
…the inquisitor himself, on hearing their guzzling hypocrisy exposed…
John Payne (1886):
the latter, feeling the hit at the broth-swilling hypocrisy of
himself and his brethren…
Cormac Ó Cuilleanáin's revision of Payne (2004):
…the inquisitor himself, feeling that the broth-swilling hypocrisy
of himself and his brethren had been punctured…
And what about Aldington (1930), who dropped the ball the other time and
rendered brodaiuolo merely as “glutton”? Here he says:
… he felt it was a stab at their thick-soup hypocrisy…
This is the fifth edition of our GHC activities report, which is intended to provide regular updates on the work on GHC and related projects that we are doing at Well-Typed. This edition covers roughly the months of February and and March 2021.
A bit of background: One aspect of our work at Well-Typed is to support GHC and the Haskell core infrastructure. Several companies, including IOHK and Facebook, are providing us with funding to do this work. We are also working with Hasura on better debugging tools. We are very grateful on behalf of the whole Haskell community for the support these companies provide.
If you are interested in also contributing funding to ensure we can continue or even scale up this kind of work, please get in touch.
Of course, GHC is a large community effort, and Well-Typed’s contributions are just a small part of this. This report does not aim to give an exhaustive picture of all GHC work that is ongoing, and there are many fantastic features currently being worked on that are omitted here simply because none of us are currently involved in them in any way. Furthermore, the aspects we do mention are still the work of many people. In many cases, we have just been helping with the last few steps of integration. We are immensely grateful to everyone contributing to GHC. Please keep doing so (or start)!
Release management
Ben Gamari finalized releases of GHC 8.10.4 and 9.0.1, and started work on 9.0.2. Additionally, he worked to finish and merge the outstanding patches pending for GHC 9.2, culminating in an alpha release candidate (9.2.1-alpha1).
Ben also prepared a blog post updating the community on the state of GHC on Apple M1 hardware.
Frontend
Doug Wilson proposed adding interprocess semaphore support to allow multiple concurrent compiler processes to make better use of available parallelism (!5176).
Matthew Pickering performed some simulations and experiments of the proposed changes to increase the parallelism in --make mode (#14095) and the effect of the Doug’s -jsem flag (!5176). The results show both changes have potential to improve compile times when many cores are available.
Matt opened a GHC proposal to modify the import syntax to distinguish between modules which are used at compile time and runtime. This will mean that enabling TemplateHaskell in your project will cause less recompilation when using ghci and haskell-language-server.
Ben fixed a long-standing bug rendering the GHCi linker unable to locate dynamic libraries when not explicitly added to LD_LIBRARY_PATH (#19350)
Profiling and Debugging
Matt landed the last of the ghc-debug patches into GHC, so it will be ready for use when GHC 9.2 is released.
Matt has finished off a patch which Ben and David Eichmann started which fixes races in the eventlog implementation. This makes it possible to reliably restart the eventlog and opens up the possibility of remote monitoring tooling which connects to the eventlog over a socket. Matt has made a start on implementing libraries which allow monitoring in this way.
Matt enhanced the restart support for the eventlog by ensuring certain initialisation events are re-posted every time the eventlog is restarted (!5186).
Matt added new events to the eventlog which track the current number of allocated blocks and some statistics about memory fragmentation (!5126).
To aid in recent compiler-performance investigations, Ben has been working on merging corediff, a utility for performing structural comparisons on Core, into his ghc-dump tool.
RTS
Matt investigated some discrepancies between OS reported memory usage and live bytes used by a program. This resulted in a new RTS flag (-Fd) in !5036 and a fix which reduces fragmentation due to nursery blocks (!5175). This work is described in more detail in a recent blog post.
Compiler Performance
Adam Gundry has been investigating the extremely poor performance of type families that do significant computation (#8095), building on previous work by Ben and Simon Peyton Jones on “coercion zapping”. He has an experimental patch that can significantly improve performance in some cases (!5286), although more work is needed.
Andreas Klebinger refactored the way the simplifier creates uniques (!4804). This reduces compiler allocations by between 0.1% to 0.5% when compiling with -O.
Andreas also stomped out some sources of thunks in the simplifier (!4808). The impact varies by program but allocations for GHC were reduced by ~1% in common cases and by up to 4% for particular tests.
Andreas also finished a similar patch for the code generator (!5236). This mostly benefits users frequently compiling with -O0 where we save around 0.2% of allocations.
Andreas and Ben also applied the one-shot trick (#18202) to a few more monads inside of GHC. In aggregate this should reduce GHC’s allocations by another half percent in the common case.
Matt started working on compile time performance and fixed some long standing leaks in the demand analyser. He used ghc-debug and -hi profiling in order to quickly find where the leaks were coming from.
Matt and Ben started work on a public Grafana dashboard to display the long-term trends in compiler benchmarks in an easy to understand way, taking advantage of the performance measurement infrastructure developed over the last 12 months.
Ben is investigating a significant regression in compiler performance of the aeson library (#19478) in GHC 9.0.
Ben reworked the derivation logic for Enum instances, significantly reducing the quantity of code required for such instances.
Runtime performance
Andreas has fixed a few more obstacles which will allow GHC to turn on -fstrict-dicts by default (!2575) when compiling with -O2.
Andreas investigated a runtime regression with GHC 9.0.1 (#19474) which ultimately was caused by vector regressing under certain circumstances. This was promptly fixed by Ben upstream.
After a review by Simon Peyton Jones, Andreas is evaluating a possible simplification in the implementation of his tag inference analysis.
Ben investigated and fixed a runtime regression in bytestring caused by GHC 9.0’s less-aggressive simplification of unsafeCoerce (#19539).
Compiler correctness
Andreas investigated and fixed (!4926) an issue where the behaviour of unsafeDupablePerformIO had changed in GHC 9.0 as a result of more aggressive optimisation of runRW#. In particular when a user wrote code like:
unsafePerformIO $dolet x = f x writeIORef ref xreturn x
Before the fix it was possible for x to be evaluated before the write to the IORef occurred.
Ben fixed a bug resulting in the linker failing to load libraries with long paths on Windows (#19541)
Ben fixed a rather serious garbage-collector bug affecting code unloading in GHC 8.10.4 and 9.0.1 (#19417).
Compiler functionality and language extensions
Adam assisted with landing the NoFieldSelectors extension (!4743) and the record dot syntax extensions (!4532). Both of these features will be in GHC 9.2, although record dot syntax will be properly supported only for selection, not for record update. Adam is working on a GHC proposal to address outstanding design questions regarding record update.
Ben finished and merged the BoxedRep implementation started by Andrew Martin, allowing true levity polymorphism in GHC 9.2 (#17526)
Ben has been working with a contributor to finish stack snapshotting functionality, which will allow backtraces on all platforms supported by GHC.
A few weeks ago I noticed that I no longer could use
haskell-hoogle-lookup-from-website in Emacs. After a bit of experimentation I
found that the reason was that I couldn't use xdg-open in a Nix shell.
Yesterday I finally got around to look into further.
It's caused by direnv overwriting XDG_DATA_DIRS rather than appending to it.
Of course someone already reported a bug already.
A few months ago I was pondering what it might be like to be Donald
Trump. Pretty fucking terrible, I imagine. What's it like, I
wondered, to wake up every morning and know that every person in your
life is only interested in what they can get from you, that your kids
are eagerly waiting for you to die and get out of their way, and that
there is nobody in the world who loves you? How do you get out of bed
and face that bitter world? I don't know if I could do it. It
doesn't get him off the hook for his terrible behavior, of course, but
I do feel real pity for the man.
It got me to thinking about another pitiable rich guy, Ebeneezer
Scrooge. Scrooge in the end is redeemed when he is brought face to
face with the fact that his situation is similar to Trump's. Who
cares that Scrooge has died? Certainly not his former business
associates, who discuss whether they will attend his funeral:
“It's likely to be a very cheap funeral,” said the same speaker;
“for, upon my life, I don't know of anybody to go to it. Suppose we
make up a party, and volunteer.”
“I don't mind going if a lunch is
provided," observed the gentleman with the excresence on his nose.
Later, the Spirit shows Scrooge the people who are selling the
curtains stolen from his bed and the shirt stolen from his corpse, and
Scrooge begs:
“If there is any person in the town who feels emotion caused by this
man's death," said Scrooge, quite agonized, “show that person to me,
Spirit, I beseech you!”
The Spirit complies, by finding a couple who had owed Scrooge money,
and who will now, because he has died, have time to pay.
I can easily replace Scrooge with Trump in any of these scenes, right
up to the end of chapter 4. But Scrooge in the end is redeemed. He
did once love a woman, although she left him. Scrooge did have
friends, long ago. He did have a sister who loved him, and though she
is gone her son Fred still wants to welcome him back into the family.
Did Donald Trump ever have any of those things?
This blog post is a summary of the meeting minutes for the Haskell Foundation board meeting that took place on April 8, 2021. This is the first time I'm writing these up, and potentially the only time I'm putting them on this blog. So this post is going to be a bit weird; we'll start with some questions.
Questions
Why are you writing these meeting minutes? As you'll see below, one of the decisions at the meeting was selection of additional officers for the board. I was selected as Secretary, which seems to put meeting minutes into my camp.
OK, but why are you publishing them on your personal blog? As you'll also see below, the new Haskell Foundation website is nearing completion, and in the future I hope these posts go there. But I wanted to kick things off with what I hope is close to the methodology going forward.
Isn't a blog post for each meeting excessive? Yes. As I mentioned in my transparency blog post, transparency includes a balance between too much and too little information. I intend to leverage announcement blog posts for things that deserve an announcement.
Where are the actual meeting minutes? They're on Google Drive. I have a bit more information on the Google Drive setup below.
With those out of the way, let's get into the noteworthy information from this meeting.
Opening up Slack
A lot of the work in the board so far has been in the "ways of working" direction. Basically: how does the foundation operate, what are the responsibilities of the board, what do officers do, how elections work, etc. Included in that, and in my opinion of significant interest to the community, is how we communicate. All of this information, and the decision making process around it, can be followed in the hf/meta repo on gitlab.haskell.org. I'm not going to try and summarize everything in those documents. Instead, the announcement here is around merge request !12. Specifically:
HF is standardizing on Slack as its avenue of text chatting.
We're going to start opening this up to everyone in the community interested in participating in discussions.
On a personal note, I'm very excited about this. I think a central place for ecosystem discussions is vital.
There's some hesitation/concern about moderation, off-topic discussions, and other points, which we'll need to work out over time.
I'd encourage anyone interested in joining in the conversation and staying up to date with topics to request an invite. We're already starting separate topic-specific channels to iterate on various technical topics.
Note that Slack is not a replacement for Discourse or other existing platforms. We'll still use Discourse for more official discussions and announcements. (This blog post is an example, the official discussion for it lives on Discourse.) Like many things, we'll likely be figuring out the details over time.
Board officers
There are a total of six board officers, who have all now been selected:
Chair: Richard Eisenberg
Vice Chair: Tom Ellis
Treasurer: Ryan Trinkle
Vice Treasurer: José Pedro Magalhães
Secretary: Michael Snoyman
Vice Secretary: Théophile Hécate Choutri
New Haskell Foundation website
There is a new Haskell Foundation website in the works, which should be ready to go live in the next week. It is fully open source and viewable now:
I'm hoping that, in the future, I'll be putting posts like this one on that site instead!
What's an announcement? Where do minutes go? Where's transparency?!?
I'm going to try and reserve these kinds of announcement posts to topics that I think will have widespread interest. I may make mistakes in that judgement, I apologize in advance. My goal is that anyone who wants to stay up to speed on large decisions coming from the board will be able to without using up a lot of their bandwidth.
That said, every board meeting (and, for that matter, most or all working group meetings) take and keep meeting notes. We've been sharing these on Discourse, but in the future may simply publish them in the hf/minutes repo.
Until now, we've been creating a new Discourse thread and posting the meeting minutes for each meeting. I proposed reducing how often we do that, and instead leave the minutes in Google Drive for those interested. My hope is that a combination of "information is all available in Drive" with "important things get an announcement" will cover most people. But if people would like to see a new Discourse thread for each meeting, I'd like to hear about it. Please comment on this (and other topics) in the Discourse thread linked.
Get involved!
We've been in planning and discussion mode for the past few months on the board. The various working groups are beginning to get some clarity around how they want to function, and are ready for more people to get involved. There's technical work to do, documentation, outreach, and much more. If you're excited to be a part of the efforts of the Haskell Foundation to improve the overall state of Haskell, now's a great time to get in and influence the direction. I strongly encourage everyone to check out the Slack, ask questions, interact on Discourse, and overall dive in!
In the last blog post we covered
extensible records and how we infer their types in Giml.
In this blog post we'll take a closer look at another interesting feature
and the dual of extensible records - polymorphic variants.
There are several approaches for typing polymorphic variants and they have different tradeoffs
as well. Giml's system is very similar to OCaml's polymorphic variants
system and also shares its limitations.
If you are looking for different approaches you might want to look at
this/this or
this.
I'm also not sure polymorphic variants is the best we can do to have extensible sum types,
as having recursive polymorphic variants is quite problematic. But there are still plenty
of usecases where they could be used so it's worth including them in a language.
[Word of warning, this is a bit more complicated than previous posts and it might be best to
look at this section as a cookbook. If you see something weird or not explained well,
it might be a testement to that I'm a little fuzzy on the details myself]
Polymorphic Variants
Polymorphic variants can be seen as the dual of extensible records.
Records are product types, they let us hold multiple values together in one structure at the same time,
and by giving a label for each value we can refer to each part.
Variants are sum types, they let us define multiple values as alternatives to one another,
and we label each alternative as well.
There are two operations we can do with variants, one is creating a tagged value,
for example #Ok 1, the other is pattern matching on variants. For example:
withDefault def x =
case x of
| #Ok v -> v
| #Err _ -> def
end
The types for polymorphic variants are a bit more complicated.
There are actually three different kinds of polymorphic variants:
closed, lower-bounded and upper-bounded.
In a closed variant we list the different labels and the type of their payload,
for example: [ Err : String, Ok : Int ] is a type that have two possible alternative
, #Err <some-string> and #Ok <some-integer>.
This type is something that can be generated by applying a polymorphic variant
to a function that take polymorphic variants as input.
An upper-bounded polymorphic variant is open, it describes the maximum set of variants that could be
represented by this type, and is represented syntactically with a little < right after
the opening bracket (for example: [< Err : String, Ok : Int ]).
Why does this exist? This type can be used when we want to use a variant in a pattern matching
expression, and we have a set of variants that we can handle.
The pattern matching example above is one like that. In that pattern matching expression
we could match any subset of [ Err : a, Ok : b ]. So we use an upper-bounded polymorphic
variant to express that in types.
Lower-bounded polymorphic variant is open as well, it describes the minimum set of variants that could be
represented by this type, and is represented will a little > right after the opening bracket
(for example: [> Err : String, Ok : Int ]).
It can unify with any superset of the specified alternatives.
Why does this exist? There are a couple of cases:
One, this type is used when we want to pass a variant to a function that does
pattern matching on that variant, and has the ability to match on unspecified amount of
variants, for example with a wildcard (_) or variable pattern. For example:
withDefault def x =
case x of
| #Ok 0 -> def
| #Ok v -> v
| _ -> def
end
In this case, even if we passed #Foo 182 this pattern matching should work,
it will just match the wildcard pattern and return def.
But note that expressions cannot have multiple types (at least not in Giml),
therefore we must specify in our case that the #Ok must have a specific type (Int)
and we cannot pass #Ok "hello" instead. In the pattern matching above we could match
on [ Ok : Int ] or any other type, and we write this like this: [> Ok : Int ].
Another case is the type of variant "literals" themselves! This could be considered
the dual of record selection: when we specify a tagged value (say, #Ok 1),
what we really want for it is to fit anywhere that expect at least [ Ok : Int ].
Implementation
Type definition
So we add 3 new constructors to represent the three variant types:
TypeVariant [(Label, Type)] for closed variant
TypePolyVariantLB [(Constr, Type)] TypeVar for lower-bounded polymorphic variant
TypePolyVariantUB TypeVar [(Constr, Type)] for upper-bounded polymorphic variant
One thing that wasn't visible before when we talked about the syntactic
representation of variants is the row type variable.
Here, just like with records, we use the row type variable to encoded hidden knowledge about our types.
For lower-bounded polymorphic variants (from now on, LB), we use the row type variable to represent
other variants (similar to how with records the row type variable was used to represent other fields).
For upper-bound polymorphic variants (from now on, UB) the row type variable has a different role.
We use the row type variable to keep track of which UB we want to merge together (those with the same row type variable),
how to convert a UB to LB (constrain the UB row type variable with the merge of the UB and LB fields + the LB row type variable),
and when to treat two UBs as normal variants (those with a different row type variable).
We'll explore each scenario soon.
Elaboration and constraint generation
Creating variants
When we create a variant, we:
Elaborate the type of the payload
Generate a type variable
Annotate the type of the AST node as TypePolyVariantLB [(<label>, <payload-type>)] <typevar>
We will see how this unify with other variants.
Pattern matching
For matching an expression with one or more patterns of polymorphic variants,
we elaborate the expression and constrain the type we got with a newly generated
type variable which is going to be our hidden row type variable and representative
of the type of the expression (and let's call it tv for now).
For each variant pattern matching we run into, we add the constraint:
Equality (TypeVar tv) (TypePolyVariantUB tv [(<label>, <payload-type>)])
By making all variant patterns equal to tv, we will eventually need
to unify all the types that arise from the patterns, and by keeping the
hidden row type variable tv the same for all of them, we annotate that
these TypePolyVariantUBs should unify by merging the unique fields
on each side and unifying the shared fields.
Another special case we have is a regular type variable pattern (capture pattern) and wildcard.
In our system, a capture pattern should capture any type, including any variant.
This case produces the opposite constraint:
Equality (TypeVar tv) (TypePolyVariantLB [] tv2)
Note that this special type, TypePolyVariantLB [] tv2, unifies with none variant types
as if it a type variable, but with variant types it behave like other TypePolyVariantLB.
Constraint solving
We have 3 new types, TypeVariant, TypePolyVariantLB and TypePolyVariantUB.
And in total we have 7 unique new cases to handle:
Equality TypeVariant TypeVariant
Equality TypeVariant TypePolyVariantLB
Equality TypeVariant TypePolyVariantUB
Equality TypePolyVariantLB TypePolyVariantLB
Equality TypePolyVariantLB TypePolyVariantUB
Equality TypePolyVariantUB TypePolyVariantUB (where the row type variables match)
Equality TypePolyVariantUB TypePolyVariantUB (where the row type variables don't match)
The cases are symmetrical, so we'll look at one side.
Equality TypeVariant TypeVariant
For two TypeVariant, like with two TypeRec, we unify all the fields.
If there are any that are only on one side, we fail.
Equality TypeVariant TypePolyVariantLB
This case is similar to the Equality TypeRec TypeRecExt case,
all Fields in the LB side must appear and unify in the TypeVariant side,
and the fields that are unique to the TypeVariant should unify with the
row type variable of LB.
We don't want the LB have any extra fields that are not found in the regular variant side!
Equality TypeVariant TypePolyVariantUB
When an upper-bounded polymorphic variant meets a regular variant,
we treat the UB as a regular variant as well. This is something that can makes seemingly well typed programs
to be rejected by the typechecker
(in a manner that is similar to not having let-polymorphism)
but we do not represent the constraint of a subset/subtype in Giml, only equality, and the two types just aren't equal.
Equality TypePolyVariantLB TypePolyVariantLB
Two lower-bounded polymorphic variant should unify all the shared fields,
and their row type variants should unify with a new LB that contains the non-shared fields (and a new row type variable).
Equality TypePolyVariantLB TypePolyVariantUB
LB means any superset of the mentioned fields, and UB means any subset of the mentioned fields.
Meaning, we don't want to have a field in the LB said that doesn't exist in the UB side.
This is basically the same case as Equality TypeVariant TypePolyVariantLB.
Equality TypePolyVariantUB TypePolyVariantUB (where the row type variables match)
When the two row type variables match, this means that the two types represent alternative pattern matches,
so we make sure their fields unify and merge them: we generate a constraint that unifies
the merged UB (with the same row type variable) with the row type variable.
This is so we can establish the most up-to-date type of the row type variable in the substitution.
Equality (TypePolyVariantUB tv <merged-fields>) (TypeVar tv)
This way we add to the list of alternative variants, add keep track of the row type variable
is the substitution and subsequent constraints.
Equality TypePolyVariantUB TypePolyVariantUB (where the row type variables don't match)
We treat these two as unrelated variants that represent the same subest of fields,
so the fields should match just like regular variants (Equality TypeVariant TypeVariant).
Instantiation
Instantiation is straightforward, just like all other cases: we instantiate the row type variables here as well.
Substitution
For substitution we need to handle a few cases:
For UB, if the row type variable maps to another type variable, we just switch the type variable. Otherwise we merge the variant fields we have with the type.
For LB with no variant fields (TypePolyVariantLB [] tv) we return the value mapped in the substitution (remember that this case is also used for wildcards and captures in pattern matching and not just variants).
For LB with variant fields, we try to merge the variant fields with the type.
Merging variants happens the same as with records, but we don't have to worry about duplicate variants as
the constraint solving algorithm makes sure they have the same type.
Note that this is where a UB can flip and become LB (if it's merged with an LB).
Example
As always, let's finish of with an example and infer the types of the following program:
withDefault def x =
case x of
| #Some v -> v
| #Nil _ -> def
end
one = withDefault 0 (#Some 1)
-- Constraints:
[ Equality t12 (t15 -> t16)
, Equality top0_i11 (Int -> t12)
, Equality top1 t16
, Equality (t14 -> [> Some : t14 | (t13)]) (Int -> t15)
, InstanceOf top0_i11 (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
]
1. Equality t12 (t15 -> t16)
-- => t12 := t15 -> t16
-- Constraints:
[ Equality top0_i11 (Int -> t15 -> t16)
, Equality top1 t16
, Equality (t14 -> [> Some : t14 | (t13)]) (Int -> t15)
, InstanceOf top0_i11 (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := t15 -> t16
]
2. Equality top0_i11 (Int -> t15 -> t16)
-- => top0_i11 := Int -> t15 -> t16
-- Constraints:
[ Equality top1 t16
, Equality (t14 -> [> Some : t14 | (t13)]) (Int -> t15)
, InstanceOf (Int -> t15 -> t16) (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := t15 -> t16
, top0_i11 := Int -> t15 -> t16
]
3. Equality top1 t16
-- => top1 := t16
-- Constraints:
[ Equality (t14 -> [> Some : t14 | (t13)]) (Int -> t15)
, InstanceOf (Int -> t15 -> t16) (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := t15 -> t16
, top0_i11 := Int -> t15 -> t16
, top1 := t16
]
4. Equality (t14 -> [> Some : t14 | (t13)]) (Int -> t15)
-- => [ Equality t14 Int, Equality [> Some : t14 | (t13)] t15 ]
-- Constraints:
[ Equality t14 Int
, Equality [> Some : t14 | (t13)] t15
, Equality (t14 -> [> Some : t14 | (t13)]) (Int -> t15)
, InstanceOf (Int -> t15 -> t16) (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
unchanged
5. Equality t14 Int
-- => t14 := Int
-- Constraints:
[ Equality [> Some : Int | (t13)] t15
, Equality (Int -> [> Some : Int | (t13)]) (Int -> t15)
, InstanceOf (Int -> t15 -> t16) (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := t15 -> t16
, top0_i11 := Int -> t15 -> t16
, top1 := t16
, t14 := Int
]
6. Equality [> Some : Int | (t13)] t15
-- => t15 := [> Some : Int | (t13)]
-- Constraints:
[ Equality (Int -> [> Some : Int | (t13)]) (Int -> [> Some : Int | (t13)])
, InstanceOf (Int -> [> Some : Int | (t13)] -> t16) (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := [> Some : Int | (t13)] -> t16
, top0_i11 := Int -> [> Some : Int | (t13)] -> t16
, top1 := t16
, t14 := Int
, t15 := [> Some : Int | (t13)]
]
7. Equality (Int -> [> Some : Int | (t13)]) (Int -> [> Some : Int | (t13)])
-- => [ Equality Int Int, Equality [> Some : Int | (t13)] [> Some : Int | (t13)] ]
-- Constraints:
[ Equality Int Int
, Equality [> Some : Int | (t13)] [> Some : Int | (t13)]
, InstanceOf (Int -> [> Some : Int | (t13)] -> t16) (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
]
-- Substitution:
unchanged
8. The two sides are equals so we skip them
9. The two sides are equals so we skip them
10. InstanceOf (Int -> [> Some : Int | (t13)] -> t16) (t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8)
-- => [ Equality (Int -> [> Some : Int | (t13)] -> t16) (t17 -> [< Nil : t18 | Some : t17 | (t19)] -> t17) ] -- (instantiation)
-- Constraints:
[ Equality (Int -> [> Some : Int | (t13)] -> t16) (t17 -> [< Nil : t18 | Some : t17 | (t19)] -> t17)
]
-- Substitution:
unchanged
11. Equality (Int -> [> Some : Int | (t13)] -> t16) (t17 -> [< Nil : t18 | Some : t17 | (t19)] -> t17)
-- => [ Equality Int t17, Equality [> Some : Int | (t13)] [< Nil : t18 | Some : t17 | (t19)], Equality t16 t17 ]
-- Constraints:
[ Equality Int t17
, Equality [> Some : Int | (t13)] [< Nil : t18 | Some : t17 | (t19)]
, Equality t16 t17
]
-- Substitution:
unchanged
11. Equality Int t17
-- => t17 := Int
-- Constraints:
[ Equality [> Some : Int | (t13)] [< Nil : t18 | Some : Int | (t19)]
, Equality t16 Int
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := [> Some : Int | (t13)] -> t16
, top0_i11 := Int -> [> Some : Int | (t13)] -> t16
, top1 := t16
, t14 := Int
, t15 := [> Some : Int | (t13)]
, t17 := Int
]
12. Equality [> Some : Int | (t13)] [< Nil : t18 | Some : Int | (t19)]
-- => Equality [> Some : Int | (t13)] [ Nil : t18 | Some : Int ] -- LB ~ UB are solved like LB and normal variant
-- Constraints:
[ Equality [> Some : Int | (t13)] [ Nil : t18 | Some : Int ]
, Equality t16 Int
]
-- Substitution:
unchanged
13. Equality [> Some : Int | (t13)] [ Nil : t18 | Some : Int ]
-- => [ Equality Int Int, Equality [ Nil : t18 ] t13 ]
-- Constraints:
[ Equality Int Int
, Equality [ Nil : t18 ] t13
, Equality t16 Int
]
-- Substitution:
unchanged
14. Two sides are equal, skip
15. Equality [ Nil : t18 ] t13
-- => t13 := [ Nil : t18 ]
-- Constraints:
[ Equality t16 Int
]
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := [ Nil : t18 | Some : Int ] -> t16 -- * substitution of LB with normal variant
, top0_i11 := Int -> [ Nil : t18 | Some : Int ] -> t16 -- * here too
, top1 := t16
, t14 := Int
, t15 := [ Nil : t18 | Some : Int ] -- * and here
, t17 := Int
]
16. Equality t16 Int
-- => t16 := Int
-- Constraints: []
-- Substitution:
[ t4 := [< Nil : t10 | Some : t8 | (t6)]
, t5 := t8
, t7 := t8
, t6 := [< Nil : t10 | Some : t8 | (t6)]
, targ3 := t8
, t9 := t10
, tfun2 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, top0 := t8 -> [< Nil : t10 | Some : t8 | (t6)] -> t8
, t12 := [ Nil : t18 | Some : Int ] -> Int -- * substitution of LB with normal variant
, top0_i11 := Int -> [ Nil : t18 | Some : Int ] -> Int -- * here too
, top1 := Int
, t14 := Int
, t15 := [ Nil : t18 | Some : Int ] -- * and here
, t17 := Int
, t16 := Int
]
... and done.
Result
Let's substitute the type variables in our program and hide the row type variables:
+---- t8 -> [< Nil : t10 | Some : t8 ] -> t8
|
| +---- t8
| |
| | +----- [< Nil : t10 | Some : t8 ]
| | |
withDefault def x =
+------------ [< Nil : t10 | Some : t8 ]
|
case x of +---- t8
|
| #Some v -> v
| #Nil _ -> def
|
|
+---- t8
end
+-- Int
| +--- Int -> [ Nil : t18 | Some : Int ] -> Int
| |
| | +---- Int
| | |
| | | +---- Int -> [ Nil : t18 | Some : Int ]
| | | |
| | | | +--- Int
| | | | |
one = withDefault 0 (#Some 1)
-------
|
+--- [ Nil : t18 | Some : Int ]
Phew, looks correct!
Alternative approach: simulation in userland
Another cool thing I'd like to note is that polymorphic variants could potentially be simulated in userland
if you have extensible records in the language (and this is what purescript-variant) does for example).
One approach I tweeted about that also helped me
reach the polymorphic variant implementation for Giml is to:
Make variants functions - functions that take a record of labels to function that take the variant's payload
Make patterns are these records from labels to functions on the payload
Make pattern matching just applying variants with patterns
Though in this approach we can't add a "default handler" label.
Summary
Polymorphic variants are a bit complicated and implementing them is a bit tricky, and they also have
several limitations. But I believe that for some use cases
they can really make a difference and it's worth having them in the language.
If you're interesting to see a video of me live coding polymorphic variants in Giml, check out Part 17 of my live streaming Giml development series.
The ten storytellers in The Decameron aren't all well-drawn or easy
to tell apart. In the introduction of my favorite edition, the
editor, Cormac Ó Cuilleanáin, says:
Early in the book we are given hints that we are going to get to
know these ten frame characters…. Among the Decameron
storytellers, for instance, Pampinea emerges as being bossy, while
Dioneo has a filthy mind. But little further character development
takes place.
I agree, mostly. I can see Dioneo more clearly than Ó Cuilleanáin
suggests. Dioneo reminds me of Roberto Benigni's Roman filthy-minded
Roman taxi driver in Night on Earth. I
also get a picture of Bocaccio's character Filostrato, who is a whiny
emo poet boy who complains that he woman he was simping for got tired
of him and dumped him for someone else:
To be humble and obedient to her and to follow all her whims as
closely as I could, was all of no avail to me, and I was soon
abandoned for another. Thus I go from bad to worse, and believe I
shall until I die.… The person who gave me the nickname of
Filostrato [ “victim of love” ] knew what she was doing.
When it's Filostrato's turn to choose the theme for the day's stories,
he makes the others tell stories of ill-starred love with unhappy
endings. They comply, but are relieved when it is over. (Dioneo, who
is excused from the required themes, tells instead a farcical story of
a woman who hides her secret lover in a chest after he unwittingly
drinks powerful sedative.)
Ah, but Emilia. None of the characters in the Decameron is
impressed with the manners or morals of priests. But Emilia
positively despises
them. Her story on the third day
is a good example. The protagonist, Tedaldo, is meeting his long-lost
mistress Ermellina; she broke off the affair with him seven years ago
on the advice of a friar who advised that she ought to remain faithful
to her husband. Tedaldo is disguised as a friar himself, and argues
that she should resume the affair. He begins by observing that modern
friars can not always be trusted:
Time was when the friars were most holy and worthy men, but those
who today take the name and claim the reputation of friars have
nothing of the friar but the costume. No, not even that,…
Modern friars, narrates Emilia, "strut about like peacocks" showing
off their fine clothes. She goes on from there, complaining about
friars' vanity, and greed, and lust, and hypocrisy, getting more and
more worked up until you can imagine her frothing at the mouth. This
goes on for about fifteen hundred words before she gets back to
Tedaldo and Ermellina, just at the same time that I get around to what
I actually meant to write about in this article: Emilia has Tedaldo
belittle the specific friar who was the original cause of his troubles,
who must without a doubt have been some soup-guzzling pie-muncher…
This was so delightful that I had to write a whole blog post just to
show it to you. I look forward to calling other people soup-guzzling
pie-munchers in the coming months.
A manicator is a gobbler; it's akin to “munch”, “manger”, and
“mandible”, to modern Italian mangia and related French manger.
A manicator di torte is literally a gobbler of pies.
Delightful! I love Bocaccio.
While I was researching this article I ran into some
other English translations of the phrase. The translation at Brown
University's Decameron Web is by J.M. Rigg:
some broth-guzzling, pastry-gorging knave without a doubt
which I award full marks. The translation of John Payne has
must for certain have been some broth-swilling, pastry-gorger
and two revised versions of Payne, by Singleton and Ó Cuilleanáin,
translate it similarly.
But the translation of Richard Aldington only says:
who must certainly have been some fat-witted glutton.
which I find disappointing.
I often wonder why translators opt to water down their translations
like this. Why discard the vivid and specific soup and pie in favor
of the abstract "fat-witted glutton"? What could possibly be the
justification?
Translators have a tough job. A mediocre translator will capture only
the surface meaning and miss the subtle allusions, the wordplay, the
connotations. But here, Aldington hasn't even captured the surface
meaning! How hard is it to see torte and include pie in your
translation somewhere? I can't believe that his omitting it was pure
carelessness, only that Aldington thought that he was somehow
improving on the original. But how, I can't imagine.
Well, I can imagine a little. Translations can also be too
literal. Let's consider the offensive Spanish epithet pendejo.
Literally, this is a pubic hair. But to translate it in English as
"pubic hair" would be a mistake, since English doesn't use that term
in the same way. A better English translation is "asshole". This is
anatomically illogical, but linguistically correct, because the
metaphor in both languages has worn thin. When an anglophone hears
someone called an “asshole” they don't normally imagine a literal
anus, and I think similarly Spanish-speakers don't picture a literal
pubic hair for pendejo. Brodaiuolo could be similar. Would a
14th-century Florentine, hearing brodaiuolo, picture a generic
glutton, or would they imagine someone literally holding a soup bowl
up to their face? We probably don't know. But I'm inclined to think
that “soup-guzzler” is not too rich, because by this point in Emilia's
rant we can almost see the little flecks of spittle flying out of here
mouth.
I'm offended by Aldington's omission of pie-munching.
The UK holds elections on 6 May. From the gov.uk site:
Register by 11:59pm on 19 April to vote in the following elections on 6 May:
local government elections and referendums in England
Police and Crime Commissioner elections in England and Wales
Scottish Parliament elections
Senedd (Welsh Parliament) elections
Mayor of London and London Assembly elections
Register Online. It usually takes about 5 minutes. Start Now.
Registration is easy: a rare example of a well-designed web site.
You can also support your party with a donation. Mine is the Edinburgh Green Party; feel free to add yours via the comments. Current polling shows Green on track to win 10 seats in the regional lists, and Alba on track to get no seats.
In the last blog post we covered
the general structure and algorithms of Giml's type inference engine.
In this blog post we'll take a closer look at extensible records in Giml and how to infer their types.
Records
A record is a collection of values, each associated with a name (also called a label).
For example, { name = "Giml", age = 0 }.
The type of records is very similar to their structure for example the type of the record above is:
{ age : Int, name : String }.
Records are used to build compound data aggregating multiple values into one value that
contains all of the information. The different values (also called fields) inside the record
can be accessed (or selected) by their labels, like this: <record>.<label>.
Records can also be extended with additional fields:
let
lang = { name = "Giml", age = 0 }
in
{ website = "https://giml-lang.org" | lang }
The result of the expression is a new record that has all of the fields lang has and
has one extra field website as well.
Note that in Giml, we can't have multiple values with the same label. When we add a new
field with an existing label, that previous one is replaced with the new value.
There are other reasonable behaviours here such as disallowing such operation or
having a scope of labels, but this is the behaviour I chose for Giml :)
Records are first class and can be used anywhere an Int can be expected.
They can also be pattern matched:
case { name = "Giml" } of
| { name = "Giml" } ->
"Yeah I heard about it, funny name."
| { name = "Haskell" } ->
"Haskell is pretty cool!"
| other ->
concat (concat "I'm sure " other.name) " is pretty good too!"
end
So far we've seen record literals and three operations on records: field selection, extension and pattern matching.
When we approach to type inference of features we need to take into consideration a few things:
What the types of the feature look like
What to do for each operation (elaboration and constraints generation)
How to unify the types (constraint solving)
How to instantiate
How to substitute
We are going to first try a naive approach: Just represent the types of record as TypeRec [(Label, Type)]
and see how it's just not enough to make this feature useful without type annotations.
Let see how we elaborate each operation of Record literals, field selection, record extension, pattern matching:
Record literals: simple, we have an expression that looks like this: { label1 = expr1, label2 = expr2, ... }, so we elaborate each expression and annotate the node with the type TypeRec [(label1, t1), ...].
Record selection: Also simple, we have something that looks like expr.label, so we elaborate the type of the expression and constrain it as equals to TypeRec [(label, t)].
But hold up, if we try to work with this scheme we'll see that it is not good enough.
Trying to typecheck this expression { a = 1, b = 1 }.a will create this constraint:
Equality { a : Int, b : Int } { a : t }, and while they both have a field a,
one of them is missing a field! So they are definitely not equal.
So we need to go at this differently and there are several approaches,
one is to create a new type of constraint that describe the relationship of "subtyping",
another is to describe the types differently.
For Giml I chose the second approach, we add extra information to the types to encode that
"this record type may have more fields". This approach is called row polymorphism.
Row Polymorphism
Row polymorphism provides us with the ability to represent extra information in an
additional type variable (called a row type variable) in the type.
So in addition to the variant we created TypeRec [(Label, Type)], we add another one:
TypeRecExt [(Label, Type)] TypeVar. Syntactically the type will look like this:
What this extra type variable does is represent additional fields that may be part of the type
that we don't know yet.
Let's use this for record field selection,
instead of saying "this type is equals to a record with one field of type t",
we can now say "this type is equals to a record that has this field with this type, and other fields represented by this type variable".
With this scheme we can move forward. Let's try again:
Elaboration and constraint generation
Record literals
same as before
Record selection
We have something that looks like expr.label, so we:
Generate a type variable representing the return type (which is the field type), let's call this t
Elaborate the type of the expression
Generate a type variable representing the rest of the fields, let's call this ext
Constrain the type of the expression as equals to { label : t | ext }.
Now when we try to typecheck the expression { a = 1, b = 1 }.a, which generates this constraint:
Equality { a : Int, b : Int } { a : t | ext }, we can match the fields and types that we know
(a : Int with a : t), and also match the fields that we don't ({ b : Int } with ext) - more on that later.
And when we find out what the real type of ext is, we'll substitute it (we'll talk about that later as well).
Record extension
We have something that looks like { <field1> = <expr1>, ... | <expression> }, so we:
Elaborate the type of each field
Elaborate the type of the expression
Generate a type variable for the expression which we will call ext
Constrain the type of the expression to be equal to { | ext }
Annotate the node with the type { <field1> : <type1>, ... | ext }
Pattern matching
Pattern matching is similar to record literals case, but instead of matching the expression in the case
with a rigid TypeRec, we generate a type variable and match with TypeRecExt.
That way we can select less fields in the pattern than might be available in the expression type.
Constraint solving
We need to handle 3 new cases - equality between:
Two TypeRec
A TypeRec and a TypeRecExt
Two TypeRecExt
Two TypeRec
The two records should have the same labels and for each label we generate a new constraint of Equality between the types for the label on each side.
So for example { a : Int, b : String } and { a : t1, b : t2 }, we generate the two constraint:
Equality Int t1 and Equality String t2.
If one TypeRec has a field that the other do not have, we throw an error of a missing field.
A TypeRec and a TypeRecExt
Each field in the TypeRecExt should match with the matching field in TypeRec.
If there's a field in TypeRecExt that does not exist in TypeRec we throw an error.
The other side is different, all the missing field in TypeRecExt that exist in TypeRec will
be matched with the row type variable of TypeRecExt. Remember - we said that with row polymorphism
we use a type variable as a representative of fields we don't know of yet! So we treat the type variable
in TypeRecExt as if it is a TypeVar that matches the fields that exist in the TypeRec but not in the TypeRecExt.
So for example in Equality { a : Int, b : String } { a : t1 | ext }
we generate 2 new constraints: Equality Int t1 and Equality { b : String } ext.
Two TypeRecExt
This scenario is slightly trickier, of course - the specified fields in both sides should
be matched just like in previous cases, but what do we do with the missing fields from each side?
Let's check a concrete example, we'll make it simpler by not including matching fields.
Equality { a : Int | t1 } { b : Int | t2 }
What this constraint mean, is that t2 is equals to { a : Int | t1 }without{ b : Int }, and t1 is equals to { b : Int | t2 }without{ a : Int }.
Since we don't have any notion of subtracting a type from a type (some other type system do support this),
we can try and represent this differently, we could represent this subset as a new type variable,
and translate the Equality above to these two constraints:
Equality { a : Int | t' } t2
Equality { b : Int | t' } t1
Now t' represents t2 without { a : Int } and t' also represents t1 without { b : Int }.
So more generally, when we try to unify two TypeRecExt, we match the matching fields and also
create a new row type variable to represents all of the unspecified fields in both types, and match each row type variable
with the fields specified on the other side.
Let's describe this one more time in psuedo code, adding new constraints:
Instantiation occurs as usual, the row type variable in TypeRecExt should be instantiated as well.
Substitution
What do we do if the row type variable in TypeRecExt appears in the substitution?
One, if the row type variable is mapped to a different type variable, we just replace it.
Two, if it's mapped to a TypeRec, we merge it with the existing fields and return a TypeRec with all fields,
but it's important to note that some fields in the TypeRec from the substitution may be the same as ones from our TypeRecExt.
There are a few approaches one could take here, one is to keep a scope of labels,
another is to try and unify the types of the field.
In Giml we take the type of the left-most instance, discarding the right one.
So semantically this expression is legal: { a = "Hi!" | { a = 1} } and its type is { a : String }.
Three, if the row type variable is mapped to a TypeRecExt, we do the same thing as in Two, but return a TypeRecExt instad of a TypeRec with the row type variable from the mapped value as our new row type variable.
This is basically what we need to do to infer the type of extensible records.
Example
Let's see how to infer the type of a simple record field access example.
This process is a bit much, but I've included it for those who want a
live demonstration of how this works. Feel free to skip it!
giml = { name = "Giml", age = 0 }
getName record = record.name
gimlName = getName giml
Elaboration
Through elaboration, we end up with the following AST and constraints:
-- Ast:
+-- { age : Int, name : String }
| |
| |
| __________|_______________
giml = { name = "Giml", age = 0 }
-- Constraints:
[ Equality top0 {age : Int, name : String}
]
The first stage of inference is to group dependencies and order them topologically.
Since we don't have any mutual dependencies, each definition is standalone
and the order of dependencies is kept (the last definition uses the previous two).
Also note that the names of type variables do not matter to the algorithm. They are a bit different so I have better time knowing where they are introduced.
Constraint solving
Now we go over group by group and solve the constraints:
First group
-- Constraints: [Equality top0 {age : Int, name : String}]
-- Substitution: []
1. Equality top0 {age : Int, name : String}
-- => add `top0 := {age : Int, name : String}` to the substitution
-- Constraints: []
-- Substitution: [top0 := {age : Int, name : String}]
This look correct! (and yey me who did the constraint solving by hand)
Summary
I found implementing extensible records using row polymorphism to be relatively straightforward.
I hope this article makes it more approachable as well.
In the next blog post we'll discuss polymorphic variants and how to infer their types.
If the nim sum of a position is zero, then the position is lost: any move loses (and makes the nim sum nonzero). If the nim sum is nonzero, the position can be won: choose a move that reduces the nim sum to zero. (What is the computational complexity of finding such a move? Typically, how many winning moves are there? Below, we will give some positions with more than one winning move.)
Important details omitted: if the nim sum is nonzero, there always exists a move that makes it zero. If the nim sum is zero, there are no moves that keep it zero.
Call a nim position obviously lost if the following mirroring strategy is applicable. The heaps can be organized into pairs of equal-sized heaps. Whatever the first player does to one heap of a pair, the second player can do to the corresponding other heap of the pair and thereby eventually win.
All 2-heap nim positions that are lost are obviously lost. In other words, the lost 2-heap nim positions are exactly those which have 2 equal sized heaps.
Call a position non-obviously lost if it is lost but not obviously lost.
The simplest non-obviously lost position is [1,2,3].
The positions [1,3,3] [2,2,3] [2,3,3] are the three simplest winning positions that have more than one winning move. All can be reduced to [1,2,3] or to [a,a].
Below are the non-obviously lost 3-heap positions whose smallest heap has size 1. All entries are [1, 2n, 2n+1].
For casual play, let the initial position be a non-obviously lost position. No move is objectively better than another from a lost position, so this will tend to produce varied games from as early as the first move. (The first player explores different ways to try to swindle a win.) [3,4,7] mentioned above is a nice small 3-heap initial position for casual play. Some nice 4-heap (lost) initial positions that are easy to remember are [2,3,4,5] (14 coins total), [4,5,6,7] (22 coins), and [6,7,8,9] (30 coins).
We propose non-obviously lost initial positions with more heaps that satisfy the following aesthetic constraints: distinct heap sizes, minimize total number of coins, minimize largest heap, maximize smallest heap. We avoid heaps of equal size in the initial position to make the mirroring strategy somewhat less likely to be usable toward the beginning of the game. However, equal-sized heaps can easily appear after the first move.
The unique 3-heap satisfying these constraints is [1,2,3] (6 coins).
4-heap: [2,3,4,5] (14 coins).
5-heap: [3,4,6,8,9] (30 coins).
6-heap: [1,2,4,6,8,9] (30 coins).
The perfect triangle 7-heap [1,2,3,4,5,6,7] satisfies the constraints. It has size 28, smaller than the 5 or 6 heaps above.
8-heap: [2,3,4,5,6,7,8,9] (44 coins).
9-heap: [2,3,4,5,7,8,9,10,12] or [2,3,4,5,6,8,9,11,12] (60 coins).
10-heap: [1,2,3,4,5,6,8,9,10,12] (60 coins).
Best 11-heap is another perfect triangle: [1,2,3,4,5,6,7,8,9,10,11] (66 coins).
13-heap: [1,2,3,4,5,6,7,9,10,12,14,16,17], [1,2,3,4,5,6,7,8,11,12,14,16,17], [1,2,3,4,5,6,7,8,10,13,14,16,17], or [1,2,3,4,5,6,7,8,10,12,15,16,17] (106 coins)
The values of n for which a perfect triangle [1,2,3,...,n] is a lost position: 0 3 7 11 15 19 23 27 31 35 39 43 47 51 55 59 63 67 71 75 79 83 87 91 95 99 ... The values are 4k-1.
There are no non-obviously lost positions with all heaps having the same number of coins. An even number of identical heaps is obviously lost. An odd number of identical heaps is won: just take an entire heap to leave your opponent with an even number of identical heaps, a position which is lost.
The following trapezoids (heap sizes of consecutive integers, at least 2 and at most 20 coins in a heap) are non-obviously lost. It appears the number of heaps must be a multiple of 4, and the smallest heap must have an even number of coins.
Future work: organize positions by their distance from being obviously lost. (Previously, we analyzed Chomp in this way.) What other classes of positions can be quickly recognized as lost? (We noted [1, 2n, 2n+1] above.)
Future work: repeat this analysis for the "subtraction game" in which the number of coins that can be removed in one move is limited (up to k). Surprisingly, optimal play can be calculated quickly: compute nim sum mod (k+1).
This post is essentially a longer version of this Reddit post by Ashley Yakeley that provides more explanation and historical context.
Sometimes in Haskell you need to write a function that “dispatches” only on a type and not on a value of that type. Using the example from the above post, we might want to write a function that, given an input type, prints the name of that type.
Approach 1 - undefined
One naive approach would be to do this:
classTypeName a where typeName :: a ->String instanceTypeNameBoolwhere typeName _ ="Bool" instanceTypeNameIntwhere typeName _ ="Int"
However, this approach does not work well because we must provide the typeName method with a concrete value of type a. Not only is this superfluous (we don’t care which value we supply) but in some cases we might not even be able to supply such a value.
There is a perfectly valid name associated with this type, but we cannot retrieve the name without cheating because we cannot produce a (total) value of type Void. Instead, we have to use something like undefined:
>>> typeName (undefined ::Void) "Void"
The base package uses this undefined-based approach. For example, Foreign.Storable provides a sizeOf function that works just like this:
classStorable a where -- | Computes the storage requirements (in bytes) of the argument. The value -- of the argument is not used. sizeOf :: a ->Int …
… and to this day the idiomatic way to use sizeOf is to provide a fake value using undefined:
>>> sizeOf (undefined ::Bool) 4
This works because sizeOf never evaluates its argument. It’s technically safe, albeit not very appealing to depend on undefined.
Approach 2A - Proxy
The next evolution of this approach was to use the Proxy type (now part of base in the Data.Proxy module). As the documentation notes:
Historically, Proxy :: Proxy a is a safer alternative to the undefined :: a idiom.
I’m not exactly sure what the name Proxy was originally meant to convey, but I believe the intention was that a term (the Proxy constructor) stands in as a “proxy” for a type (specifically, the type argument to the Proxy type constructor).
We can amend our original example to use the Proxy type like this:
importData.Proxy (Proxy(..)) importData.Void (Void) classTypeName a where typeName ::Proxy a ->String instanceTypeNameBoolwhere typeName _ ="Bool" instanceTypeNameIntwhere typeName _ ="Int" instanceTypeNameVoidwhere typeName _ ="Void"
… and now we can safely get the name of a type without providing a specific value of that type. Instead we always provide a Proxy constructor and give it a type annotation which “stores” the type we wish to use:
A minor variation on the previous approach is to use proxy (with a lowercase “p”) in the typeclass definition:
importData.Void (Void) classTypeName a where typeName :: proxy a ->String -- ↑ instanceTypeNameBoolwhere typeName _ ="Bool" instanceTypeNameIntwhere typeName _ ="Int" instanceTypeNameVoidwhere typeName _ ="Void"
Everything else works the same, but now neither the author nor the consumer of the typeclass needs to depend on the Data.Proxy module specifically. For example, the consumer could use any other type constructor just fine:
… or (more likely) the consumer could define their own Proxy type to use instead of the one from Data.Proxy, which would also work fine:
>>>dataProxy a =Proxy >>> typeName (Proxy ::ProxyInt) "Int"
This trick helped back when Proxy was not a part of the base package. Even now that Proxy is in base you still see this trick when people author typeclass instances because it’s easier and there’s no downside.
Both of these Proxy-based solutions are definitely better than using undefined, but they are both still a bit unsatisfying because we have to supply a Proxy argument to specify the desired type. The ideal user experience should only require the type and the type alone as an input to our function.
We previously noted that we could shorten the Proxy-based solution by using TypeApplications:
>>> typeName @BoolProxy "Bool"
Well, what if we could shorten things even further and just drop the Proxy, like this:
>>> typeName @Bool
Actually, we can! This brings us to a more recent approach (the one summarized in the linked Reddit post), which is to use AllowAmbiguousTypes + TypeApplications, like this:
{-# LANGUAGE AllowAmbiguousTypes #-} importData.Void (Void) classTypeName a where typeName ::String instanceTypeNameBoolwhere typeName ="Bool" instanceTypeNameIntwhere typeName ="Int" instanceTypeNameVoidwhere typeName ="Void"
The use of TypeApplications is essential, since otherwise GHC would have no way to infer which typeclass instance we meant. Even a type annotation would not work:
>>> typeName ::String-- Clearly, this type annotation is not very helpful <interactive>:1:1:error: • Ambiguoustype variable ‘a0’ arising from a use of ‘typeName’ prevents the constraint ‘(TypeName a0)’ from being solved. Probable fix: use a type annotation to specify what ‘a0’ should be. These potential instances exist: instance [safe] TypeNameVoid-- Defined at Example.hs:14:10 instance [safe] TypeNameBool-- Defined at Example.hs:8:10 instance [safe] TypeNameInt-- Defined at Example.hs:11:10 • In the expression: typeName ::String In an equation for ‘it’: it = typeName ::String
Type applications work here because you can think of a polymorphic function as really having one extra function argument: the polymorphic type. I elaborate on this a bit in my post on Polymorphism for dummies, but the basic idea is that TypeApplications makes this extra function argument for the type explicit. This means that you can directly tell the compiler which type to use by “applying” the function to the right type instead of trying to indirectly persuade the compiler into using the the right type with a type annotation.
Conclusion
My personal preference is to use the last approach with AllowAmbiguousTypes and TypeApplications. Not only is it more concise, but it also appeals to my own coding aesthetic. Specifically, guiding compiler behavior using type-annotations feels more like logic programming to me and using explicit type abstractions and TypeApplications feels more like functional programming to me (and I tend to prefer functional programming over logic programming).
However, the Proxy-based approach requires no language extensions, so that approach might appeal to you if you prefer to use the simplest subset of the language possible.
The capability library is an alternative to the venerable mtl
(see our earlier blog posts on the subject).
It features a set of
“mtl-style” type classes, representing effects, along with deriving combinators
to define interpreters as type class instances. It relies on the -XDerivingVia
extension to discharge effects declaratively, close to the definition of the
application’s monad. Business logic can be written in a familiar, idiomatic way.
As an example, consider the following computation:
This function assumes a Reader effect "foo" of type Int, and a State effect
"bar" of type Bool. It computes whether or not "foo" is an even number and
stores the result in "bar".
Save for the tags "foo" and "bar", used to enable multiple Reader or State
effects within the same monad (an impossible thing with mtl), this is fairly
standard Haskell: Type classes are used to constrain what kind of effects the
function can perform, while decoupling the computation from any concrete
implementation. At use-site, it relies on GHC’s built-in resolution mechanism to
“inject” required dependencies. Any seasoned Haskeller should feel right at home
!
Providing instances
To actually call this admittedly silly function, we need to provide interpreters
for the "foo" and "bar" effects. Following the ReaderT
design pattern, we’ll pack everything we need into a single context record, then
interpret our effects over this context in the IO monad, using the deriving
combinators provided by the library:
dataCtx=Ctx{foo::Int,bar::IORefBool}derivingGenericnewtypeMa=M{runM::Ctx->IOa}deriving(Functor,Applicative,Monad)viaReaderTCtxIO-- Use DerivingVia to derive a HasReader instance.deriving(HasReader"foo"Int,HasSource"foo"Int)via-- Pick the field foo from the Ctx record in the ReaderT environment.Field"foo""ctx"(MonadReader(ReaderTCtxIO))-- Use DerivingVia to derive a HasState instance.deriving(HasState"bar"Bool,HasSource"bar"Bool,HasSink"bar"Bool)via-- Convert a reader of IORef to a state capability.ReaderIORef(Field"bar""ctx"(MonadReader(ReaderTCtxIO)))
Thus equipped, we can now make use of our testParity function in an actual
program:
How do we test a function such as testParity in isolation? In our contrived
example, this is quite easy: the example function could be easily converted
into a test-case. In the Real World™, though, our context Ctx could be much
bigger, providing a pool of database connections, logging handles, etc. Surely,
we don’t want to spawn a database instance to test such a simple
function!
Ad-hoc interpreters
In previous iterations of capability, the solution to this problem would have
been to create a new monad for testing purposes, leaving out the capabilities
we don’t want. While it works, it is not always the best tool for the job:
You need to define a new monad for each combination of effects you want to
test.
Test cases are no longer self-contained; their implementation is spread
across multiple places. It makes things less readable and harder to maintain.
A solution, supported by fancier effect system libraries such as
polysemy or fused-effects, is to define ad-hoc
interpreters in the executable code itself. At first glance, it might seem like
this is not possible in capability. Indeed, since interpreters are provided as
type class instances, and type classes are an inherently static mechanism, surely
there is no way of specifying those dynamically. Or is there?
As of version 0.4.0.0, the capability library features an experimental
Capability.Reflection module, addressing this very
limitation. It is inspired by, and uses, Edward Kmett’s reflection
library, and uses similar type class wrangling magic to let you define
interpreters as explicit dictionaries.
Interpreters as reified dictionaries
Making use of those new features, the example function can be rewritten as:
importqualified Control.Monad.Reader as MTLReaderexample::IO()example=doletrunTestParity::(Int,IORefBool)->IO()runTestParity(foo,bar)=flipMTLReader.runReaderTfoo$-- Interpret the effects into 'ReaderT Int IO'.---- Write the 'HasReader "foo" Int' dictionary-- in terms of mtl functions.---- Forward the 'MonadIO' capability.interpret@"foo"@'[MonadIO]ReifiedReader{_reader=MTLReader.reader,_local=MTLReader.local,_readerSource=ReifiedSource{_await=MTLReader.ask}}$-- Use 'MonadIO' to write the 'HasState "bar" Bool' dictionary.-- Forward the 'HasReader "foo" Int' capability.---- The 'MonadIO' capability is not forwarded, and hence forgotten.interpret@"bar"@'[HasReader"foo"Int]ReifiedState{_state=\f->dob<-liftIO$readIORefbarlet(a,b')=fbliftIO$writeIORefbarb'
purea,_stateSource=ReifiedSource{_await=liftIO$readIORefbar},_stateSink=ReifiedSink{_yield=liftIO . writeIORefbar}}testParityrEven<-newIORefFalserunTestParity(2,rEven)readIORefrEven>>=print
Defining a test monad is no longer required: the effects are interpreted
directly in terms of the underlying ReaderT Int IO monad. Type-class
dictionaries are passed to the interpret function as mere records of functions
and superclass dictionaries — just like GHC does under the hood as hidden
parameters when we use statically defined instances.
Omitting the extra Proxy# arguments, which are here for technical reasons, the
first two attributes, _reader and _local, correspond directly to the methods
of the HasReader t type class:
This is quite boilerplatey, though. If we’re writing a lot of test cases, we are
bound to redefine those interpreters several times. This is tedious,
error-prone, and clutters our beautiful test logic. Maybe this is could all be
factored out? Sure thing!
The (typelevel) cs :: [(* -> *) -> Constraint] argument is a list of
capabilities that we wish to retain in the underlying action.
Since we interpret the State effect with a mutable IORef reference, we
require that the underlying monad be an instance of MonadIO. Moreover, we
ask that our target monad also implement all the required capabilities by
adding the All cs m constraint to the context (All is a type family that
applies a list of capabilities to a monad to generate a single constraint;
for example, All '[MonadIO, HasSource "baz" Baz] m is equivalent to
(MonadIO m, HasSource "baz" Baz m)).
The IORef used to store our state is passed as a standard function
argument. This was not possible without ad-hoc interpreters: we
needed to add the IORef to the Ctx type. With ad-hoc
interpreters, on the other hand, we can write instances which
capture references in their closures.
The last argument is a monadic action that makes use of HasState "bar" Bool
along with the forwarded cs capabilities. It is required to be polymorphic
in the monad type, which guarantees that the action cannot use other effects.
Now that we have factored out the interpretation of the "foo" and "bar"
effects into dedicated functions, they can be neatly composed to provide just
the effects we need to run testParity:
Truth be told, in this example, the dictionaries we’ve been writing aren’t so different
from a custom type class with capabilities provided by deriving-via. While the extra power that comes with
dynamic dictionaries can be very useful, it isn’t always warranted.
There is a middle ground, however: we can provide capabilities
locally, but with deriving-via combinators using a function that we call
derive. You would typically use derive to derive high-level
capabilities from lower-level capabilities. In our case, we can replace:
runTestParity::(Int,IORefBool)->IO()runTestParityctx=flipMTLReader.runReaderTctx$derive-- Strategy@(ReaderIORef:.:Rename2:.:Pos2_:.:MonadReader)-- New capability@'[HasState"bar"Bool]-- Forwarded capability@'[MTLReader.MonadReader(Int,IORefBool)]$derive@(Rename1:.:Pos1_:.:MonadReader)@'[HasReader"foo"Int]@'[HasState"bar"Bool]testParity
thus getting rid of the interpret{Foo,Bar} helpers entirely. For instance, the
HasState "bar" Bool capability is derived from the IORef Bool in the second
position of the tuple provided by the ambient MonadReader (Int, IORef Bool)
instance. Think DerivingVia, but dynamically!
Conclusion
Wrapping things up:
At its core, the capability library is just mtl on steroids, modeling
effects with type classes.
The standard way of using capability is to define interpreters
declaratively, using the provided combinators; this programming-style does
not allow defining ad-hoc interpreters, at runtime.
The new version of capability provides a way of overcoming this limitation
with reified dictionaries.
Standard deriving strategies can be used to provide dynamic instances with
less boilerplate, using the underlying deriving mechanism.
Writing tests is just one example. Another application might be to dynamically
select the interpretation of an effect based on a configuration parameter. All
this is still experimental: the API and ergonomics are likely to change a bit
over the next few releases, but we hope this post motivates you to give it a
try.
Giml's type inference engine uses unification-based constraint solving.
It has 5 important parts:
Topologically order definitions and group those that depend on one another
Elaboration and constraint generation
Constraint solving
Instantiation
Substitution
In summary, we sort and group definitions by their dependencies, we elaborate the program with
type variables and types that we know and collect constraints on those variables,
we then solve the constraints for each group using unification and create substitutions which
are mapping from type variables to types and merge them together, and we then substitute the
type variables we generated in the elaboration stage with their mapped types from the substitution
in the program.
Instantiation occurs during the elaboration stage and the constraint solving stage, when we want
to use a polymorphic type somewhere.
Group and order definitions
Before we start actually doing type inference, we figure out the dependencies between
the various definitions. This is because we want to figure out the type of a definition
before it's use, and because we want definitions that depend on each other (mutual recursion)
to be solved together.
So for example if we have this file as input:
main = ffi("console.log", odd 17)
even x =
if int_equals x 0
then True
else if int_equals x 1
then False
else odd (sub x 1)
odd x =
if int_equals x 0
then False
else if int_equals x 1
then True
else even (sub x 1)
The output of the reordering and grouping algorithm will be:
[["even", "odd"], ["main"]]
This is because even and odd are mutually recursive, so we want to typecheck them together,
and because main depends on odd we want to typecheck it after we finished typechecking odd.
Now the real work begins. We start by going over the program's AST and annotate it with the types we know.
When we don't know what the type of something is we "invent" a new fresh type variable as a placeholder
and mark that type with a constraint according to its usage.
Types
Language.Giml.Types.Types contains a datatype that can be used to represent Giml types. For example:
TypeCon "Int" represents the type Int
TypeVar "t1" represents a polymorphic type variable
TypeApp (TypeCon "Option") (TypeCon "Int") represents the type Option Int
TypeRec [("x", TypeCon "Int")] represents a record with the type { x : Int }
Constraints
A Constraint between two types describes their relationship.
The most common constraint is Equality. Which means that the two types are
interchangeable and should unify to the same type. For example from the constraint Equality (TypeVar t1) (TypeCon "Int") we learn that t1 is the same as Int.
The other, less common constraint, is InstanceOf. Which means that the first type has
an Equality relationship with the instantiation of the second type.
an instantiation is a monomorphic instance of a polymorphic type,
for example (a -> b) -> List a -> List b is polymorphic, and can be instantiated, among other examples, to (Int -> String) -> List Int -> List String by replacing the polymorphic type variables a and b with monomorphic types such as Int and String.
We will talk about solving these constraints a bit later.
Implementation
We also use State, Reader and Except in our algorithm:
A monotonically increasing integer for generating type variables
A list of constraints we generated through the elaboration phase, which we will return as output at the end
An environment of data definitions, some are built-in and others are user defined and added as part of the elaboration stage (maybe I should've put this in the Reader part...)
The currently scoped variables in the environment and their type (which might be polymorphic!)
The type of builtin functions and expressions (such as int_equals and pure, which might also be polymorphic)
The Except part can throw a few errors such as "unbound variable" error.
Most of the action happens in elaborateExpr and I'm not going to cover every usecase, but lets look at a few examples:
Elaborating literals
Elaborating literals is simple - we already know what the types are - if we see a string literal
we annotate the AST node with the type String, if we see an integer literal we annotate it
with Int, and we don't need to generate new constraints because we know exactly what the type is.
Elaborating lambda functions
Elaborating lambda expressions is interesting, because we don't know what the type for the arguments
is going to be: it might be inferred by the usage like in this case: \n -> add 1 n or not
like in this case \x y -> x. What we do here is:
Generate a type variable for each function argument
Elaborate the body of the lambda with the extended environment containing the mapping between the function arguments and their newly generated types
Annotate the lambda AST node with the type of a function from the argument types to the type of the body, which we get from the elaborated node after (2)
And we will trust the constraint solving algorithm to unify all of the types.
Elaborating function application
To elaborate a function application we:
Elaborate the arguments
Elaborate the applied expression
Generate a fresh type variable to represent the return type
Constrain the type of the applied expression as a function from the arguments to the return type
Annotate the function application AST node with the return type (which we just generated)
Remember that the type of the applied expression may just be a generated type variable, so we
can't examine its type to see what the return type is. For example if we had this expression
\f x -> f x, as we saw before we generated type variables for f and x!
So we have to make sure the type inference engine knows what their type should be
(by generating a constraint) according to the usage and trust that it'll do its job.
Elaborating variables
For variables, we expect an earlier stage (such as elaborating lambdas, lets or definitions)
to have already elaborated their type and added them to the environment, so we look them up.
We first try in the scoped environment and if that fails we try the built-ins environment.
If that fails as well, we throw an error that the variable is unbound.
If we do find a variable in the scoped environment, there are two scenarios we need to take care of:
The variable was bound in a place that expect it to be monomorphic, such as a lambda function argument
The variable was bound in a place that expect it to be polymorphic, such as a term definition
Monomorphic types can be used as is, but polymorphic types represent an instantiation of a type.
When we find that the variable we are elaborating is polymorphic, we will generate a new type
variable for it and constrain it as an InstanceOf the type we found in the environment.
Later in the constraints solving stage we will instantiate the polymorphic type we found in
the environment and say that the type variable we generate is equals to the instantiation.
We delay this because we want the polymorphic type to be solved first before we instantiate it.
If we do not find the variable in the scoped environment, we check the builtins environment.
If we find the type there, we instantiate it immediately and annotate the AST node with this type.
Here we instantiate the type immediately because it is already solved, so we don't need to wait.
Constraints solving
The heart of the type inference engine. Here we visit each the constraints generated from
each group of substitutions separately and generate a substitution
(mapping from type variables to types) as output.
the solveConstraints function takes an accumulated substitution and a list of constraints as input
and tries to solve the first constraint on the list.
Solving a constraint might generate more constraints to solve, and may return a substitution as well.
We then apply this substitution (substitute all occurences of the keys with the values) to the
rest of the constraints and the accumulated substitution itself, merge the new substitution
with the accumulated substitution, and keep going until there are no more constraints left to solve.
(solveConstraint) uses unification to solve a constraint. Here are a few important cases that it handles:
Equality t1 t2 (where t1 == t2)
When the two types are the same (for example Int and Int, or t1 and t1),
the unification succeeds without extra work needed,
we don't need to create new constraints or create a substitution.
Equality (TypeVar tv) t2
When one of the types is a type variable, we substitute (replace) it with t2 in all places we know of
(accumulated substitution and rest of the constraints). So the rest of the constraints that
contain it will use t2 instead, and when substituting it with another type it'll be t2.
Equality (TypeApp f1 a1) (TypeApp f2 a2)
We generate two new constraints: Equality f1 a2 and Equality f2 a2, reducing a constraint
of complex types into something we already know how to handle: a constraint on simple types.
InstanceOf t1 t2
When we run into an InstanceOf constraint, we instantiate the second type t2
and produce an Equality constraint between t1 and the instantiation of t2.
Instantiating is basically "copying" the type signature, and producing the Equality
constraint with t1 makes the type inference engine find the right type for the instantiated
type variables.
We use InstanceOf instead of Equality when the type is polymorphic,
for example when we want to use id that we defined as id x = x.
If we were to use an Equality constraint instead, every use of id would constrain
it to have a particular type (so id 1 would make the type of idInt -> Int instead of only where we use it).
There are a few more cases, but this is the gist of it.
When we fail to unify we throw an error of type mismatch.
Example
Let's see an example for this (very) short program:
one = (\x -> x) 1
After the elaboration stage we generate the following AST and constraints:
The lambda expression case generated the type t1 for x, so it's type is t1 -> t1
The literal case annotated 1 as Int
The function application generated the return type t2 and generated the constraint
Equality (t1 -> t1) (Int -> t2) which is "the type of the applied expression is equals to a function from the argument to the return type I just generated"
The definition of one generated the type variable top0 for it so it can be used in other definitions polymorphically, but for this function it should be used monomorphically, so it's constrainted as equal to the return type and generated the constraint Equality top0 t2.
How do we unify them?
Let's go over them in order. We begin with an empty substitution:
- Constraints: [Equality top0 t2, Equality (t1 -> t1) (Int -> t2)]
- Substitution: []
1. Equality top0 t2
-- ==> The first is a type variable, so add `top0 := t2` to the substitution
- Constraints: [Equality (t1 -> t1) (Int -> t2)]
- Substitution: [top0 := t2]
2. Equality (t1 -> t1) (Int -> t2)
-- ==> replace with two new constraints:
-- 1. Equality t1 Int
-- 2. Equality t1 t2
- Constraints: [Equality (-> t1) (-> Int), Equality t1 t2]
- Substitution: [top0 := t2]
3. Equality (-> t1) (-> Int)
-- ==> replace again with two new constraints:
-- 1. Equality (->) (->)
-- 2. Equality t1 Int
- Constraints: [Equality (->) (->), Equality t1 Int, Equality t1 t2]
- Substitution: [top0 := t2]
4. Equality (->) (->)
-- ==> these two types are identical, so we can continue
- Constraints: [Equality t1 Int, Equality t1 t2]
- Substitution: [top0 := t2]
5. Equality t1 Int
-- ==> The first is a type variable, so add `t1 := Int` to the substitution
- Constraints: [Equality Int t2]
- Substitution: [top0 := t2, t1 := Int]
6. Equality Int t2
-- ==> The second is a type variable, so add `t2 := Int` to the substitution
- Constraints: []
- Substitution: [top0 := Int, t1 := Int, t2 := Int]
And we're done! The output is the substitution [top0 := Int, t1 := Int, t2 := Int],
we can now apply it to our elaborated program and get the fully type checked program:
+-- Int -> Int
|
---------
one = (\x -> x) 1
| | |
+-- Int | +------ Int
|
+---- Int
Two more things we talk to talk about before we wrap this up:
Instantiation
We instantiate a type by looking up all of the type variables in the type and generate
a fresh new type variable for each one
(though all instances of a type variable should be mapped to the same new type variable).
So for example in the type (a -> b) -> List a -> List b, if t1 and t2 are two
new type variables we haven't seen before, we can replace a with t1
and b with t2 everywhere in the type and get (t1 -> t2) -> List t1 -> List t2
After that, we let the Equality constraints and the constraint solving algorithm
find the right monomorphic type for t1 and t2.
Substitution
When we apply a substitution to a constraint or another substitution, we replace all occurences
of a type variable tv with t if t := tv is in the substitution. But,
we also need to check that tv does not contain t inside of it! (this is called an occurs check)
if tvdoes contain t, it means that the we ran into an infinite type and we don't support those here,
so we will throw an error.
A simple way to generate such case is this:
x y = x
The constraint generated from this definition is: Equality top0 (t1 -> top0), which will fail
the occurs check.
If you try to desugar this definition into a lambda expression, you'll quickly discover that
you've been had: anytime you'll go to replace x with \y -> x you'll just add another level
of lambda.
Summary
This is the general overview of Giml's type inference engine.
If you are interested in learning more about it, check out the source code.
If this helped you, there's something you want me to clarify,
you think there's something I missed, there's something you want to know more about,
or there's something I got totally wrong,
do contact me on Twitter or via Email.
As we all know, blockchain is a disruptive technology that has helped us find new efficiencies in our lives. For example, prior to Bitcoin, creation of new currency required wasteful processes like digging gold out of the ground or printing paper currency. Bitcoin replaces that with cheap electricity. The world is better off for the reduced resource usage. Efficiency!
More industries are beginning to adopt various blockchain solutions to solve common problems. These are creating synergies in enterprises and forming new markets. But so far, blockchain has been used almost exclusively to solve industrial and financial systems issues. Today, I'm happy to introduce a new blockchain technology aimed at solving problems faced by everyday users.
The problem
It's a perennial issue, with a huge economic impact (detailed below). We estimate that in addition to the massive economic impact, the emotional and psychological damage may even be the larger factor.
I'm talking, of course, about mismatched and lost socks. While "the dryer ate my socks" is a common term, we have in fact identified multiple related but separable subproblems:
The standard missing-sock
Similar, yet slightly different, versions of the same socks, leading to excess effort in identifying a match. This problem breaks down further into:
Socks which have been worn a different number of times and thereby stretched differently
Different manufacturing lines leading to microvariations within construction
Despite all claims to the contrary, there are in fact left vs right variations of socks, either due to manufacture or usage. Identifying a correct pairing becomes an NP-complete problem
This is in fact just the tip of the iceberg. We fully believe there may be at least 13 further classifications of sock-related inefficiencies. Based just upon our current knowledge, we have calculated the following impacts:
7 billion sock-wearing individuals worldwide (some populations are, of course, part of the rabid anti-socker cabal).
Individuals tend to change their socks approximately every 1.2 days.
Individuals tend to launder socks approximately every 8 days.
On average, 3.4 individuals run laundry concurrently.
This leads to an average socks pair/laundry of ~23, or ~46 socks per laundry cycle.
As this is an NP-hard problem to solve, with a comparison taking on average 2.3 seconds.
This leads to approximately 75,000 minutes spent sorting socks per laundry cycle. While there may be some ad-hoc optimizations, we expect no greater than a 10x speedup from such optimizations, and believe each laundry cycle requires at least 7,500 minutes, just on sock sorting.
Combining this with our estimates for laundry cycles per individual, we estimate that the global economy receives an impact of 1.3 billion person years wasted per year just on sock sorting!
As you can see, sock sorting accounts for easily the greatest modern crisis known to humankind.
But further than this is the emotional and psychological harm caused by these situations. We estimate 1 in 4 divorces are caused by mismatched socks. 3 in 10 children will wear a mismatched pair of socks to school at least once per month, with a 27% chance of cyberbullying as a result.
Enter the sockchain
The sockchain is the natural, obvious, and modern solution to this problem. We leverage our novel Proof-of-Work/Proof-of-Stake hybrid technology to create a distributed application that uses Near Field Communication (NFC) to tag each pair of socks. Dryers throughout the world can easily and cheaply be outfitted with NFC detectors. Miners will be able to detect the presence of such socks in distributed drying systems, and provide a searchable, distributed ledger of sock-pair-locations (SPLs).
Our fee incentive structure will guarantee payment of at least 2 socks-pairs/block, leading to economies of scale. The fee structure will appropriately respond to the evolving and dynamic socketplace.
We envision further development as well. Already, some of our partners are embedded fungus detection systems within our NFCs for athlete's foot elimination, dealing with yet another worldwide pandemic.
Cost
It's right to question the costs of such endeavors. As mentioned above, the average labor cost per year currently is 1.3 billion person years. Even taking a modest estimate of $150/hour of labor, we end up with an average cost per year of nearly two quadrillion dollars, which is almost the marketcap of Bitcoin itself.
We estimate 80 billion active pairs of socks worldwide. At an NFC cost of $20 per pair, we end up with a modest hardware cost of $1.6 trillion, an easily affordable expense given the current human impact. Outfitting the dryers will cost an additional $50 trillion. And we estimate that electricity costs for our novel Proof of Work system to be no greater than $100 trillion. Including training, marketing, and our very slim administrative expenses, the project will cost approximately $300 trillion. Compared to the current cost of $2 quadrillion, we're looking at an 85% cost savings.
It would be inhumane not to undertake such a project.
The GHC developers are very happy to announce the availability of the first alpha release in the 9.2.1 series. Binary distributions, source distributions, and documentation are available from downloads.haskell.org.
GHC 9.2 will bring a number of exciting features including:
Many changes in the area of records, including the new RecordDotSyntax and NoFieldSelectors language extensions, as well as Support for DuplicateRecordFields with PatternSynonyms.
Introduction of the new GHC2021 language extension set, giving users convenient access to a larger set of language extensions which have been long considered stable.
Merge of ghc-exactprint into the GHC tree, providing infrastructure for source-to-source program rewriting out-of-the-box.
Introduction of a BoxedRepRuntimeRep, allowing for polymorphism over levity of boxed objects (#17526)
Implementation of the UnliftedDataTypes extension, allowing users to define types which do not admit lazy evaluation
The new -hi profiling mechanism which provides significantly improved insight into thunk leaks.
Support for the ghc-debug out-of-process heap inspection library
Support for profiling of pinned objects with the cost-centre profiler (#7275)
Introduction of Haddock documentation support in TemplateHaskell (#5467)
In addition, the final 9.2.1 release will bring a new native code generator for ARM, providing fast, first-class for Haskell on Apple ARM hardware, although this functionality is not yet present in this alpha.
As always, do give this a try and open a ticket if you see anything amiss.
Two weeks back, I wrote a blog post with a proposal for unification of vector-like types in bytestring, text, and vector. This continued with a discussion on Discourse, and has been part of some brainstorming sessions at the Haskell Foundation tech track to lock down some of the concrete details. (Meeting minutes have been posted to Discourse regularly.) I've discussed with a few other interested parties, and invited feedback from people who have been working on related projects. (And I even received some such feedback!)
At this point, I wanted to summarize what's come up, and propose some concrete next steps. Also, since this is one of the first such proposals we're trying to get to happen through the Haskell Foundation, I'll take a step back with some meta comments about the process itself.
Refined goals
I threw quite a bit into the original proposal. The brainstorming since has helped make it clear what can be achieved, what will have major breaking impacts, and what will deliver the most value. Here are some outcomes:
As much as I still think a single, unified Vector type that does boxing or unboxing depending on the contained type would be a Very Good Thing, there are technical hurdles, and not everyone is bought into it. I doubt it's going to happen any time soon.
The temporary pinning of memory is likely not going to happen, since it would be too big a GC change. However, larger ByteArray#s (3kb IIRC) are already pinned, and using the copy-when-unpinned technique will not be too expensive for things under 3kb. So we have a path forward that requires no (or few) changes to GHC.
It seems like one of the biggest advantages we may get out of this proposal is to move ByteString to unpinned memory. This would be good, since it would reduce memory fragmentation. The downside is the need to copy smaller ByteStrings before performing FFI calls. But overall, as mentioned previously, we don't think that will have a major performance impact.
There are already efforts underway, and have been for a while, to rewrite text to use UTF-8. Combined with this proposal, we could be seeing some serious improvements to how textual data is consumed and worked with in Haskell, but that's a bit too far off right now.
Refined design
With that in place, a semi-coherent design is beginning to form around this proposal:
We're not ready to move types into base yet, but fortunately we already have another package that's a good candidate for shared-vector-types: primitive. It's already used by vector, and can be used by bytestring and text.
We have a PrimArray type present in primitive, but it doesn't support slicing. Let's add a PrimVector type with the minimal machinery necessary to get slicing in place.
The big change: let's rewrite ByteString (the strict variant) to be newtype ByteString = ByteString (PrimVector Word8).
We can recover backwards compatibility in most of the package, including in the .Unsafe module.
People directly using the .Internal module will likely be broken, though there may be some clever tricks to recover backwards compat there too.
We'll get the non-memory-fragmentation benefit immediately.
Not yet discussed, but putting down for future brainstorming: what should we do with ShortByteString? If we move it over to PrimVector, it will end up with a little more overhead for slicing. Do we leave it alone instead? Move it to PrimArray?
There are additional steps I could write around text and vector, but honestly: let's stop it right there. If we get a working bytestring package on top of primitive's PrimVector, I think that's a great time to take a break, analyze the performance impact, and see ecosystem impact, likely by compiling a Stackage snapshot with the tweaked bytestring and primitive packages.
Action items
Next steps: find out who's interested in doing the work and dive in! This is still proof of concept, so no real buy-in is needed. We're exploring a possibility. There's a bunch of code that needs to be rewritten in bytestring to see if this is possible.
And while brainstorming in calls has been good, I don't think it's great. I'd like to move to discussions in a chat room to make it easier for others to engage. I'll comment a bit more on this below.
Other packages
I've reached out to some authors of other packages to get their input on this proposal. I've received some, and incorporated some of that here. For example, both Alexey Kuleshevich and Alexey Khudyakov proposed using primitive as the new home for the shared data types. Others have expressed that they'd rather first see a concrete proposal. We'll see if there is further collaboration possible in the future.
Process level comments
Let me summarize, at a high level, what the process was that was involved in this proposal:
Free-form discussions on the Haskell Foundation tech track in a live meeting. People discussing different ideas, different concerns, which ultimately triggered the idea of this proposal.
Blog post to the world and Discourse thread laying out some initial ideas and looking for feedback.
Public discussion.
A few private emails reaching out to specific interested parties to get their input.
In concert with (3) and (4): further tech track live meetings to discuss details further.
This second blog post (and associated Discourse thread) to update everyone and call to action.
Hopefully: some actual coding.
Overall, I think this is a good procedure. If I could make one change, it would be towards leveraging asynchronous communication more and live meetings less. I absolutely see huge value in live meetings of people to brainstorm, as happened in (1). But I think one of the best things we can do as the Haskell Foundation is encourage more people to easily get involved in specific topics they care about.
On the bright side, the significant usage of Discourse for collaboration and reporting on meeting minutes has been a Good Thing. I think blog posts like this one and my previous one are a Good Thing for collecting thoughts coherently.
That said, I realize I'm in the driver's seat on this proposal, and have a skewed view of how the outside world sees things. If people have concerns with how this was handled, or ideas for improvement, bring them up. I think figuring out how to foster serious discussion of complex technical issues in the Haskell ecosystem is vital to its continued success.
One silver-lining to the cloud of COVID has been the development of virtual forms of participation. A SIGPLAN blog post by five early-career researchers offers their perspective on what we should do next.
We propose that SIGPLAN form a Committee on Conference Data. The committee would be made up of: one organizing-committee representative from each of the flagship SIGPLAN conferences, one early career representative, and, crucially, a professional data collection specialist hired by SIGPLAN. The group would identify and collect key data that is pertinent to conference organization, especially with respect to physical versus virtual conference formats. The committee would make data-driven recommendations to SIGPLAN organizers based on the collected data and guided by core tenets such as community building, inclusivity, research dissemination, and climate responsibility. We realize that this is not a small request, but we are confident that it is both necessary and achievable. If the committee were to form by May 1, 2021, it would be able to start collecting data at PLDI 2021 and continue through the next two years, providing enormous clarity for SIGPLAN organizers at a time when so much is unclear.
![\displaystyle \begin{array}{@{}l} \mathit{primes} = \mathit{sieve}\;[2.\,.] \;\textbf{where} \\ \qquad \mathit{sieve}\;(p:\mathit{xs}) = p : \mathit{sieve}\;[ x \mid x \leftarrow \mathit{xs}, x \mathbin{\mathrm{mod}} p \ne 0] \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%7D+%5Cmathit%7Bprimes%7D+%3D+%5Cmathit%7Bsieve%7D%5C%3B%5B2.%5C%2C.%5D+%5C%3B%5Ctextbf%7Bwhere%7D+%5C%5C+%5Cqquad+%5Cmathit%7Bsieve%7D%5C%3B%28p%3A%5Cmathit%7Bxs%7D%29+%3D+p+%3A+%5Cmathit%7Bsieve%7D%5C%3B%5B+x+%5Cmid+x+%5Cleftarrow+%5Cmathit%7Bxs%7D%2C+x+%5Cmathbin%7B%5Cmathrm%7Bmod%7D%7D+p+%5Cne+0%5D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
takes a stream of candidate primes; the head 
of this stream is a prime, and the remaining primes are obtained by removing all multiples of 
is tested for divisibility. O’Neill calls this bogus common version “trial division”, and argues that the Genuine Sieve of Eratosthenes should eliminate every multiple of 
of the natural numbers are touched when eliminating multiples of 2, less than 
of the remaining candidates for multiples of 3, and so on. As an additional optimization, it suffices to eliminate multiples of 
, since by that point all composite numbers with a smaller nontrivial factor will already have been eliminated.![\displaystyle \begin{array}{@{}l@{\;}l} \multicolumn{2}{@{}l}{\mathit{primes}, \mathit{composites} :: [\mathit{Integer}]} \\ \mathit{primes} & = 2 : ([3.\,.] \mathbin{\backslash\!\backslash} \mathit{composites}) \\ \mathit{composites} & = \mathit{mergeAll}\;[ \mathit{multiples}\;p \mid p \leftarrow \mathit{primes} ] \\ \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%40%7B%5C%3B%7Dl%7D+%5Cmulticolumn%7B2%7D%7B%40%7B%7Dl%7D%7B%5Cmathit%7Bprimes%7D%2C+%5Cmathit%7Bcomposites%7D+%3A%3A+%5B%5Cmathit%7BInteger%7D%5D%7D+%5C%5C+%5Cmathit%7Bprimes%7D+%26+%3D+2+%3A+%28%5B3.%5C%2C.%5D+%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D+%5Cmathit%7Bcomposites%7D%29+%5C%5C+%5Cmathit%7Bcomposites%7D+%26+%3D+%5Cmathit%7BmergeAll%7D%5C%3B%5B+%5Cmathit%7Bmultiples%7D%5C%3Bp+%5Cmid+p+%5Cleftarrow+%5Cmathit%7Bprimes%7D+%5D+%5C%5C+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
, which unions a set of sets to a set; but 
is the obvious implementation of list difference of sorted streams (hence, representing set difference):![\displaystyle \begin{array}{@{}l@{\;}l@{\;}l} \multicolumn{3}{@{}l}{(\mathbin{\backslash\!\backslash}) :: \mathit{Ord}\;a \Rightarrow [a] \rightarrow [a] \rightarrow [a]} \\ (x:\mathit{xs}) \mathbin{\backslash\!\backslash} (y:\mathit{ys}) & \mid x < y & = x:(\mathit{xs} \mathbin{\backslash\!\backslash} (y:\mathit{ys})) \\ & \mid x == y & = \mathit{xs} \mathbin{\backslash\!\backslash} \mathit{ys} \\ & \mid x > y & = (x:\mathit{xs}) \mathbin{\backslash\!\backslash} \mathit{ys} \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%40%7B%5C%3B%7Dl%40%7B%5C%3B%7Dl%7D+%5Cmulticolumn%7B3%7D%7B%40%7B%7Dl%7D%7B%28%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D%29+%3A%3A+%5Cmathit%7BOrd%7D%5C%3Ba+%5CRightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D%7D+%5C%5C+%28x%3A%5Cmathit%7Bxs%7D%29+%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D+%28y%3A%5Cmathit%7Bys%7D%29+%26+%5Cmid+x+%3C+y+%26+%3D+x%3A%28%5Cmathit%7Bxs%7D+%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D+%28y%3A%5Cmathit%7Bys%7D%29%29+%5C%5C+%26+%5Cmid+x+%3D%3D+y+%26+%3D+%5Cmathit%7Bxs%7D+%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D+%5Cmathit%7Bys%7D+%5C%5C+%26+%5Cmid+x+%3E+y+%26+%3D+%28x%3A%5Cmathit%7Bxs%7D%29+%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D+%5Cmathit%7Bys%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
generates the multiples of ![\displaystyle \begin{array}{@{}l} \mathit{multiples}\;p = \mathit{map}\; (p\times)\;[p.\,.] \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%7D+%5Cmathit%7Bmultiples%7D%5C%3Bp+%3D+%5Cmathit%7Bmap%7D%5C%3B+%28p%5Ctimes%29%5C%3B%5Bp.%5C%2C.%5D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
![{[ [ 4, 6.\,.], [ 9, 12.\,.], [ 25, 30.\,.], \dots ]}](https://s0.wp.com/latex.php?latex=%7B%5B+%5B+4%2C+6.%5C%2C.%5D%2C+%5B+9%2C+12.%5C%2C.%5D%2C+%5B+25%2C+30.%5C%2C.%5D%2C+%5Cdots+%5D%7D&bg=ffffff&fg=000000&s=0&c=20201002)
. You might think that you could have defined instead ![{\mathit{primes} = [2.\,.] \mathbin{\backslash\!\backslash} \mathit{composites}}](https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bprimes%7D+%3D+%5B2.%5C%2C.%5D+%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D+%5Cmathit%7Bcomposites%7D%7D&bg=ffffff&fg=000000&s=0&c=20201002)
, but this doesn’t work: this won’t compute the first prime without first computing some composites, and you can’t compute any composites without at least the first prime, so this definition is unproductive. Somewhat surprisingly, it suffices to “prime the pump” (so to speak) just with 2, and everything else flows freely from there.
, which merges two sorted duplicate-free streams into one (hence, representing set union):![\displaystyle \begin{array}{@{}l@{\;}l@{\;}l} \multicolumn{3}{@{}l}{\mathit{merge} :: \mathit{Ord}\;a \Rightarrow [a] \rightarrow [a] \rightarrow [a]} \\ \mathit{merge}\;(x:\mathit{xs})\;(y:\mathit{ys}) & \mid x < y & = x : \mathit{merge}\;\mathit{xs}\;(y:\mathit{ys}) \\ & \mid x == y & = x : \mathit{merge}\;\mathit{xs}\;\mathit{ys} \\ & \mid x > y & = y : \mathit{merge}\;(x:\mathit{xs})\;\mathit{ys} \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%40%7B%5C%3B%7Dl%40%7B%5C%3B%7Dl%7D+%5Cmulticolumn%7B3%7D%7B%40%7B%7Dl%7D%7B%5Cmathit%7Bmerge%7D+%3A%3A+%5Cmathit%7BOrd%7D%5C%3Ba+%5CRightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D%7D+%5C%5C+%5Cmathit%7Bmerge%7D%5C%3B%28x%3A%5Cmathit%7Bxs%7D%29%5C%3B%28y%3A%5Cmathit%7Bys%7D%29+%26+%5Cmid+x+%3C+y+%26+%3D+x+%3A+%5Cmathit%7Bmerge%7D%5C%3B%5Cmathit%7Bxs%7D%5C%3B%28y%3A%5Cmathit%7Bys%7D%29+%5C%5C+%26+%5Cmid+x+%3D%3D+y+%26+%3D+x+%3A+%5Cmathit%7Bmerge%7D%5C%3B%5Cmathit%7Bxs%7D%5C%3B%5Cmathit%7Bys%7D+%5C%5C+%26+%5Cmid+x+%3E+y+%26+%3D+y+%3A+%5Cmathit%7Bmerge%7D%5C%3B%28x%3A%5Cmathit%7Bxs%7D%29%5C%3B%5Cmathit%7Bys%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
, but again this is unproductive. After all, you can’t merge the infinite stream of sorted streams ![{[ [5,6.\,.], [4,5.\,.], [3,4.\,.], \dots ]}](https://s0.wp.com/latex.php?latex=%7B%5B+%5B5%2C6.%5C%2C.%5D%2C+%5B4%2C5.%5C%2C.%5D%2C+%5B3%2C4.%5C%2C.%5D%2C+%5Cdots+%5D%7D&bg=ffffff&fg=000000&s=0&c=20201002)
into a single sorted stream, because there is no least element. Instead, we have to exploit the fact that we have a sorted stream of sorted streams; then the binary merge can exploit the fact that the head of the left stream is the head of the result, without even examining the right stream. So, we define:![\displaystyle \begin{array}{@{}l} \mathit{mergeAll} :: \mathit{Ord}\;a \Rightarrow [[a]] \rightarrow [a] \\ \mathit{mergeAll}\;(\mathit{xs}:xss) = \mathit{xmerge}\;\mathit{xs}\;(\mathit{mergeAll}\;xss) \medskip \\ \mathit{xmerge} :: \mathit{Ord}\;a \Rightarrow [a] \rightarrow [a] \rightarrow [a] \\ \mathit{xmerge}\;(x:\mathit{xs})\;\mathit{ys} = x : \mathit{merge}\;\mathit{xs}\;\mathit{ys} \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%7D+%5Cmathit%7BmergeAll%7D+%3A%3A+%5Cmathit%7BOrd%7D%5C%3Ba+%5CRightarrow+%5B%5Ba%5D%5D+%5Crightarrow+%5Ba%5D+%5C%5C+%5Cmathit%7BmergeAll%7D%5C%3B%28%5Cmathit%7Bxs%7D%3Axss%29+%3D+%5Cmathit%7Bxmerge%7D%5C%3B%5Cmathit%7Bxs%7D%5C%3B%28%5Cmathit%7BmergeAll%7D%5C%3Bxss%29+%5Cmedskip+%5C%5C+%5Cmathit%7Bxmerge%7D+%3A%3A+%5Cmathit%7BOrd%7D%5C%3Ba+%5CRightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D+%5C%5C+%5Cmathit%7Bxmerge%7D%5C%3B%28x%3A%5Cmathit%7Bxs%7D%29%5C%3B%5Cmathit%7Bys%7D+%3D+x+%3A+%5Cmathit%7Bmerge%7D%5C%3B%5Cmathit%7Bxs%7D%5C%3B%5Cmathit%7Bys%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
yields the infinite sequence of prime numbers, using the genuine algorithm of Eratosthenes.
, 
,
![\displaystyle \begin{array}{@{}l@{\;}l@{\;}l} \multicolumn{3}{@{}l}{\mathit{approx} :: \mathit{Integer} \rightarrow [a] \rightarrow [a]} \\ \mathit{approx}\;(n+1)\;[\,] & = & [\,] \\ \mathit{approx}\;(n+1)\;(x:\mathit{xs}) & = & x : \mathit{approx}\;n\;\mathit{xs} \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%40%7B%5C%3B%7Dl%40%7B%5C%3B%7Dl%7D+%5Cmulticolumn%7B3%7D%7B%40%7B%7Dl%7D%7B%5Cmathit%7Bapprox%7D+%3A%3A+%5Cmathit%7BInteger%7D+%5Crightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D%7D+%5C%5C+%5Cmathit%7Bapprox%7D%5C%3B%28n%2B1%29%5C%3B%5B%5C%2C%5D+%26+%3D+%26+%5B%5C%2C%5D+%5C%5C+%5Cmathit%7Bapprox%7D%5C%3B%28n%2B1%29%5C%3B%28x%3A%5Cmathit%7Bxs%7D%29+%26+%3D+%26+x+%3A+%5Cmathit%7Bapprox%7D%5C%3Bn%5C%3B%5Cmathit%7Bxs%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
is undefined; the function 
preserves the outermost 
constructors of a list, but then chops off anything deeper and replaces it with 
(undefined). So, the lemma states that to prove two infinite lists equal, it suffices to prove equal all their finite approximations.
is the 
th prime (so 
). Bird therefore defines
![\displaystyle \begin{array}{@{}ll} \mathit{prs}\;n & = \mathit{approx}\;n\;(2 : ([3.\,.] \mathbin{\backslash\!\backslash} \mathit{crs}\;n)) \\ \mathit{crs}\;n & = \mathit{mergeAll}\;[\mathit{multiples}\;p \mid p \leftarrow \mathit{prs}\;n] \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dll%7D+%5Cmathit%7Bprs%7D%5C%3Bn+%26+%3D+%5Cmathit%7Bapprox%7D%5C%3Bn%5C%3B%282+%3A+%28%5B3.%5C%2C.%5D+%5Cmathbin%7B%5Cbackslash%5C%21%5Cbackslash%7D+%5Cmathit%7Bcrs%7D%5C%3Bn%29%29+%5C%5C+%5Cmathit%7Bcrs%7D%5C%3Bn+%26+%3D+%5Cmathit%7BmergeAll%7D%5C%3B%5B%5Cmathit%7Bmultiples%7D%5C%3Bp+%5Cmid+p+%5Cleftarrow+%5Cmathit%7Bprs%7D%5C%3Bn%5D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
to be well defined at least up to the first composite number greater than 
, because then 
, which will in turn supply 
, and so on. It is a “non-trivial result in Number Theory” (in fact, a consequence of 
; therefore it suffices that
is the 
) and 
. Completing the proof is set as Exercise 9.I of TFWH, and Answer 9.I gives a hint about using induction to show that 
. (Incidentally, there is a typo in TFWH: both the body of the chapter and the exercise and its solution have “
” instead of “
(which equals 
) could be constructed from 
(which equals 
) and 
(which equals ![{[9,12.\,.]}](https://s0.wp.com/latex.php?latex=%7B%5B9%2C12.%5C%2C.%5D%7D&bg=ffffff&fg=000000&s=0&c=20201002)
); but this means that the 6 and 8 would have to come out of nowhere. Nevertheless, the claim in Exercise 9.I is valid. What should the hint for the proof have been?![\displaystyle \mathit{crsOf}\;\mathit{qs} = \mathit{mergeAll}\;[\mathit{multiples}\;p \mid p \leftarrow \mathit{qs}]](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathit%7BcrsOf%7D%5C%3B%5Cmathit%7Bqs%7D+%3D+%5Cmathit%7BmergeAll%7D%5C%3B%5B%5Cmathit%7Bmultiples%7D%5C%3Bp+%5Cmid+p+%5Cleftarrow+%5Cmathit%7Bqs%7D%5D+&bg=ffffff&fg=000000&s=0&c=20201002)
; this allows us to consider 
in isolation from 
. In fact, I claim (following a suggestion generated by Geraint Jones and Guillaume Boisseau at our Algebra of Programming research group) that if 
where 
, then 
has a well-defined prefix consisting of all the composite numbers 
such that 
and 
with 
, in ascending order, then becomes undefined. That’s not so difficult to see from the definition: 
contains the multiples of 
; all these streams get merged; but the result gets undefined (in 
of the last stream. In particular, 
has a well-defined prefix consisting of all the composites up to 
, then becomes undefined, as required.
works along the grain of 
. I believe we want something instead like an “Approx While Lemma”, where 
is to 
is to 
:![\displaystyle \begin{array}{@{}l} \mathit{approxWhile} :: (a \rightarrow \mathit{Bool}) \rightarrow [a] \rightarrow [a] \\ \mathit{approxWhile}\;p\;(x:\mathit{xs}) \mid p\;x = x : \mathit{approxWhile}\;p\;\mathit{xs} \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dl%7D+%5Cmathit%7BapproxWhile%7D+%3A%3A+%28a+%5Crightarrow+%5Cmathit%7BBool%7D%29+%5Crightarrow+%5Ba%5D+%5Crightarrow+%5Ba%5D+%5C%5C+%5Cmathit%7BapproxWhile%7D%5C%3Bp%5C%3B%28x%3A%5Cmathit%7Bxs%7D%29+%5Cmid+p%5C%3Bx+%3D+x+%3A+%5Cmathit%7BapproxWhile%7D%5C%3Bp%5C%3B%5Cmathit%7Bxs%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
retains elements of 
and unbounded streams 
(that is, 
),
and hence for 
