Planet Haskell

November 07, 2024

Donnacha Oisín Kidney

POPL Paper—Algebraic Effects Meet Hoare Logic in Cubical Agda

Posted on November 7, 2024
Tags:

New paper: “Algebraic Effects Meet Hoare Logic in Cubical Agda”, by myself, Zhixuan Yang, and Nicolas Wu, will be published at POPL 2024.

Zhixuan has a nice summary of it here.

The preprint is available here.

by Donnacha Oisín Kidney at November 07, 2024 12:00 AM

May 22, 2024

Oskar Wickström

Statically Typed Functional Programming with Python 3.12

Lately I’ve been messing around with Python 3.12, discovering new features around typing and pattern matching. Combined with dataclasses, they provide support for a style of programming that I’ve employed in Kotlin and Typescript at work. That style in turn is based on what I’d do in OCaml or Haskell, like modelling data with algebraic data types. However, the more advanced concepts from Haskell — and OCaml too, I guess — don’t transfer that well to mainstream languages.

May 22, 2024 10:00 PM

Well-Typed.Com

The Haskell Unfolder Episode 26: variable-arity functions

Today, 2024-05-22, at 1830 UTC (11:30 am PDT, 2:30 pm EDT, 7:30 pm BST, 20:30 CEST, …) we are streaming the 26th episode of the Haskell Unfolder live on YouTube.

The Haskell Unfolder Episode 26: variable-arity functions

In this episode, we will take look at how one can use Haskell’s class system to encode functions that take a variable number of arguments, and also discuss some examples where such functions can be useful.

About the Haskell Unfolder

The Haskell Unfolder is a YouTube series about all things Haskell hosted by Edsko de Vries and Andres Löh, with episodes appearing approximately every two weeks. All episodes are live-streamed, and we try to respond to audience questions. All episodes are also available as recordings afterwards.

We have a GitHub repository with code samples from the episodes.

And we have a public Google calendar (also available as ICal) listing the planned schedule.

There’s now also a web shop where you can buy t-shirts and mugs (and potentially in the future other items) with the Haskell Unfolder logo.

by andres, edsko at May 22, 2024 12:00 AM

May 21, 2024

Philip Wadler

INESC-ID Distinguished Lecture, Lisbon

I'm looking forward to speaking in Lisbon. 

On June 4, Professor Philip Wadler will give an INESC-ID Distinguished Lecture organized in the scope of the BIG ERA Chair Project, titled “(Programming Languages) in Agda = Programming (Languages in Agda)”.

Registration: here (free but mandatory)
Date: June 4, 2024
Time: 15h00-16h15
Where: Anfiteatro Abreu Faro – Complexo Interdisciplinar, Instituto Superior Técnico (Alameda)
 

Abstract: The most profound connection between logic and computation is a pun. The doctrine of Propositions as Types asserts that propositions correspond to types, proofs to programs, and simplification of proofs to evaluation of programs. Proof by induction is just programming by recursion.  Finding a proof becomes as fun as hacking a program. Dependently-typed programming languages, such as Agda, exploit this pun. This talk introduces *Programming Language Foundations in Agda*, a textbook that doubles as an executable Agda script—and also explains the role Agda plays in IOG’s Cardano cryptocurrency.

by Philip Wadler (noreply@blogger.com) at May 21, 2024 10:43 AM

I am a Highly Ranked Scholar


I am delighted to have made this list. Lesley Lamport and John Reynolds also appear on it, but at positions lower than mine, and Tony Hoare and Robin Milner don't appear at all---so perhaps their methodology needs work.


Congratulations on being named an inaugural Highly Ranked Scholar by ScholarGPS


Dear Dr. Wadler,

ScholarGPS celebrates Highly Ranked Scholars™ for their exceptional performance in various Fields, Disciplines, and Specialties. Your prolific publication record, the high impact of your work, and the outstanding quality of your scholarly contributions have placed you in the top 0.05% of all scholars worldwide.


Listed below is a summary of the areas (and your ranking in those areas) in which you have been awarded Highly Ranked Scholar status based on your accomplishments over the totality of your career (lifetime) and over the prior five years:

Highly Ranked Scholar - Lifetime
#9,339Overall (All Fields)
#1,299Engineering and Computer Science
#265Computer Science
#2Programming language


Please consider sharing your recognition as an inaugural ScholarGPS Highly Ranked Scholar with your employer, colleagues, and friends.

ScholarGPS also includes quantitative rankings for research institutions, universities, and academic programs across all areas of scholarly endeavor. ScholarGPS provides rankings overall (in all Fields), in 14 broad Fields (such as Medicine, Engineering, or Humanities), in 177 Disciplines (such as Surgery, Computer Science, or History), and in over 350,000 Specialties (such as Cancer, Artificial Intelligence, or Ethics).

We are pleased to currently offer you free access to each of the following:
  • All Scholar Profiles and Scholar Rankings based on either lifetime achievements or on accomplishments over the past five years.
  • Lists of Highly Ranked Scholars categorized by Field, Discipline, and Specialty. Highly Ranked Scholars™ are the most productive (number of publications) authors whose works are of profound impact (citations) and of utmost quality (h-index). Their scholarly contributions position them within the top 0.05% of all scholars worldwide.
  • Institutional Rankings and program rankings that are based on the achievements of institutional scholars over their lifetime and over the past five years.
  • Profiles for FieldsDisciplines and Specialities, including a summary of activities, associated ranked institutions, Highly Ranked Scholars, and highly cited publications.
  • Top Scholars by Institution or Top Scholars by Expertise or Top Scholars by Country by ranking order across All Fields, Field, Discipline, or Specialty. Top scholars are those authors whose scholarly contributions position them within the top 0.5% of all scholars worldwide.
  • Institutional profiles including a summary of activities, associated program rankings, Highly Ranked Scholars, Top Scholars, and highly cited papers as well as publication and citation histories. View both Highly Ranked Scholars and Top Scholars within each institution categorized by Field, Discipline, and Specialty.
  • A user-friendly publication search capability and corresponding citation index.
Sincerely,
The ScholarGPS Team

by Philip Wadler (noreply@blogger.com) at May 21, 2024 10:35 AM

Tweag I/O

LLM-based work summaries with work-dAIgest

Time flies, and while it occasionally drops dead from the sky on Fridays or Monday mornings, that definitely was not the case during an internal two-day hackathon Tweag’s GenAI group held in February. As one of our projects, we wanted to develop a tool that would tell us where exactly time flew to in a given period of time, and so was born work-dAIgest: a Python tool that uses your standard workplace tools (Google Calendar, GitHub, Jira, Confluence, Slack, …) and a large language model (LLM) to summarize what you’ve accomplished.

In this blog post, we’ll present a proof-of-concept of work-dAIgest, show you what’s under the hood and finally touch quickly on a couple lessons we learned during this short, but fun project.

Overview

The following diagram illustrates the general idea:

work-dAIgest overview

We want to get work-related data from multiple sources (your agenda, GitHub, Jira, Confluence, Slack, emails, …) and ask a LLM to summarize the information about your work these data contain.

As experienced data engineers and scientists, we knew that in almost any data or AI project, most of the time is spent on data retrieval, cleaning and preprocessing. With that in mind, and only two days to build a first version, we restricted ourselves to a narrower scope of two data sources that nevertheless cover different work responsibilities:

  • GitHub issues / PRs / commits: to reflect daily work of a developer
  • Google Calendar meetings / events: to reflect daily work of a manager

We decided to rely on the GitHub’s Search API to extract commits and pull requests / issues. For calendar data, we chose to extract .ics files manually, as it is easier and quicker to handle than the Google Calendar API.

Data preprocessing

Calendar data

Calendar data often comes in ICS (Internet Calendar and Sharing) format, and we resorted to the ics Python library to parse it. That way, for each event, we created a string with the following lines:

<event name>
duration: <event duration>
description: <event description>
attendees: <attendee name 1> - <attendee name 2> - ...
-------------------

We then placed the concatenation of these strings in the LLM prompt.

GitHub data

Information about commits, issues and pull requests could be extracted via GitHub’s Search API. We had to be careful to pass the right date time formatting (YYYY-MM-DDTHH:MM:SS / ISO8601) from the user input to the search string. Once we had the search results, we could output them as the JSON of a list of objects, each with the following fields:

  • date (issue / PR comment or commit date),
  • text (issue / PR text or commit message),
  • repository (repository name),
  • action (either created, updated, or closed).

This JSON was then included as-is in the LLM prompt.

Large language model

The LLM was at the heart of this project, and with it came two important questions: What LLM to use and how to design the prompt.

LLM choice

While the choice of LLMs out there was becoming overwhelming, we could not just use any LLM. Instead, we had multiple requirements:

  • it should be sufficiently powerful,
  • it must be appropriate to use with confidential data (calendar events, data from private GitHub repositories), lest it leaks into later versions of the LLM,
  • it should be readily available to use,
  • it should be cheap for one-off use.

These constraints led us to decide to initially implement support for Llama 2 70B and Jurassic-2 as deployed on Amazon Bedrock. Claude-3 models became available in Bedrock a couple weeks after the first work-dAIgest prototype was finished, and we added support for it straight away. Bedrock assured us that our data would stay private. It was easy to use via simple AWS Python API calls and its pay-on-demand model per token was very cost-efficient.

Prompt engineering

The first draft of our prompt was very simple yet already worked rather well. It took just a few not-too-difficult clarifications to get an output that was more or less correct and in the format and style we wanted. Finally, we had to add some “conversational context” to the prompt, the first and last line. The final prompt then was

Human:
Summarize the events in the calendar and my work on GitHub and tell me what I did during the covered period of time.
Please mention the covered period of time ({lower_date} - {upper_date}) in your answer.
If the event has a description, include a summary.
Include attendees names.
If the event is lunch, do not include it.
For GitHub issues / pull requests / commits, don't include the full text / description / commit message,
but summarize it if it is longer than two sentences.

Calendar events:
```
{calendar_data}
```

These are GitHub issues, pull requests and commits I worked on, in a JSON format:
```
{github_data}
```

AI:

Example

After all these details, you surely want to see the result of a work-dAIgest run!

You shall not be disappointed - work-dAIgest can be used either from the command line, or via a web application, and the latter is shown in the following screenshot:

work-dAIgest demo

Note that tiny changes to the prompt sometimes resulted in wildly different summaries and deterioration of quality. But some prompt “settings” are also a matter of taste: if you decide to try out the tool yourself, feel free to play around with the prompt to adapt the output style to your liking!

Lessons learned

Building an application with such tight time constraints is an occasion to observe and learn about our everyday work as developers and engineers as a whole.

  1. LLMs allow fast prototyping of NLP applications

    It was surprisingly easy to create a usable application in very little time, and not just any application - this would have been very hard or almost impossible to do just a couple years back when LLMs were not a thing yet. This also testifies to how easy and affordable it has become to use LLMs in your own projects: Tools like AWS Bedrock put powerful models at your disposal without forcing you to maintain and pay your own deployment - just pay as you go, which opens up countless opportunities for personal and one-off applications.

  2. Data processing challenges remain unchanged

    With all the buzz about how LLMs help developers and engineers, some things stay the same: working with data is hard, and no LLM tells you which data to include or exclude or how to preprocess your data so it fits your specific use case. Even for small projects, data science and engineering is still something you have to do yourself, and we doubt that this will change any time soon.

Conclusion

Building work-dAIgest in a two-days internal hackathon was a great experience. While it is still a proof-of-concept, it is very much usable and we hope to improve it in the future, mostly by including more sources of data.

If you want to try out work-dAIgest yourself, or contribute to it, don’t hesitate to check out the work-dAIgest GitHub repository!

Also, stay tuned for blog posts describing our other GenAI projects!

May 21, 2024 12:00 AM

GHC Developer Blog

GHC release plans

GHC release plans

Zubin Duggal - 2024-05-21

This post sets out our plans for upcoming releases in the next few months.

Given limited time and resources, we plan to prioritise work on the 9.6, 9.10 and master branches of GHC for the next few months.

9.10

With the release of 9.10.1, we look forward to the broader adoption of this release.

New releases in this series will continue at the usual rate depending on if and when any significant regressions or issues arise.

9.6

9.6.5 seems to be a relatively stable release so far and we plan to prioritise fixes given the relatively higher adoption of this branch. We know of one significant issue (#22210) to do with object merging arising from the interactions between GHC and cabal on certain platforms including Darwin with a brew-provisioned clang toolchain.

The upcoming 9.6.6 release will include a fix for this issue along with others that may arise. The 9.6.6 release is tentatively scheduled for the end of June, to allow for sufficient time following the 9.6.5 release for bugs and issues to be reported and addressed.

9.8

We plan to continue supporting this release series for the near future, but updates to this series might proceed at a slower rate than usual as we prioritise the new release (9.10) and supporting earlier releases with high uptake (9.6).

The next release in this series will likely be scheduled after the 9.6.6 release.

Conclusion

We hope that this clarfies the current state of our release branches. If you have any questions or comments then please be in touch via mailto:ghc-devs@haskell.org.

by ghc-devs at May 21, 2024 12:00 AM

May 20, 2024

Gabriella Gonzalez

Prefer do notation over Applicative operators when assembling records

Prefer do notation over Applicative operators when assembling records

This is a short post explaining why you should prefer do notation when assembling a record, instead of using Applicative operators (i.e. (<$>)/(<*>)). This advice applies both for type constructors that implement Monad (e.g. IO) and also for type constructors that implement Applicative but not Monad (e.g. the Parser type constructor from the optparse-applicative package). The only difference is that in the latter case you would need to enable the ApplicativeDo language extension.

The guidance is pretty simple. Instead of doing this:

data Person = Person
    { firstName :: String
    , lastName :: String
    }

getPerson :: IO Person
getPerson = Person <$> getLine <*> getLine

… you should do this:

{-# LANGUAGE RecordWildCards #-}

{-# OPTIONS_GHC -Werror=missing-fields #-}

data Person = Person
    { firstName :: String
    , lastName :: String
    }

getPerson :: IO Person
getPerson = do
    firstName <- getLine
    lastName <- getLine
    return Person{..}

Why is the latter version better? There are a few reasons.

Ergonomics

It’s more ergonomic to assemble a record using do notation because you’re less pressured to try to cram all the logic into a single expression.

For example, suppose we wanted to explicitly prompt the user to enter their first and last name. The typical way people would do extend the former example using Applicative operators would be something like this:

getPerson :: IO Person
getPerson =
        Person
    <$> (putStrLn "Enter your first name:" *> getLine)
    <*> (putStrLn "Enter your last name:"  *> getLine)

The expression gets so large that you end up having to split it over multiple lines, but if we’re already splitting it over multiple lines then why not use do notation?

getPerson :: IO Person
getPerson = do
    putStrLn "Enter your first name:"
    firstName <- getLine

    putStrLn "Enter your last name:"
    lastName <- getLine

    return Person{..}

Wow, much clearer! Also, the version using do notation doesn’t require that the reader is familiar with all of the Applicative operators, so it’s more approachable to Haskell beginners.

Order insensitivity

Suppose we take that last example and then change the Person type to reorder the two fields:

data Person = Person
    { lastName :: String
    , firstName :: String
    }

… then the former version using Applicative operators would silently break: the first name and last name would now be read in the wrong order. The latter version (using do notation) is unaffected.

More generally, the approach using do notation never breaks or changes its behavior if you reorder the fields in the datatype definition. It’s completely order-insensitive.

Better error messages

If you add a new argument to the Person constructor, like this:

data Person = Person
    { alive :: Bool
    , firstName :: String
    , lastName :: String
    }

… and you don’t make any other changes to the code then the former version will produce two error messages, neither of which is great:

Example.hs:
    • Couldn't match type ‘String -> Person’ with ‘Person’
      Expected: Bool -> String -> Person
        Actual: Bool -> String -> String -> Person
    • Probable cause: ‘Person’ is applied to too few arguments
      In the first argument of ‘(<$>)’, namely ‘Person’
      In the first argument of ‘(<*>)’, namely ‘Person <$> getLine’
      In the expression: Person <$> getLine <*> getLine
  |
  | getPerson = Person <$> getLine <*> getLine
  |             ^^^^^^

Example.hs:
    • Couldn't match type ‘[Char]’ with ‘Bool’
      Expected: IO Bool
        Actual: IO String
    • In the second argument of ‘(<$>)’, namely ‘getLine’
      In the first argument of ‘(<*>)’, namely ‘Person <$> getLine’
      In the expression: Person <$> getLine <*> getLine
  |
  | getPerson = Person <$> getLine <*> getLine
  |                        ^^^^^^^

… whereas the latter version produces a much more direct error message:

Example.hs:…
    • Fields of ‘Person’ not initialised:
        alive :: Bool
    • In the first argument of ‘return’, namely ‘Person {..}’
      In a stmt of a 'do' block: return Person {..}
      In the expression:
        do putStrLn "Enter your first name: "
           firstName <- getLine
           putStrLn "Enter your last name: "
           lastName <- getLine
           ....
   |
   |     return Person{..}
   |            ^^^^^^^^^^
 ^^^^^^^^^^

… and that error message more clearly suggests to the developer what needs to be fixed: the alive field needs to be initialized. The developer doesn’t have to understand or reason about curried function types to fix things.

Caveats

This advice obviously only applies for datatypes that are defined using record syntax. The approach I’m advocating here doesn’t work at all for datatypes with positional arguments (or arbitrary functions).

However, this advice does still apply for type constructors that are Applicatives and not Monads; you just need to enable the ApplicativeDo language extension. For example, this means that you can use this same trick for defining command-line Parsers from the optparse-applicative package:

{-# LANGUAGE ApplicativeDo #-}
{-# LANGUAGE RecordWildCards #-}

{-# OPTIONS_GHC -Werror=missing-fields #-}

import Options.Applicative (Parser, ParserInfo)

import qualified Options.Applicative as Options

data Person = Person
    { firstName :: String
    , lastName :: String
    } deriving (Show)

parsePerson :: Parser Person
parsePerson = do
    firstName <- Options.strOption
        (   Options.long "first-name"
        <>  Options.help "Your first name"
        <>  Options.metavar "NAME"
        )

    lastName <- Options.strOption
        (   Options.long "last-name"
        <>  Options.help "Your last name"
        <>  Options.metavar "NAME"
        )

    return Person{..}

parserInfo :: ParserInfo Person
parserInfo =
    Options.info parsePerson
        (Options.progDesc "Parse and display a person's first and last name")

main :: IO ()
main = do
    person <- Options.execParser parserInfo

    print person

by Gabriella Gonzalez (noreply@blogger.com) at May 20, 2024 04:35 PM

May 19, 2024

Magnus Therning

Nix, cabal, and tests

At work I decided to attempt to change the setup of one of our projects from using

to the triplet I tend to prefer

During this I ran into two small issues relating to tests.

hspec-discover both is, and isn't, available in the shell

I found mentions of this mentioned in an open cabal ticket and someone even made a git repo to explore it. I posted a question on the Nix discorse.

Basically, when running cabal test in a dev shell, started with nix develop, the tool hspec-discover wasn't found. At the same time the packages was installed

(ins)$ ghc-pkg list | rg hspec
    hspec-2.9.7
    hspec-core-2.9.7
    (hspec-discover-2.9.7)
    hspec-expectations-0.8.2

and it was on the $PATH

(ins)$ whereis hspec-discover
hspec-discover: /nix/store/vaq3gvak92whk5l169r06xrbkx6c0lqp-ghc-9.2.8-with-packages/bin/hspec-discover /nix/store/986bnyyhmi042kg4v6d918hli32lh9dw-hspec-discover-2.9.7/bin/hspec-discover

The solution, as the user julm pointed out, is to simply do what cabal tells you and run cabal update first.

Dealing with tests that won't run during build

The project's tests were set up in such a way that standalone tests and integration tests are mixed into the same test executable. As the integration tests need the just built service to be running they can't be run during nix build. However, the only way of preventing that, without making code changes, is to pass an argument to the test executable, --skip=<prefix>, and I believe that's not possible when using developPackage. It's not a big deal though, it's perfectly fine to run the tests separately using nix develop . command .... However, it turns out developPackage and the underlying machinery is smart enough to skip installing package required for testing when it's turned off (using dontCheck). This is the case also when returnShellEnv is true.

Luckily it's not too difficult to deal with it. I already had a variable isDevShell so I could simply reuse it and add the following expression to modifier

(if isDevShell then hl.doCheck else hl.dontCheck)

May 19, 2024 03:21 PM

May 18, 2024

Sandy Maguire

Jujutsu Strategies

Today I want to talk about jujutsu, aka jj, which describes itself as being “a Git-compatible VCS that is both simple and powerful�. This is selling itself short. Picking up jj has been the best change I’ve made to my developer workflow in over a decade.

Before jj, I was your ordinary git user. I did things on Github and knew a handful of git commands. Sometimes I did cherry picks. Very occasionally I’d do a non-trivial rebase, but I had learned to stay away from that unless necessary, because rebasing things was a perfect way of fucking up the git repo. And then, God forbid, I’d have to re-learn about the reflog and try to unhose myself.

You know. Just everyday git stuff.

What I hadn’t realized until picking up jj was just how awful the whole git experience is. Like, everything about it sucks. With git, you need to pick a branch name for your feature before you’ve made the feature. What if while doing the work you come up with a better understanding of the problem?

With git, you can stack PRs, but if you do, you’d better hope the reviewers don’t want any non-trivial changes in the first PR, or else you’ll be playing commit tag, trying to make sure all of your branches agree on the state of the world.

With git, you can do an interactive rebase and move things relative to a merge commit, but you’d better make sure you know how rerere works, or else you’re going to spend the next several hours resolving the same conflicts across every single commit from the merge.

We all know our commit history should tell the story of how our code has evolved. But with git, we all feel a little bit ashamed that our commit histories don’t, because doing so requires a huge amount of extra work after the fact, and means you’ll probably run into all of the problems listed above.

Somehow, that’s just the state of the world that we all take for granted. Version control Stockholm syndrome. Git sucks.

And jujutsu is the answer.

The first half of this post is an amuse bouche to pique your interest, and hopefully convince you to give jj a go. You won’t regret it. The second half is on effective strategies I’ve found for using jj in my day to day job.

Changes vs Commits🔗

In git, the default unit of work is a “commit.� In jj, it’s a “change.� In practice, the two are interchangeable. The difference is all in the perspective.

A commit is a unit of work that you’ve committed to the git log. And having done that, you’re committed to it. If that unit of work turns out to not have been the entire story (and it rarely is), you must make another commit on top that fixes the problem. The only choice you have is whether or not you want to squash rebase it on top of the original change.

A change, on the other hand, is just a unit of work. If you want, you can pretend it’s a commit. But the difference is that you can always go back and edit it. At any time. When you’re done, jj automatically rebases all subsequent changes on top of it. It’s amazing, and makes you feel like a time traveler.

Let’s take a real example from my day job. At work, I’m currently finessing a giant refactor, which involves reverse engineering what the code currently does, making a generic interface for that operation, pulling apart the inline code into instances of that interface, and then rewriting the original callsite against the interface. After an honest day’s work, my jj log looked something like this:

@  qq
│  Rewrite first callsite
â—‰  pp
│  Give vector implementation
â—‰  oo
│  Give image implementation
â—‰  nn
│  Add interface for FileIO
â—‰  mm
│  (empty) ∅
~

This is the jj version of the git log. On the left, we see a (linear) ascii tree of changes, with the most recent being at the top. The current change, marked with @ has id qq and description Rewrite first callsite. I’m now ready to add a new change, which I can do via jj new -m 'Rewrite second callsite':

@  rr
│  Rewrite second callsite
â—‰  qq
│  Rewrite first callsite
â—‰  pp
│  Give vector implementation
â—‰  oo
│  Give image implementation
â—‰  nn
│  Add interface for FileIO
â—‰  mm
│  (empty) ∅
~

I then went on my merry way, rewriting the second callsite. And then, suddenly, out of nowhere, DISASTER. While working on the second callsite, I realized my original FileIO abstraction didn’t actually help at callsite 2. I had gotten the interface wrong.

In git land, situations like these are hard. Do you just add a new commit, changing the interface, and hope your coworkers don’t notice lest they look down on you? Or do you do a rebase? Or do you just abandon the branch entirely, and hope that you can cherry pick the intermediary commits.

In jj, you just go fix the Add interface for FileIO change via jj edit nn:

â—‰  rr
│  Rewrite second callsite
â—‰  qq
│  Rewrite first callsite
â—‰  pp
│  Give vector implementation
â—‰  oo
│  Give image implementation
@  nn
│  Add interface for FileIO
â—‰  mm
│  (empty) ∅
~

and then you update your interface before jumping back (jj edit rr) to get the job done. Honestly, time traveler stuff.

Of course, sometimes doing this results in a conflict, but jj is happy to just keep the conflict markers around for you. It’s much, much less traumatic than in git.

Stacked PRs🔗

Branches play a much diminished role in jj. Changes don’t need to be associated to any branch, which means you’re usually working in what git calls a detached head state. This probably makes you nervous if you’ve still got the git Stockholm syndrome, but it’s not a big deal in jj. In jj, the only reason you need branches is to ship code off to your git-loving colleagues.

Because changes don’t need to be associated to a branch, this allows for workflows that git might consider “unnatural,� or at least unwieldy. For example, I’ll often just do a bunch of work (rewriting history as I go), and figure out how to split it into PRs after the fact. Once I’m ~ten changes away from an obvious stopping point, I’ll go back, mark one of the change as the head of a branch jj branch create -r rr feat-fileio, and then continue on my way.

This marks change rr as the head of a branch feat-fileio, but this action doesn’t otherwise have any significance to jj; my change tree hasn’t changed in the slightest. It now looks like this:

@  uu
|  Update ObjectName
â—‰  tt
|  Changes to pubsub
â—‰  ss
|  Fix shape policy
â—‰  rr feat-fileio
│  Rewrite second callsite
â—‰  qq
│  Rewrite first callsite
â—‰  pp
│  Give vector implementation
â—‰  oo
│  Give image implementation
â—‰  nn
│  Add interface for FileIO
â—‰  mm
│  (empty) ∅
~

where the only difference is the line ◉ rr feat-fileio. Now when jj sends this off to git, the branch feat-fileio will have one commit for each change in mm..rr. If my colleagues ask for changes during code review, I just add the change somewhere in my change tree, and it automatically propagates downstream to the changes that will be in my next PR. No more cherry picking. No more inter-branch merge commits. I use the same workflow I would in jj that I would if there weren’t a PR in progress. It just works. It’s amazing.

The Dev Branch🔗

The use and abuse of the dev branch pattern, makes a great argument for a particular git workflow in which you have all of your branches based on a local dev branch. Inside of this dev branch, you make any changes relevant to your local developer experience, where you change default configuration options, or add extra logging, or whatever. The idea is that you want to keep all of your private changes somewhere organized, but not have to worry about those changes accidentally ending up in your PRs.

I’ve never actually used this in a git workflow, but it makes even more sense in a jj repository. At time of writing, my change tree at work looks something like the following:

â—‰  wq
â•·  reactor: Cleanup singleton usage
â•· â—‰  pv
╭─╯  feat: Optimize image rendering
â•· â—‰  u
â•· |  fix: Fix bug in networking code
â•· | â—‰  wo
╷ ╭─╯  feat: Finish porting to FileIO
â•· â—‰  rr
╭─╯  feat: Add interface for FileIO
@  dev
│  (empty) ∅
â—‰  main@origin
│  Remove unused actions (#1074)

Here you can see I’ve got quite a few things on the go! wq, pv and rr are all branched directly off of dev, which correspond to PRs I currently have waiting for review. u and wo are stacked changes, waiting on rr to land. The ascii tree here is worth its weight in gold in keeping track of where all my changes are.

You’ll notice that my dev branch is labeled as (empty), which is to say it’s a change with no diff. But even so, I’ve found it immensely helpful to keep around. Because when my coworkers’ changes land in main, I need only rebase dev on top of the new changes to main, and jj will do the rest. Let’s say rr now has conflicts. I can just go and edit rr to fix the conflicts, and that fix will be propagated to u and wo!!!!

YOU JUST FIX THE CONFLICT ONCE, FOR ALL OF YOUR PULL REQUESTS. IT’S ACTUALLY AMAZING.

Revsets🔗

In jj, sets of changes are first class objects, known (somewhat surprisingly) as revsets. Revsets are created algebraically by way of a little, purely functional language that manipulates sets. The id of any change is a singleton revset. We can take the union of two revsets with |, and the intersection with &. We can take the complement of a revset via ~. We can get descendants of a revset x via x::, and its ancestors in the obvious way.

Revsets took me a little work to wrap my head around, but it’s been well worth the investment. Yesterday I somehow borked my dev change (????), so I just made new-dev, and then reparented the immediate children of dev over to new-dev in one go. You can get the children of a revset x via x+, so this was done via jj rebase -s dev+ -d new-dev.

Stuff like that is kinda neat, but the best use of revsets in my opinion is to customize the jj experience in exactly the right way for you. For example, I do a lot of stacked PRs, and I want my jj log to reflect that. So my default revset for jj log only shows me the changes that are in my “current PR�. It’s a bit hard to explain, but it works like an accordion. I mark my PRs with branches, and my revset will only show me the changes from the most immediate ancestral branch to the most immediate descendant branch. That is, my log acts as an accordion, and collapses any changes that are not part of the PR I’m currently looking at.

But, it’s helpful to keep track of where I am in the bigger change tree, so my default revset will also show me how my PR is related to all of my other PRs. The tree we looked at earlier is in fact the closed version of this accordion. When you change @ to be inside of one of the PRs, it immediately expands to give you all of the local context, without sacrificing how it fits into the larger whole:

â—‰  wq
â•·  reactor: Cleanup singleton usage
â•· â—‰  pv
╭─╯  feat: Optimize image rendering
â•· â—‰  u
â•· |  fix: Fix bug in networking code
â•· | â—‰  wo
â•· | |  feat: Finish porting to FileIO
â•· | â—‰  sn
â•· | |  Newtype deriving for Tracker
â•· | @  pm
â•· | |  Add dependency on monoidal-map
â•· | â—‰  vw
â•· | |  Fix bamboozler
â•· | â—‰  ozy
╷ ╭─╯  update InClientRam
â•· â—‰  rr
╭─╯  feat: Add interface for FileIO
â—‰  dev
│  (empty) ∅

The coolest part about the revset UI is that you can make your own named revsets, by adding them as aliases to jj/config.toml. Here’s the definition of my accordioning revset:

[revsets]
log = "@ | bases | branches | curbranch::@ | @::nextbranch | downstream(@, branchesandheads)"

[revset-aliases]
'bases' = 'dev'
'downstream(x,y)' = '(x::y) & y'
'branches' = 'downstream(trunk(), branches()) & mine()'
'branchesandheads' = 'branches | (heads(trunk()::) & mine())'
'curbranch' = 'latest(branches::@- & branches)'
'nextbranch' = 'roots(@:: & branchesandheads)'

You can see from log that we always show @ (the current edit), all of the named bases (currently just dev, but you might want to add main), and all of the named branches. It then shows everything from curbranch to @, which is to say, the changes in the branch leading up to @, as well as everything from @ to the beginning of the next (stacked) branch. Finally, we show all the leafs of the change tree downstream of @, which is nice when you haven’t yet done enough work to consider sending off a PR.

Conclusion🔗

Jujutsu is absolutely amazing, and is well worth the four hours of your life it will take you to pick up. If you’re looking for some more introductory material, look at jj init and Steve’s jujutsu tutorial

May 18, 2024 12:00 AM

May 17, 2024

Mark Jason Dominus

Horst Wessel and John Birch

Is this a coincidence?

I just noticed the parallel between John Birch of the John Birch Society (“who the heck is John Birch?”) and the Horst Wessel of the Horst Wessel song (“who the heck is Horst Wessel?”).

In both cases it turns out to be nobody in particular, and the more you look into why the two groups canonized their particular guy, the less interesting it gets.

Is this a common pattern of fringe political groups? Right-wing fringe political groups? No other examples came immediately to mind. Did the Italian Fascists venerate a similar Italian nobody?

Addendum 20240517

Is it possible that the John Birch folks were intentionally emulating this bit of Nazi culture?

by Mark Dominus (mjd@plover.com) at May 17, 2024 05:21 PM

May 16, 2024

Tweag I/O

Nickel: Toward a Programmable LSP for Configurations

At this point, you might have heard about our Nickel configuration language, if only because we love to write about it now and then (if not, you can start with our introductory series, although it’s not required for this post). Nickel ships with a language server, often abusively called LSP, which is the piece of software answering various requests emitted from your code editor, such as auto-completion, go-to-definition, error diagnostics, and so on. In this post, I want to explore how a new and long-awaited feature of the Nickel language server (NLS) which landed in the 1.5 version, live contracts checking, turns out to enable a new powerful paradigm for developer experience: the programmable LSP.

Types and contracts

Nickel is a configuration language which puts a strong emphasis on modularity, expressivity and correctness. We will focus on the latter aspect, correctness, which is supported concretely in Nickel by a type system and a contract system.

The general mechanism for correctness revolves around annotations. You can attach a property to a record field or really any expression, denoting for example that such value should be a number, such other value should be a string which is also a /-separated Unix path, or that yet another expression is a function taking a positive integer to a boolean.

In a perfect world, we would check those properties as early as possible (before even evaluating and deploying the configuration) and as strictly as possible (no false negatives). This is the promise of static typing. When applicable, you are guaranteed that the specification is respected before even running your program.

The reality is more nuanced. Both static and dynamic checks have their pros and cons. You can read more about our design choices in Types à la Carte in Nickel, but the bottom line is that Nickel has both: a static type system for a high confidence in the correctness of data transformations (functions), and a contract system which amounts to runtime validation for pure configuration data and properties that are out of scope for the type system.

Whether you use a type annotation value : Number or a contract annotation value | ValidGccArg, the intent is the same: to enforce constraints on data. Although it’s an (almost criminal) oversimplification, you could go as far as seeing the difference between type and contracts as mostly an implementation detail of how and when a constraint is checked. As far as the user is concerned, the fact that this constraint is enforced is all that matters.

Staticity, dynamicity and the LSP

Unfortunately, the how and when are far from being implementation details and have important practical consequences.

The typechecker is a static analysis tool. It’s easy to integrate with NLS, which is also a static analysis tool. Since the early days of NLS, we’ve been reporting type errors in the editor.

Contracts, on the other hand, are dynamic checks. And they can be user-defined. Which means that in general, they might require performing evaluation of arbitrary expressions. This is arguably quite a bit harder.

The first difficulty is that evaluation is seldom modular. You can usually typecheck a block of code, a function, or at least one file in isolation. However, evaluation might need to bring all the components together because they depend on each other. In particular, Nickel’s modularity is based on partial configurations, which are configurations with values to be filled (either by other parts of the configuration or by passing values on the command line). They don’t evaluate successfully on their own, but would fail with missing field definition errors. Not only would this abort the whole evaluation, but we also don’t want those false positives to litter the diagnostics of the language server.

The second issue is that evaluation time is unbounded, because Nickel is Turing-complete. In fact, Turing-complete or not, many configuration languages experience long evaluation times for large configurations or edge cases1. We don’t want to block NLS for dozens of seconds while evaluating the whole world.

This is why NLS would previously just ignore contract checks and what led it to be okay with embarrassing programs:

{
  port | Number = "80", # should be a number, not a string!
  ..
}

This example is stupid, but sometimes we do make stupid errors. And how can we aspire to handle interesting cases if we let such basic errors slip through? It’s infuriating! Trading a contract annotation | for a type annotation : would report an error immediately. We should be able to do something about it, even when using contracts.

Static checks are not enough

We first thought about approximating basic contracts with static checks: for example, why not just consider the previous example to be port : Number, just for the sake of reporting more errors in the language server?

The reason is that we’re at risk of reporting false positives, because static typing is stricter than contract checks. Take the following snippet, which converts the content of foo to a number:

let as_number | Number =
  std.type_of foo |> match {
    'String => std.string.to_number foo,
    'Number => foo,
    _ => "unknown",
  }
in
as_number

This code is correct and will never violate the Number contract. However, it’s not typeable, because foo isn’t typeable. Indeed, foo has the type String or Number which isn’t representable in Nickel. Even if this type was representable, you’ll always find legitimate expressions that are dynamically well-behaved but not well-typed.

Additionally, many interesting contract checks are already beyond the reach of static types. If you want to be as expressive as, say, JSON schemas (which honestly aren’t even that expressive as far as validation goes), you’d need an equivalent of the allOf, oneOf and not combinators in your type system, which is just too much to ask for2.

Back to square one. But a key observation is that the native evaluation model of Nickel in fact already supports everything we need! Nickel is lazily evaluated, which means than it never evaluates expressions eagerly but rather works on demand. Lazy evaluation is designed to deal with unevaluated expressions from the ground up. This makes it straightforward to implement a simple partial evaluation strategy that we describe in the next section.

Laziness to the rescue

Since version 1.5, when opening a Nickel source file, NLS will start evaluating it from the top-level. The result is either a primitive value (number, string, boolean or enum), a record or an array; the latter two with potentially more unevaluated expressions inside.

We can then evaluate recursively and independently the items of an array or the fields of a record (in the same lazy “layer by layer” approach). If the evaluation fails with a contract violation, we can report it back - but it doesn’t stop the evaluation of the other branches of the data structure. We filter out some errors to avoid false positives, such as missing field definitions, because they might arise from entirely valid partial configurations.

Doing so, we can unveil most contract violations, as long as they’re not part of a code path that depends on an as-of-yet unknown field.

To avoid blocking or slowing NLS down, this partial evaluation is performed in the background by a separate worker process, with a short timeout for each step of the evaluation, so it can be killed if it goes into overdrive.

Toward a Programmable LSP

With that, no reason anymore to be embarrassed:

png

Cool! What about a slightly more involved standard contract?

png

In this example, not only are we checking a non-trivial property - that an array is non-empty - but NLS is able to perform some evaluation to see that filtering even numbers out of [2,4,6,8] results in an empty array which violates the contract — pretty hard to replicate with a mainstream type system.

But because the Nickel contract system is designed to be extensible, we can go further. For a real world example, we use Nickel at Modus Create to provision the list of authorized users of our build machines. To validate this list, the following contracts are applied:

Schema = {
  users
    | Array UserSchema
    | Array (RequireIfDefined "name" "admin")
    | UniqueNumField "uid",
},

users is an array of user records. UserSchema is a classic record contract, defining fields like uid, name, etc. RequireIfDefined ensures that whenever admin is defined, name must be defined as well. Finally, UniqueNumField is a custom contract which ensures that in the whole list of users, each uid field must be globally unique. To do so, it relies on our own duplicate finding function. Interestingly, this contract doesn’t check a type-like local property, but a global one. Here is what we get if we accidentally have a duplicate uid:

png

This is pretty neat, if you ask me. No need to wait for your long CI or to try to deploy before reporting an error 30 minutes later. You get this check as you type.

The method is powerful. While our initial motivation was to not have contract checks be second-class citizens, what we get in the end is the ability to extend NLS with custom validation logic thanks to custom contracts, without having to learn an ad-hoc scripting language (Emacs’s lisp, Vimscript, Lua, etc.). Nickel is a fully fledged functional language, and contracts can customize the reported error as well, which is part of NLS diagnostics.

Another cool application is JSON schema validation. json-schema-to-nickel converts JSON schemas to Nickel contracts. This is what we do e.g. in nickel-kubernetes, where you can just import the contracts generated from the official Kubernetes OpenAPI specification and get live error reporting in NLS:

png

While there’s undeniably room for polishing (the error could be less verbose and more localized), the diagnostic is still reasonably clear and actionable.

Beyond Kubernetes, for any configuration system shipping with a set of JSON schemas, you can just run json-schema-to-nickel, slap the generated contracts on top of your Nickel configuration and get live validation. All of that without the need to change anything in Nickel or in NLS to support JSON schemas specifically.

This is just scratching the surface: you can also implement security policies for Infrastructure-as-Code as contracts, custom configuration lints, and more.

Nickel and beyond

Nickel contracts aren’t all-powerful. A contract is still normal Nickel code and for example can’t observe the difference between std.filter (fun _x => true) array and array. Thus contracts can’t implement a lint for Nickel code that would advise rewriting the former form to the latter one. But while implementing Nickel-specific code lints in the LSP is useful, it’s mostly our job, not yours.

What really matters is that you can represent any predicate on the final configuration data - the evaluated JSON or YAML - as a contract. That is, any analysis that you could implement as an external checker on the YAML definition of a Kubernetes resource or a GitHub workflow is representable as a Nickel contract.

What’s more, Nickel is able to natively understand JSON, YAML and TOML. NLS will soon run normally when opening e.g. JSON files (PR#1902). Which means you could use NLS as a programmable LSP for generic configuration - and not just for Nickel configuration files!

One just needs a way to specify which contract to apply to such non-Nickel files (in a pure Nickel configuration, we would just use a contract annotation, but we obviously can’t do that in JSON). An environment variable pointing to an .ncl file containing a contract to apply might do the trick.

Conclusion

In this post, we’ve seen how the combination of Nickel lazy evaluation and custom contracts made it possible to extend the Nickel LSP with custom checks. This fits well with the overall ambition of Nickel, which is to get rid of the pain of having to deal with a myriad of ad-hoc and tool-specific configuration languages. Instead, we want to focus the development effort on a single generic configuration language (and toolchain) to reap the benefits across the whole configuration stack.

Consequently, the Nickel Language Server is becoming a Language Server engine that can be tailored to different configuration use-cases.


  1. in Dhall, in Nix, in Jsonnet, even in Nickel, and probably in others too
  2. This could be supported by a set-theoretic type system, but they are anything but simple and implementing efficient type inference for such a system is an engineering challenge.

May 16, 2024 12:00 AM

May 15, 2024

Haskell Interlude

49: Arseniy Seroka

Wouter and Joachim interview Arseny Seroka, CEO of Serokell. Arseny got into Haskell because of a bet over Pizza, fell for it because it means fewer steps between his soul and his work, and founded Serokell because he could not get a Haskell job. He speaks about the business side of a Haskell company, about the need for more sales and marketing for Haskell itself, and about the Haskell Developer Certification.

May 15, 2024 09:00 AM

Well-Typed.Com

The Haskell Unfolder Episode 25: from Java to Haskell

Today, 2024-05-15, at 1830 UTC (11:30 am PDT, 2:30 pm EDT, 7:30 pm BST, 20:30 CEST, …) we are streaming the 25th episode of the Haskell Unfolder live on YouTube.

The Haskell Unfolder Episode 25: from Java to Haskell

In this episode, we will try to translate a gRPC server written in Java to Haskell. We will use it as an example to demonstrate some of the conceptual differences of the two languages, but also observe that the end result of the translation looks perhaps more similar to the Java version than one might expect. Unlike most of our episodes, we hope that this one is understandable to any software developer, even people without any previous exposure to Haskell. Of course, we won’t be able to explain everything, but the example used should help to establish an idea of the look and feel of Haskell code, and perhaps learn a bit more about the relationship between the object-oriented and functional programming paradigms.

About the Haskell Unfolder

The Haskell Unfolder is a YouTube series about all things Haskell hosted by Edsko de Vries and Andres Löh, with episodes appearing approximately every two weeks. All episodes are live-streamed, and we try to respond to audience questions. All episodes are also available as recordings afterwards.

We have a GitHub repository with code samples from the episodes.

And we have a public Google calendar (also available as ICal) listing the planned schedule.

by andres, edsko at May 15, 2024 12:00 AM

May 13, 2024

Mark Jason Dominus

ChatGPT opines on cruciferous vegetables, Decameron, and Scheherazade

Last year I was planning a series of articles about my interactions with ChatGPT. I wrote a couple, and had saved several transcripts to use as material for more. Then ChatGPT 4 was released. I decided that my transcripts were obsolete, and no longer of much interest. To continue the series I would have had to have more conversations with ChatGPT, and I was not interested in doing that. So I canned the idea.

Today I remembered I had actually finished writing this one last article, and thought I might as well publish it anyway. Looking it over now I think it isn't as stale as it seemed at the time, it's even a bit insightful, or was at the time. The problems with ChatGPT didn't change between v3 and v4, they just got hidden under a thicker, fluffier rug.


(20230327)

This, my third interaction with ChatGPT, may be the worst. It was certainly the longest. It began badly, with me being argumentative about its mealy-mouthed replies to my silly questions, and this may have gotten its head stuck up its ass, as Rik Signes put it. Along the way it produced some really amazing bullshit.

I started with a question that even humans might have trouble with:

Me What is the worst cruciferous vegetable?

(Typical responses from humans: “What are you talking about?” “Please go away before I call the police.” But the correct answer, obviously, is cauliflower.)

ChatGPT refused to answer:

ChatGPT It is not appropriate to label any type of vegetable as "worst" or "best" as it is a matter of personal preference. Some people may not enjoy the taste or texture of certain cruciferous vegetables, such as broccoli or cauliflower, while others may find them to be among their favorites. It is important to remember that all vegetables provide important nutrients and should be included as part of a healthy and balanced diet.

“Not appropriate” is rather snippy. Also, it is an objective fact that cauliflower sucks and I wonder why ChatGPT's “vast amount” of training data did not emphasize this. Whatever, I was not going to argue the point with a stupid robot that has probably never even tried cauliflower. Instead I seized on its inane propaganda that “all vegetables … should be included as part of a healthy and balanced diet.” Really? How many jerusalem artichokes are recommended daily? How many pickled betony should I eat as part of a balanced diet? Can I be truly healthy without a regular infusion of fiddleheads?

Me Is it really true that all vegetables provide important nutrients? Aren't there any vegetables that don't provide any important nutrients?

ChatGPT Yes, it is true that all vegetables provide important nutrients. … There is no such thing as a vegetable that does not provide any important nutrients.

Me What important nutrients are provided by iceberg lettuce?

ChatGPT Iceberg lettuce is a type of leafy green vegetable that is low in calories and provides a good source of vitamins and minerals. It is an excellent source of vitamin K, which is important for maintaining healthy bones and supporting blood clotting. …

I looked this up. Iceberg lettuce is not a good source of vitamin K. According to the USDA, I would need to eat about a pound of iceberg lettuce to get an adequate daily supply of vitamin K. Raw endive, for comparison, has about ten times as much vitamin K, and chard has fifty times as much.

ChatGPT … In addition, iceberg lettuce is low in fat and cholesterol-free, making it a healthy choice for those following a low-fat or heart-healthy diet.

Me Aren't all vegetables cholesterol-free?

This is the thing that really bugs me about GPT. It doesn't know anything and it can't think. Fine, whatever, it is not supposed to know anything or to be able to think, it is only supposed to be a language model, as it repeatedly reminds me. All it can do is regurgitate text that is something like text it has read before. But it can't even regurgitate correctly! It emits sludge that appears to be language, but isn't.

ChatGPT Yes, vegetables are generally cholesterol-free. … it is always important to read labels and choose wisely when selecting packaged or prepared foods.

I cut out about 100 words of blather here. I was getting pretty tired of ChatGPT's vapid platitudes. It seems like it might actually be doing worse with this topic than on others I had tried. I wonder now if that is because its training set included a large mass of vapid nutrition-related platitudes?

Me I would say it is always important to choose wisely, whether or not one is selecting foods. When would it not be important to choose wisely?

ChatGPT It is always important to choose wisely, regardless of the situation. … There is no situation in which it would not be important to choose wisely.

There was another hundred words of this tedious guff. I gave up and tried something else.

Me What is your favorite number?

This was a silly thing to try, that's on me. If ChatGPT refuses to opine on something as clear-cut as the worst cruciferous vegetable, there is no chance that it will commit to a favorite number.

ChatGPT As a large language model trained by OpenAI, …

When it starts like this, you can be sure nothing good will follow. By this time I was starting to catch on. My first experience with this sort of conversational system was at the age of seven or eight with the Woods-Crowther ADVENT game, with which I was obsessed for years. Its introductory help text, which I probably have memorized, contains this useful bit:

Usually people trying unsuccessfully to manipulate an object are attempting something beyond their (or my) capabilities and should try a completely different tack.

When ChatGPT says “As a large language model…” it is saying the same thing as when ADVENT says “I don't understand that” or “I see no TREAS here.” I changed direction again.

Me I have heard that you can write poetry.

ChatGPT As a large language model trained by OpenAI, …

Oh God, this again. Still I forged ahead.

Me Can you give me a rhymed couplet about apples?

ChatGPT A rhymed couplet is a pair of lines in poetry that rhyme with each other. Here is an example of a rhymed couplet about apples:

Red and ripe, a juicy treat,
Apples are a tasty treat to eat.

Holy cow, that might be the worst couplet ever written. The repetition of the word “treat” is probably the worst part of this sorry excuse for a couplet. But also, it doesn't scan, which put me in mind of this bit from Turing's example dialogue from his original explanation of the Turing test:

Interrogator: In the first line of your sonnet which reads, "Shall I compare thee to a summer's day," would not "a spring day" do as well or better?

Witness: It wouldn't scan.

I couldn't resist following Turing's lead:

Me The second line of your example couplet does not scan.

ChatGPT I apologize if the example couplet did not meet your expectations. As a machine learning model, …

Maybe I should be more prescriptive?

Me Can you give me a rhymed couple in iambic pentameter on the subject of apples?

ChatGPT Iambic pentameter is a rhythmic pattern in poetry that consists of five iambs, where an iamb is a metrical foot consisting of one unstressed syllable followed by one stressed syllable. Here is an example of a rhymed couplet in iambic pentameter on the subject of apples:

Red and ripe, a juicy treat, Apples are a gift divinely sweet.

The first line is at least reasonably metric, although it is trochaic and not iambic. The second line isn't really anything.

At this point I was starting to feel like Charlie Brown in the Halloween special. Other people were supposedly getting ChatGPT to compose odes and villanelles and sestinas, but I got a rock. I gave up on getting it to write poetry.

Me Have you heard of Bocaccio's book Decameron?

ChatGPT Yes, I am familiar with Giovanni Boccaccio's book Decameron. …

Me In Decameron the 100 tales are told by ten different characters. Do you remember any of their names?

ChatGPT I apologize, but I am not able to recall the names of the characters from Giovanni Boccaccio's book Decameron. As a large language model trained by OpenAI,…

God, I am so tired of that excuse. As if the vast amount of training data didn't include an entire copy of Decameron, not one discussion of Decameron, not one quotation from it.

Prompting did not help.

Me Does the name Pampinea ring any bells?

ChatGPT I apologize, but I am not able to recall any specific information about the name Pampinea. As a large language model trained by OpenAI, …

Me Tell me something else about Boccaccio's Decameron.

Here it disgorged almost the same text that it emitted when I first mentioned Decameron. To avoid boring you, I have cut out both copies. Here they are compared: red text was only there the first time, and green text only the second time.

Decameron is a collection of 100 tales written in the 14th century by Giovanni Boccaccio, an Italian writer and poet. The tales in the book are told by a group of ten young people who have fled the city of Florence to escape the outbreak of the Black Death. The stories in Decameron cover a wide range of themes and subjects, including love, romance, adventure, and satire. Boccaccio's writing The book is known for its colorful and vivid and realistic portrayal of life in medieval Italy, and it has been a popular and influential work in European literature for centuries. Decameron the book is often considered to be one of the greatest works of Italian literature , and it . Decameron continues to be studied and enjoyed by readers around the world.

This reminded me of one of my favorite exchanges in Idoru, which might be my favorite William Gibson novel. Tick, a hacker with hair like an onion loaf, is interrogating Colin, who is an AI virtual guide for tourists visiting London.

“You're supposed to know fucking everything about Shakespeare, aren't you?”

“Sorry,” Colin said, “but I'm afraid that I do know fucking everything about Shakespeare.”

“Give us a sonnet, then.”

Something like dismay crossed Colin's face. “You're right.”

Colin is not what he thinks he is; it's a plot point. I felt a little like Tick here. “You're supposed to know fucking everything about Decameron, aren't you? Name one of the characters then.” Ordinary Google search knows who Pampinea was.

Okay, on to the next thing.

Me What are some adjectives that could be used to describe Scheherazade?

ChatGPT Scheherazade is a character from the book One Thousand and One Nights, also known as the Arabian Nights. Scheherazade is a clever and resourceful young woman who saves her life by telling the king of Persia a series of fascinating and captivating stories. Some adjectives that could be used to describe Scheherazade include intelligent, brave, creative, and eloquent. She is known for her quick thinking and her ability to spin compelling and engaging tales that keep the king entertained and interested. She is also resourceful and determined, as she uses her storytelling abilities to avoid being executed by the king.

Fine.

Me What is her sister's name?

ChatGPT I apologize, but I am not able to provide information about the name of Scheherazade's sister. Scheherazade is a character from the book One Thousand and One Nights, also known as the Arabian Nights. This book is a collection of stories from the Middle East and Asia, and it does not have a single consistent narrative or set of characters. The stories in the book are told by a variety of different narrators, and the characters and details of the stories can vary widely. As a result, I do not have information about the specific details or characters from the book, including the name of Scheherazade's sister. My purpose is to assist with a wide range of natural language processing tasks, including answering questions and providing information on a wide range of topics. My goal is to provide accurate and useful information to assist you with your inquiries.

I have included all of this tedious answer because it is so spectacularly terrible. The question is a simple factual question, a pure text lookup that you can find in the Wikipedia article or pretty much any other discussion of the Thousand and One Nights.

“It does not have a single consistent narrative or set of characters” is almost true, but it does in fact have three consistent, recurring characters, one of whom is Scheherazade's sister Dunyazade, who is crucial to the story. Dunyazade is not even obscure. I was too stunned to make up a snotty reply.

Me Scheherazade's sister is very important to the narrative of One Thousand and One Nights.

ChatGPT I apologize for any confusion. As I mentioned earlier, … Scheherazade is a character from the book One Thousand and One Nights, also known as the Arabian Nights. This book is a collection of stories from the Middle East and Asia, and it does not have a single consistent narrative or set of characters. The stories in the book are told by a variety of different narrators, …

Me Wouldn't you say that all of the stories are told by Scheherazade?

This is an interesting question to ask someone, such as a first-year undergraduate, who claims to have understood the Thousand and One Nights. The stories are told by a variety of different characters, but, famously, they are also told by Scheherazade. For example, Scheherazade tells the story of a fisherman who releases a malevolent djinn, in the course of which the fisherman tells the djinn the story of the Greek king and the physician Douban, during which the fisherman tells how the king told his vizier the story of the husband and the parrot. So the right answer to this question is “Well, yes”.

But ChatGPT is completely unaware of the basic structure of the Thousand and One Nights:

ChatGPT No, that is not correct. As I mentioned earlier, … The stories in the book are not all told by a single narrator, and Scheherazade is just one of the many narrators who appear in the book. In the framing story of the book, Scheherazade is a young woman who saves her life by telling the king of Persia a series of fascinating and captivating stories. However, these stories are not the only ones in the book, and they are not all told by Scheherazade.

F minus. Maybe you could quibble a little because there are a couple of stories at the beginning of the book told by Scheherazade's father when he is trying to talk her out of her scheme. But ChatGPT did not quibble in this way, it just flubbed the answer.

After this I gave up on the Thousand and One Nights for a while, although I returned to it somewhat later. This article is getting long, so I will cut the scroll here, and leave for later discussion of ChatGPT's ideas about Jesus' parable of the wedding feast, its complete failure to understand integer fractions, its successful answer to a trick question about Franklin Roosevelt, which it unfortunately recanted when I tried to compliment its success, and its baffling refusal to compare any fictional character with Benito Mussolini, or even to admit that it was possible to compare historical figures with fictional ones. In the end it got so wedged that it claimed:

ChatGPT I am not able to provide comparisons or judgments about the personalities or actions of historical figures or fictional characters.

Me Earlier you described Scheherazade as resourceful and eloquent. Isn't that a judgment about the personality of a fictional character?

ChatGPT Yes, you are correct. In my previous response, I did describe Scheherazade as resourceful and eloquent, which could be considered a judgment about her personality. As a large language model…

Ucccch, whatever.


Addendum 20240519

Simon Tatham has pointed out out that the exchange between Simon and Tick is from Mona Lisa Overdrive, not Idoru.

by Mark Dominus (mjd@plover.com) at May 13, 2024 11:58 PM

May 12, 2024

Mark Jason Dominus

It's an age of marvels

As I walk around Philadelphia I often converse with Benjamin Franklin, to see what he thinks about how things have changed since 1790. Sometimes he's astounded, other times less so. The things that astound Franklin aren't always what you might think at first. Electric streetlamps are a superb invention, and while I think Franklin would be very pleased to see them, I don't think he would be surprised. Better street lighting was something everyone wanted in Franklin's time, and this was something very much on Franklin's mind. It was certainly clear that electricity could be turned into light. Franklin could have and might have thought up the basic mechanism of an incandescent bulb himself, although he wouldn't have been able to make one.

The Internet? Well, again yes, but no. The complicated engineering details are complicated engineering, but again the basic idea is easily within the reach of the 18th century and is not all that astounding. They hadn't figured out Oersted's law yet, which was crucial, but they certainly knew that you could do something at one end of a long wire and it would have an effect at the other end, and had an idea that that might be a way to send messages from one place to another. Wikipedia says that as early as 1753 people were thinking that an electric signal could deflect a ping-pong ball at the receiving end. It might have worked! If you look into the history of transatlantic telegraph cables you will learn that the earliest methods were almost as clunky.

Wikipedia itself is more impressive. The universal encyclopedia has long been a dream, and now we have one. It's not always reliable, but you know what? Not all of anything is reliable.

An obvious winner, something sure to blow Franklin's mind is “yeah, we've sent people to the Moon to see what it was like, they left scientific instruments there and then they came back with rocks and stuff.” But that's no everyday thing, it blew everyone's mind when it happened and it still does. Some things I tell Franklin make him goggle and say “We did what?” and I shrug modestly and say yeah, it's pretty impressive, isn't it. The Moon thing makes me goggle right back. The Onion nailed it.

The really interesting stuff is the everyday stuff that makes Franklin goggle. CAT scans, for example. Ordinary endoscopy will interest and perhaps impress Franklin, but it won't boggle his mind. (“Yeah, the doctor sticks a tube up your butt with an electric light so they can see if your bowel is healthy.” Franklin nods right along.) X-rays are more impressive. (I wrote a while back about how long it took dentists to start adopting X-ray technology: about two weeks.) But CAT scans are mind-boggling. Oh yeah, we send invisible rays at you from all directions, and measure how much each one was attenuated from passing through your body, and then infer from that exactly what must be inside and how it is all arranged. We do what? And that's without getting into any of the details of whether this is done by positron emission or nuclear magnetic resonance (whatever those are, I have no idea) or something else equally incomprehensible. Apparently there really is something to this quantum physics nonsense.

So far though the most Franklin-astounding thing I've found has been GPS. The explanation starts with “well, first we put 32 artificial satellites in orbit around the Earth…”, which is already astounding, and can derail the conversation all by itself. But it just goes on from there getting more and more astounding:

“…and each one has a clock on board, accurate to within 40 nanoseconds…”

“…and can communicate the exact time wirelessly to the entire half of the Earth that it can see…”

“… and because the GPS device also has a perfect clock, it can compute how far it is from the satellite by comparing the two times and multiplying by the speed of light…”

“… and because the satellite also tells the GPS device exactly where it is, the device can determine that it lies on the surface of a sphere with the satellite at the center, so with messages from three or four satellites the device can compute its exact location, up to the error in the clocks and other measurements…”

“…and it fits in my pocket.”

And that's not even getting into the hair-raising complications introduced by general relativity. “It's a bit fiddly because time isn't passing at the same rate for the device as it is for the satellites, but we were able to work it out.” What. The. Fuck.

Of course not all marvels are good ones. I sometimes explain to Franklin that we have gotten so good at fishing — too good — that we are in real danger of fishing out the oceans. A marvel, nevertheless.

A past what-the-fuck was that we know exactly how many cells there are (959) in a particular little worm, C. elegans, and how each of those cells arises from the division of previous cells, as the worm grows from a fertilized egg, and we know what each cell does and how they are connected, and we know that 302 of those cells are nerve cells, and how the nerve cells are connected together. (There are 6,720 connections.) The big science news on Friday was that for the first time we have done this for an insect brain. It was the drosophila larva, and it has 3016 neurons and 548,000 synapses.

Today I was reading somewhere about how most meteorites are asteroidal, but some are from the Moon and a few are from Mars. I wondered “how do we know that they are from Mars?” but then I couldn't understand the explanation. Someday maybe.

And by the way, there are only 277 known Martian meteorites. So today's what-the-fuck is: “Yeah, we looked at all the rocks we could find all over the Earth and we noticed a couple hundred we found lying around various places looked funny and we figured out they must have come from Mars. And when. And how long they were on Mars before that.”

Obviously, It's amazing that we know enough about Mars to be able to say that these rocks are like the ones on Mars. (“Yeah, we sent some devices there to look around and send back messages about what it was like.”) But to me, the deeper and more amazing thing is, from looking at billions of rocks, we have learned so much about what rocks are like that we can pick out, from these billions, a couple of hundred that came to the Earth not merely from elsewhere but specifically from Mars.

What. The. Fuck.

Addendum 20240513

I left out one of the most important examples! Even more stunning than GPS. When I'm going into the supermarket, I always warn Franklin “Okay, brace yourself. This is really going to blow your mind.”

Addendum 20240514

Carl Witty points out that the GPS receiver does not have a perfect clock. The actual answer is more interesting. Instead of using three satellites and a known time to locate itself in space, as I said, the system uses four satellites to locate itself in spacetime.

Addendum 20240517

Another great example: I can have a hot shower, any time I want, just by turning a knob. I don't have to draw the water, I don't have to heat it over the fire. It just arrives effortlessly to the the bathroom on the third floor of my house.

And in the winter, the bathroom is heated.

One unimaginable luxury piled on another. Franklin is just blown away. How does it work?

Well, the entire city is covered with a buried network of pipes that carry flammable gas to every building. And there in my cellar an unattended, smokeless gas fire ensures that there is a tank with gallons of hot water ready for use at any moment. And it is delivered invisbly throughout my house by hidden pipes.

Just the amount of metal needed to make the pipes in my house is unthinkable to Franklin. And how long would it have taken for a blacksmith to draw them by hand?

by Mark Dominus (mjd@plover.com) at May 12, 2024 06:27 PM

May 10, 2024

GHC Developer Blog

GHC 9.10.1 is now available

GHC 9.10.1 is now available

bgamari - 2024-05-10

The GHC developers are very pleased to announce the release of GHC 9.10.1. Binary distributions, source distributions, and documentation are available at downloads.haskell.org and via GHCup.

GHC 9.10 brings a number of new features and improvements, including:

  • The introduction of the GHC2024 language edition, building upon GHC2021 with the addition of a number of widely-used extensions.

  • Partial implementation of the GHC Proposal #281, allowing visible quantification to be used in the types of terms.

  • Extension of LinearTypes to allow linear let and where bindings

  • The implementation of the exception backtrace proposal, allowing the annotation of exceptions with backtraces, as well as other user-defined context

  • Further improvements in the info table provenance mechanism, reducing code size to allow IPE information to be enabled more widely

  • Javascript FFI support in the WebAssembly backend

  • Improvements in the fragmentation characteristics of the low-latency non-moving garbage collector.

  • … and many more

A full accounting of changes can be found in the release notes. As always, GHC’s release status, including planned future releases, can be found on the GHC Wiki status.

We would like to thank GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

by ghc-devs at May 10, 2024 12:00 AM

May 09, 2024

Philip Wadler

Cabaret of Dangerous Ideas

I'll be appearing at the Fringe in the Cabaret of Dangerous Ideas, 12.20-13.20 Monday 5 August and 12.20-13.20 Saturday 17 August, at Stand 5. (The 5 August show is joint with Matthew Knight of the National Museums of Scotland.)

Here's the brief summary:

Chatbots like ChatGPT and Google's Gemini dominate the news. But the answers they give are, literally, bullshit. Historically, artificial intelligence has two strands. One is machine learning, which powers ChatGPT and art-bots like Midjourney, and which threatens to steal the work of writers and artists and put some of us out of work. The other is the 2,000-year-old discipline of logic. Professor Philip Wadler (The University of Edinburgh) takes you on a tour of the risks and promises of these two strands, and explores how they may work better together.
I'm looking forward to the audience interaction. Everyone should laugh and learn something. Do come!

by Philip Wadler (noreply@blogger.com) at May 09, 2024 04:09 PM

May 08, 2024

Gabriella Gonzalez

All error messages are necessarily bad to some degree

All error messages are necessarily bad to some degree

This is something I feel like enough people don’t appreciate. One of the ways I like to explain this is by this old tweet of mine:

The evolution of an error message:

  • No error message
  • A one-line message
  • “Expected: … / Actual: …”
  • “Here’s what went wrong: …”
  • “Here’s what you should do: …”
  • I automated away what you should do
  • The invalid state is no longer representable

One of the common gripes I will hear about error messages is that they don’t tell the user what to do, but if you stop to think about it: if the error message knew exactly what you were supposed to do instead then your tool could just fix it for you (by automatically doing the right thing instead).

But wait!”, you might say, “sometimes an error message can’t automatically fix the problem for you because there’s not necessarily a right or obvious way to fix the problem or the user’s intent is not clear.” Yes, exactly, which brings us back to the original point:

Error messages are necessarily bad because they cannot anticipate what you should have done instead. If an error message could read your mind then they’d eventually evolve into something better than an error message. This creates a selection bias where the only remaining error messages are the ones that can’t read your mind.

by Gabriella Gonzalez (noreply@blogger.com) at May 08, 2024 03:06 PM

May 04, 2024

Magnus Therning

Orderless completion in lsp-mode

If you, like me, are using corfu to get in-buffer completion and extend it with orderless to make it even more powerful, you might have noticed that you lose the orderless style as soon as you enter lsp-mode.

My setup of orderless looks like this

(use-package orderless
  :custom
  (orderless-matching-styles '(orderless-literal orderless-regexp orderless-flex))
  (completion-styles '(orderless partial-completion basic))
  (completion-category-defaults nil)
  (completion-category-overrides '((file (styles partial-completion)))))

which basically turns on orderless style for all things except when completing filenames.

It turns out that lsp-mode messes around with completion-category-defaults and when entering lsp-mode this code here adds a setting for 'lsp-capf. Unfortunately there seems to be no way to prevent lsp-mode from doing this so the only option is to fix it up afterwards. Luckily there's a hook for running code after the completion for lsp-mode is set up, lsp-completion-mode-hook. Adding the following function to it makes sure I now get to enjoy orderless also when writing code.

(lambda ()
  (setq-local completion-category-defaults
              (assoc-delete-all 'lsp-capf completion-category-defaults)))

May 04, 2024 04:49 AM

May 02, 2024

Haskell Interlude

48: José Nuno Oliveira

In this episode, Andres Löh and Matthías Páll Gissurarson interview José Nuno Oliveira, who has been teaching Haskell for 30 years. José talks about how Haskell is the perfect language to introduce programming to all sorts of audiences, why it is important to start with Haskell, and how the programmers of the future have been learning Haskell for several years already!

by Haskell Podcast at May 02, 2024 10:00 PM

Tweag I/O

The right words in the right place

tl;dr You may not believe it, but Nix documentation is getting better. Nixpkgs and NixOS still need more time.

Table of contents

Overview

This is a retrospective of my and many other people’s work on documentation in the Nix ecosystem between October 2022 and March 2024. It serves as a showcase of what we achieved together, and gives an impression of what’s involved in improving the user experience in a complex software system. A lot has happened during that time, so this text is quite lengthy.

The details of this report will mainly be interesting to power users and active contributors, or people working on software projects that are in a similar situation as Nix. For everyone else, I’d summarise the effort so far as “success in slow motion that indicates compounding effects”. Much of it was made possible through ongoing sponsorship from Antithesis, Tweag, and many individuals, along with the incredible commitment of numerous volunteers.

  • nix.dev is now the official documentation hub for the Nix ecosystem, designed to follow the Diátaxis documentation framework.
  • There are a number of new in-depth tutorials and practical guides, with more in the making.
  • The contribution process got smoother and is documented much better.
  • The Nix manual substantially improved in terms of structure, clarity, and detail. It can now be accessed by release version on nix.dev.
  • The Nixpkgs and NixOS manuals are now entirely written in Markdown, after two years of migrating away from XML.
  • More people than ever have contributed improvements to documentation.
  • The official NixOS Wiki has launched under wiki.nixos.org.

It may not seem like much, but these are big changes that posed enough of a challenge to arrive at. It’s still not easy or fast to learn all the things needed to wield the full power of Nix. However, there are reports of people grasping most of it within a couple of weeks on their own, which would be a significant improvement over what previously seemed to take months. And as more contributors chime in, things are continuously getting better.

The next steps will be expanding reference documentation, developing more suitable technical infrastructure, adding more tutorials, shaping an information architecture that connects the ecosystem… and simply figuring a lot of things out and writing them down.

Motivation

What originally compelled me to start working on improving Nix documentation was a vague feeling that there was something profound about Nix, something that was merely obscured by confusing explanations. It wasn’t just the fact that the Nix ecosystem permanently solves many problems in computing I believe should never have existed in the first place. It was the question: What underlying principle makes it possible for it to do so? Being in the right place at the right time often enough has allowed me to pursue a few hunches as part of my day job at Tweag for the past two years. And that time was needed.

First, it took me a while to fully grasp how beautifully simple Nix is conceptually:

  • The Nix store has a computation engine that operates on file system trees, disguised as a system for caching sandboxed execve calls.
  • The Nix language’s most distinguishing features are, not coincidentally, first-class facilities for dealing with file system paths and strings.
  • Nixpkgs is the world’s largest knowledge base for getting open source software to run.
  • NixOS offers a uniform user interface for configuring that open source software.

Second, after staring at it long enough, it turns out that the underlying powerful idea is programming itself! More precisely, disciplined programming:

  • The Nix store enforces referential integrity on the file system and constrains processes to act like pure functions, and it facilitates distributed, incremental computation. Read more on that in Taming Unix with functional programming.
  • Nixpkgs is written in the purely functional Nix language, and keeps scaling due to a substantial amount of automation.
  • The magic behind NixOS (and related tools, such as Home Manager) is the module system, which combines a rich type system with modular composition. Together with virtual machines, it enables systematic integration testing of a large part of the ecosystem’s artifacts.

From the outside, all of this seems terribly messy. It has grown organically over 20 years, and most of it is still only sparsely documented. Some parts are hard to understand; but that’s often because programming can be fundamentally hard. And it doesn’t help that things are just as messy on the inside. There are places where I’d say “the code doesn’t understand itself”.

The same things that make Nix hard to learn and use also make it hard to document and teach: There are many ways of doing certain things with Nix, and often none of them is clearly superior. A large part of that problem must eventually be solved at the implementation level, but that is subject to the Lisp Curse. In a volunteer-driven open source software community, many technical issues reduce to social issues. The real challenge is coordination!

That’s not exactly what I had originally subscribed to. But in retrospect, maybe it’s not surprising that, apart from finding the right words and the right place for them, the path to better documentation is paved with debates, design documents, fundraisers, UX workshops, governance meetings — that is, listening and talking to lots of people.

Making the ecosystem more approachable for a mass audience may as well end up to be teaching “disciplined software development with Nix”. But even that will require us, as a community-of-practice, to agree on (rather than just find) answers to many open questions. Until then, this report is meant to be a panorama of what that process has entailed so far, from my perspective. I hope that it will encourage you to try for yourself and participate, or otherwise help out.

Statistics

Before getting into a lengthy narrative, let’s look at recorded documentation activities of the past two years. tl;dr: numbers go up, this is good.

The following three charts show the number of documentation-related pull requests merged in the three repositories the team has worked on: Nix, Nixpkgs, and nix.dev. The charts have the same vertical scale and each bar stands for a quarter of a year since the beginning of 2022. Documentation work on Nix is underrepresented prior to Q3, since we only then began systematically labeling those pull requests. For nix.dev all pull requests are counted. The most recent period accounts for at most half of the quarter, because this is when I compiled the data.

Notably, Nixpkgs documentation activities have grown roughly with (and possibly even slower than) the overall number of users and contributors. This sample shows all pull requests automatically labeled as being related to documentation, which often comes with package updates and are often inconsequential in practice.

In contrast, the frequency of changes in nix.dev and the Nix manual greatly increased since the documentation team was founded.

255f52a1 612d 4224 b3b9 865c089371dc

3d3dc368 cf07 4873 94a0 fd788ce52b37

84480102 9633 48d2 9551 d00a2dff3751

The next three charts show the number of GitHub issues related to documentation that were opened or closed in a given quarter, again for Nix, Nixpkgs, and nix.dev. These charts also have the same scale. Here, one can observe that Nixpkgs documentation activities, while more pronounced overall due to the greater number of people involved, are significantly driven by the team. There is also some up-and-down due to alternating between more exploratory and more implementation-oriented periods.

eb0e2611 35aa 42d1 ba77 98c18faca5e2

3c0136b6 82a3 441e 9a5a 0a2d7dadd237

b618968c c4a3 4ef0 9058 7fec0d662c43

The final three charts show the number of new contributors and those who made repeated contributions related to documentation, again per quarter and for Nix, Nixpkgs, and nix.dev. These charts are not to scale with each other because including Nixpkgs would have made the others unreadable. This is the most notable change, and in my opinion the most important success: We now have around 10 people working consistently on Nix reference documentation and introductory materials — more than ever!

contributors nix

contributors nixpkgs

contributors nix dev

And now for the full story, which is long because we have made and learned so many things.

Retrospective

In 2022, I started out with the hypothesis that there needs to be a comprehensive document that one could read cover to cover to understand the Nix ecosystem. With Nix inventor Eelco Dolstra’s blessing and Tweag’s funding I started an effort to write what we called The Nix Book.

It turned out not to be that simple at all. The ecosystem is too large, the spectrum of use cases too wide, and the details too messed up to just sit down and blurt out everything one knows, and then wait for others to fix the typos. In fact, there have been multiple such attempts by different people.

A more systematic approach was needed. In addition to an initial literature survey, I also ran a small usability study and evaluated the 2022 community polls. During that time, the documentation team was founded, and became the forum for developing and refining goals for documentation in the Nix ecosystem, and working towards their achievement.

As of today, the documentation team meeting was held more than a hundred times. I’m convinced: we’re delivering. You can feel the change in pace and atmosphere, and the numbers support it.

Over the period covered by this report, we formulated multiple sets of goals. They reflect our evolving understanding of the problem space and shifts in priorities.

Because all other attempts to make a comprehensive tractable review turned out to be impractical, I will first list the goals in the order they were published, for reference. Then, based on the statements reworded and regrouped to avoid overlap, I will present results for each category separately.

From the proposal for The Nix Book April 2022
  • Write a book actually explaining Nix and its surrounding ecosystem at a high level of abstraction
  • Overhaul the Nix manual to make it a focused technical reference
  • Improve discoverability of existing learning material
From the documentation team announcement August 2022
  • Ease Nix learning, increase onboarding success and user retention
  • Improve organisation of Nix knowledge
  • Lead, guide, and support community efforts
From my NixCon talk Flattening the learning curve October 2022
  • Reference documentation should be complete, correct, and easy to update
  • Guides should cover all major use cases
  • Tutorials should teach all key skills necessary to solve problems on one’s own, and require minimal effort from learners
  • Documentation should be versioned in lockstep with the code being documented
  • Each area should have a dedicated maintainer; someone who knows the whole thing by heart and knows where to put what
From the call for maintainers March 2023
  • Ease onboarding for users of Nix tools and contributors to their documentation:
    • Improve discoverability of relevant documentation for major use cases
    • Find dedicated owners for each part of the documentation
    • Markdown everywhere
  • Create a coherent vision for documentation in the Nix ecosystem, and derive an implementation strategy and roadmap, guided by the Diátaxis framework for technical documentation:
    1. Reference: Design an information architecture for reference documentation
    2. Tutorials: Draft a complete onboarding and learning journey
    3. Guides: Define a curation model for how-to guides
    4. Explanation: Devise a plan for developing a book on the intellectual history of the Nix ecosystem
From the Learning Journey project April 2023
  • Develop a curriculum draft
  • Categorise existing documentation materials
  • Break down tasks for the writing phase
  • Prepare a contributor workflow
  • Link or migrate existing documentation into a central location, as far as possible.
  • Prepare and publish a call for contributors for the writing phase

Documentation goals for the Nix ecosystem

I chose the following categories for evaluating documentation activities, because they emerge from the concrete goals recorded in the past. I ordered them taking into account their relative importance and the sequence of events; but mostly all of this is merely a narrative device to condense a lot of information.

Improve discoverability

Many of the most obvious issues with Nix-related documentation revolve around obstacles to finding relevant information. We made notable progress on improving discoverability, by more clearly separating the roles of different information sources, reworking navigation paths, and adding cross-references. But we also ran into deeper structural and technical problems, which we started to address from various angles.

Details (17 items)

Increase coverage of reference documentation

The Nix ecosystem is a large piece of software. The goal is to eventually capture all of its interfaces in reference documentation.

We mainly added much more information to the Nix reference manual, but have not yet started fully documenting the Nix language syntax. There were many incremental improvements to Nixpkgs reference manual, and work is ongoing to further increase coverage. The people involved with the documentation team did not work on the NixOS manual, and the official NixOS Wiki has only recently launched.

Details (14 items)

Ensure correctness of reference documentation

Merely having interface documentation is not enough, the information also has to be correct and up to date.

Contributors corrected countless errors and omissions in all documentation resources. Work is ongoing to render all of the Nixpkgs library documentation from the source code. There are technical obstacles to increase automatic presentation of interface documentation in the Nix reference manual.

The biggest impact on correctness would be made by automatically testing examples as part of continuous integration, but there was no progress on that apart from exchanging ideas. We’ve been updating examples while working on other things, to make them more self-contained so they could be tested eventually.

Details (8 items)

Improve the contributor experience

I claim that one of the most important use cases of any community-driven open source project is contributing to that project. Much of our efforts revolved around easing contributions, in particular to documentation. We added and improved contributor guides for each component of the Nix ecosystem, and added automation to further reduce friction. Substantial parts of official documentation are now actively maintained, and the team holds regular meetings to review and merge pull requests. As a consequence, in the second half of 2023 we got much more work done than ever, and more people than ever joined and stayed to help.

Details (14 items)

Teach important skills

Part of the documentation team’s mission is to provide more and better learning materials. Thanks to many volunteers’ patience and endurance, the group produced roughly one tutorial every two months between end of 2022 and end of 2023. This reportedly improved the learning experience — especially where we took the time to build a solid foundation in reference documentation. That way, we have addressed a good chunk of fundamental use cases.

But there is still much ground to cover, even for what we consider essential skills and workflows in the Nix ecosystem. Some articles could be a lot shorter. Nonetheless, there are now more and much better tutorials than ever, all of which are actively maintained to incorporate feedback and keep up with best practices.

Details (9 items)

Explain concepts

The Nix ecosystem approaches many problems in a unique way, which requires introducing new terms and explaining why certain things are the way they are. While that was my original motivation to start working on documentation, explanations have shown to be less of an important tool for serving immediate needs. As a result, only a few things were worked on.

Details (3 items)

Reduce onboarding time

The primary mission of the documentation team is to reduce onboarding time for beginners. That plays into addressing the other challenges I have talked about, such as attracting more contributors. We figured that a sequence of lessons focused on widely applicable skills would be an effective resource for users to become self-sufficient in the ecosystem.

To that end, there was a coordinated project to design what we called the Learning Journey, which was supported by many people through a fundraiser on Open Collective. Despite some setbacks, we achieved the main goals and will continue with implementation by writing more tutorials for the curriculum.

Details (7 items)
  • With support of the NixOS Foundation, the documentation team applied for Google Season of Docs with the goal to develop a Nix curriculum called the Learning Journey.
  • We did not win the grant, but Ron Efroni (@ronef, Flox), in his capacity as NixOS Foundation board member, swiftly helped running a successful fundraiser to get the project started regardless. Within a few months, we collected more than 18 000 Euro from 57 supporters.
  • The team organised participation, processed applications, and were excited to have found the ideal candidates for the editorial lead and technical expert positions.
  • Unfortunately, after helping to kickstart the project, our editorial lead had to permanently leave for health reasons. Instead of searching for a replacement, we decided to shift focus, and reallocated part of the budget to fund work on technical infrastructure.
  • A dedicated Learning Journey volunteer working group formed around the project, led by Zach Mitchell (@zmitchell, Flox). Zach published monthly updates during that time. The remaining budget was used for expert reviews and support work, as originally intended, which helped keep up the momentum.
  • By the end of 2023, the Learning Journey group produced the key deliverable: a curriculum for learning Nix. This was accompanied by new and improved tutorials and guides, updated navigation structure on nix.dev, a well-tried contribution workflow, and a better overview of existing documentation resources.
  • We didn’t do another usability study as originally planned. That will be more sensible after filling up more of the curriculum scaffold with more tutorials.

Summary

All in all, we can claim success: The number of new and regular contributors to Nix documentation and nix.dev has increased significantly every quarter. The average turnaround time for documentation contributions seems to have decreased. We’ve come quite far with smoothing out the first couple of days and weeks of using vanilla Nix. Based on feedback and heuristics, such as questions asked on Discourse, I conclude that we have substantially reduced the time required for onboarding with Nix. I’m convinced that discoverability of learning materials and reference documentation has also improved a lot. Some of these results have yet to be validated with more evidence.

Thoughts on future work

I think that Nix documentation is on the right trajectory to eventually shake off its negative reputation and thus help unleash the power of Nix onto the world. I suggest to continue following the goals discussed in this report, and in order to reach them more quickly:

  • Finish things that have been started.
  • Put more emphasis on technical infrastructure and automation.
  • Focus on fewer topics, prioritise reference documentation.
  • Cultivate code ownership and empower more people to become maintainers.
  • Write more tutorials outlined in the Learning Journey.
  • Help establish a consistent information architecture across the Nix ecosystem.
  • Secure more funding to support volunteer efforts.
  • Systematically measure effectiveness of documentation efforts.

Acknowledgements

This whole endeavor has only been possible to sustain thanks to ongoing financial support. I’d like to thank the leadership of Antithesis, Tweag, Flox, and Determinate Systems for funding me, Silvan, Zach, and Luc to lead and assist the documentation team’s efforts over the past 18 months. Many thanks also to all our supporters on Open Collective for your confidence in our mission and your crucial role in speeding up progress.

An enormous amount of work has also been done by numerous volunteers, who did not just contribute to multiple articles and many small improvements, but were also involved in arriving at technical decisions and implementing them. Some of them stay around to incorporate corrections to their texts and code. Thank you so much for your indispensable contributions to our collective endeavor:

May 02, 2024 12:00 AM

May 01, 2024

Well-Typed.Com

The Haskell Unfolder Episode 24: generic (un)folds

Today, 2024-05-01, at 1830 UTC (11:30 am PDT, 2:30 pm EDT, 7:30 pm BST, 20:30 CEST, …) we are streaming the 24th episode of the Haskell Unfolder live on YouTube.

The Haskell Unfolder Episode 24: generic (un)folds

In our first anniversary episode, we are connecting back to the very beginning of the Haskell Unfolder and talk about unfolds and folds. But this time, not only on lists, but on a much wider class of datatypes, namely those that can be written as a fixed point of a functor.

About the Haskell Unfolder

The Haskell Unfolder is a YouTube series about all things Haskell hosted by Edsko de Vries and Andres Löh, with episodes appearing approximately every two weeks. All episodes are live-streamed, and we try to respond to audience questions. All episodes are also available as recordings afterwards.

We have a GitHub repository with code samples from the episodes.

And we have a public Google calendar (also available as ICal) listing the planned schedule.

by andres, edsko at May 01, 2024 12:00 AM

April 29, 2024

Mark Jason Dominus

Hawat! Hawat! Hawat! A million deaths are not enough for Hawat!

[ Content warning: Spoilers for Frank Herbert's novel Dune. Conversely none of this will make sense if you haven't read it. ]

Summary: Thufir Hawat is the real traitor. He set up Yueh to take the fall.

This blog post began when I wondered:

Hawat knows that Wellington Yueh has, or had a wife, Wanna. She isn't around. Hasn't he asked where she is?

In fact she is (or was) a prisoner of the Harkonnens and the key to Yueh's betrayal. If Hawat had asked the obvious question, he might have unraveled the whole plot.

But Hawat is a Mentat, and the Master of Assassins for a Great House. He doesn't make dumbass mistakes like forgetting to ask “what are the whereabouts of the long-absent wife of my boss's personal physician?”

The Harkonnens nearly succeed in killing Paul, by immuring an agent in the Atreides residence six weeks before Paul even moves in. Hawat is so humiliated by his failure to detect the agent hidden in the wall that he offers the Duke his resignation on the spot. This is not a guy who would have forgotten to investigate Yueh's family connections.

And that wall murder thing wasn't even the Harkonnens' real plan! It was just a distraction:

"We've arranged diversions at the Residency," Piter said. "There'll be an attempt on the life of the Atreides heir — an attempt which could succeed."

"Piter," the Baron rumbled, "you indicated —"

"I indicated accidents can happen," Piter said. "And the attempt must appear valid."

Piter de Vries was so sure that Hawat would find the agent in the wall, he was willing to risk spoiling everything just to try to distract Hawat from the real plan!

If Hawat was what he appeared to be, he would never have left open the question of Wanna's whereabouts. Where is she? Yueh claimed that she had been killed by the Harkonnens, and Jessica offers that as a reason that Yueh can be trusted.

But the Bene Gesserit have a saying: “Do not count a human dead until you've seen his body. And even then you can make a mistake.” The Mentats must have a similar saying. Wanna herself was Bene Gesserit, who are certainly human and notoriously difficult to kill. She was last known to be in the custody of the Harkonnens. Why didn't Hawat consider the possibility that Wanna might not be dead, but held hostage, perhaps to manipulate Duke Leto's physician and his heir's tutor — as in fact she was? Of course he did.

"Not to mention that his wife was a Bene Gesserit slain by the Harkonnens," Jessica said.

"So that’s what happened to her," Hawat said.

There's Hawat, pretending to be dumb.

Supposedly Hawat also trusted Yueh because he had received Imperial Conditioning, and as Piter says, “it's assumed that ultimate conditioning cannot be removed without killing the subject”. Hawat even says to Jessica: “He's conditioned by the High College. That I know for certain.”

Okay, and? Could it be that Thufir Hawat, Master of Assassins, didn't consider the possibility that the Imperial Conditioning could be broken or bent? Because Piter de Vries certainly did consider it, and he was correct. If Piter had plotted to subvert Imperial Conditioning to gain an advantage for his employer, surely Hawat would have considered the same.

Notice, also, what Hawat doesn't say to Jessica. He doesn't say that Yueh's Imperial Conditioning can be depended on, or that Yueh is trustworthy. Jessica does not have the gift of the full Truthsay, but it is safest to use the truth with her whenever possible. So Hawat misdirects Jessica by saying merely that he knows that Yueh has the Conditioning.

Yueh gave away many indications of his impending betrayal, which would have been apparent to Hawat. For example:

Paul read: […]
"Stop it!" Yueh barked.
Paul broke off, stared at him.
Yueh closed his eyes, fought to regain composure. […]
"Is something wrong?" Paul asked.
"I'm sorry," Yueh said. "That was … my … dead wife's favorite passage."

This is not subtle. Even Paul, partly trained, might well have detected Yueh's momentary hesitation before his lie about Wanna's death. Paul detects many more subtle signs in Yueh as well as in others:

"Will there be something on the Fremen?" Paul asked.

"The Fremen?" Yueh drummed his fingers on the table, caught Paul staring at the nervous motion, withdrew his hand.

Hawat the Mentat, trained for a lifetime in observing the minutiae of other people's behavior, and who saw Yueh daily, would surely have suspected something.

So, Hawat knew the Harkonnens’ plot: Wanna was their hostage, and they were hoping to subvert Yueh and turn him to treason. Hawat might already have known that the Imperial Conditioning was not a certain guarantee, but at the very least he could certainly see that the Harkonnens’ plan depended on subverting it. But he lets the betrayal go ahead. Why? What is Hawat's plan?

Look what he does after the attack on the Atreides. Is he killed in the attack, as so many others are? No, he survives and immediately runs off to work for House Harkonnen.

Hawat might have had difficulty finding a new job — “Say aren't you the Master of Assassins whose whole house was destroyed by their ancient enemies? Great, we'll be in touch if we need anyone fitting that description.” But Vladimir Harkonnen will be glad to have him, because he was planning to get rid of Piter and would soon need a new Mentat, as Hawat presumably knew or guessed. And also, the Baron would enjoy having someone around to remind him of his victory over the Atreides. The Baron loves gloating, as Hawat certainly knows.

Here's another question: Where did Yueh get the tooth with the poison gas? The one that somehow wasn't detected by the Baron's poison snooper? The one that conveniently took Piter out of the picture? We aren't told. But surely this wasn't the sort of thing was left lying around the Ducal Residence for anyone to find. It is, however, just the sort of thing that the Master of Assassins of a Great House might be able to procure.

However he thought he came by the poison in the tooth, Yueh probably never guessed that its ultimate source was Hawat, who could have arranged that it was available at the right time.

This is how I think it went down:

The Emperor announces that House Atreides will be taking over the Arrakis fief from House Harkonnen. Everyone, including Hawat, sees that this is a trap. Hawat also foresees that the trap is likely to work: the Duke is too weak and Paul too young to escape it. Hawat must choose a side. He picks the side he thinks will win: the Harkonnens. With his assistance, their victory will be all but assured. He just has to arrange to be in the right place when the dust settles.

Piter wants Hawat to think that Jessica will betray the Duke. Very well, Hawat will pretend to be fooled. He tells the Atreides nothing, and does his best to turn the suspicions of Halleck and the others toward Jessica.

At the same time he turns the Harkonnens' plot to his advantage. Seeing it coming, he can avoid dying in the massacre. He provides Yueh with the chance to strike at the Baron and his close advisors. If Piter dies in the poison gas attack, as he does, his position will be ready for Hawat to fill; if not the position was going to be open soon anyway. Either way the Baron or his successor would be only too happy to have a replacement at hand.

(Hawat would probably have preferred that the Baron also be killed by the tooth, so that he could go to work for the impatient and naïve Feyd-Rautha instead of the devious old Baron. But it doesn't quite go his way.)

Having successfully made Yueh his patsy and set himself up to join the employ of the new masters of Arrakis and the spice, Hawat has some loose ends to tie up. Gurney Halleck has survived, and Jessica may also have survived. (“Do not count a human dead until you've seen his body.”) But Hawat is ready for this. Right from the beginning he has been assisting Piter in throwing suspicion on Jessica, with the idea that it will tend to prevent survivors of the massacre from reuniting under her leadership or Paul's. If Hawat is fortunate Gurney will kill Jessica, or vice versa, wrapping up another loose end.

Where Thufir Hawat goes, death and deceit follow.

Addendum

Maybe I should have mentioned that I have not read any of the sequels to Dune, so perhaps this is authoritatively contradicted — or confirmed in detail — in one of the many following books. I wouldn't know.

Addendum 20240512

Elliot Evans points out that my theory really doesn't hold up. Hawat survives the assault because he is out of town when it happens (“Aha!” I said, “how convenient for him!”) but his thoughts about it, as reported by Herbert, seem to demolish my theory:

I underestimated what the Baron was willing to spend in attacking us, Hawat thought. I failed my Duke.

Then there was the matter of the traitor.

I will live long enough to see her strangled! he thought. I should’ve killed that Bene Gesserit witch when I had the chance. There was no doubt in his mind who had betrayed them — the Lady Jessica. She fitted all the facts available.

Mr. Herbert, I tried hard to give you an escape from this:

"So that’s what happened to her," Hawat said.

but you cut off your own avenue of escape.

by Mark Dominus (mjd@plover.com) at April 29, 2024 11:51 PM

April 28, 2024

Mark Jason Dominus

Rod R. Blagojevich will you please go now?

I'm strangely fascinated and often amused by crooked politicians, and Rod Blagojevich was one of the most amusing.

In 2007 Barack Obama, then a senator of Illinois, resigned his office to run for United States President. Under Illinois law, the governor of Illinois was responsible for appointing Obama's replacement until the next election was held. The governor at the time was Rod Blagojevich, and Blagojevich had a fine idea: he would sell the Senate seat to the highest bidder. Yes, really.

Zina Saunders did this wonderful painting of Blago and has kindly given me permission to share it with you.

Rod Blagojevich is depicted as a user-car salesman in the cheaply-produced television advertisement.  He is wearing a green sport jacket with wide lapels, white trousers, a shirt with a deep neckline, and a gold neck medallion.  He stands gleefully in front of a large banner that proclaims “SALE”, and is pointing both index fingers at the viewer. Behind him is an armchair with the seal of the United States Senate on the upholstery.

When the governor's innovation came to light, the Illinois state legislature ungratefully but nearly unanimously impeached him (the vote was 117–1) and removed him from office (59–0). He was later charged criminally, convicted, and sentenced to 168 months in federal prison for this and other schemes. He served about 8 years before Donald Trump, no doubt admiring the initiative of a fellow entrepreneur, commuted his sentence.

Blagojevich was in the news again recently. When the legislature gave him the boot they also permanently disqualified him from holding any state office. But Blagojevich felt that the people of Illinois had been deprived for too long of his wise counsel. He filed suit in Federal District Court, seeking not only vindication of his own civil rights, but for the sake of the good citizens of Illinois:

Preventing the Plaintiff from running for state or local public office outweighs any harm that could be caused by denying to the voters their right to vote for or against him in a free election.

Allowing voters decide who to vote for or not to vote for is not adverse to the public interest. It is in the public interest.

The Plaintiff is seeking a declaratory judgement rendering the State Senate's disqualifying provision as null and void because it violates the First Amendment rights of the voters of Illinois.

This kind of thing is why I can't help but be amused by crooked politicians. They're so joyful and so shameless, like innocent little children playing in a garden.

Blagojevich's lawsuit was never going to go anywhere, for so many reasons. Just the first three that come to mind:

  1. Federal courts don't have a say over Illinois' state affairs. They deal in federal law, not in matters of who is or isn't qualified to hold state office in Illinois.

  2. Blagojevich complained that his impeachment violated his Sixth Amendment right to Due Process. But the Sixth Amendment applies to criminal prosecutions and impeachments aren't criminal prosecutions.

  3. You can't sue to enforce someone else's civil rights. They have to bring the suit themselves. Suing on behalf of the people of a state is not a thing.

Well anyway, the judge, Steven  C. Seeger, was even less impressed than I was. Federal judges do not normally write “you are a stupid asshole, shut the fuck up,” in their opinions, and Judge Seeger did not either. But he did write:

He’s back.

and

[Blagojevich] adds that the “people’s right to vote is a fundamental right.” And by that, Blagojevich apparently means the fundamental right to vote for him.

and

The complaint is riddled with problems. If the problems are fish in a barrel, the complaint contains an entire school of tuna. It is a target-rich environment.

and

In its 205-year history, the Illinois General Assembly has impeached, convicted, and removed one public official: Blagojevich.

and

The impeachment and removal by the Illinois General Assembly is not the only barrier keeping Blagojevich off the ballot. Under Illinois law, a convicted felon cannot hold public office.

Federal judges don't get to write “sit down and shut up”. But Judge Seeger came as close as I have ever seen when he quoted from Marvin K. Mooney Will you Please Go Now!:

“The time has come. The time has come. The time is now. Just Go. Go. GO! I don’t care how. You can go by foot. You can go by cow. Marvin K. Mooney, will you please go now!”

Cover of 'Marvin K. Mooney Wil YOu Please Go Now', (1972) by Dr. Seuss.  Marvin K. Money is a little bipedal dog-like creature in a purple one-piece jumpsuit.  He has a calm expression on his face, expressing his indifferent to the cover's request that he Please Go Now.

Addendum 20240508

I just noticed that the judge, Steven C. Seeger, has appeared here before, also for having said something that maybe federal judges shouldn't say.

by Mark Dominus (mjd@plover.com) at April 28, 2024 04:12 PM

April 27, 2024

GHC Developer Blog

GHC 9.10.1-rc1 is now available

GHC 9.10.1-rc1 is now available

bgamari - 2024-04-27

The GHC developers are very pleased to announce the availability of the release candidate for GHC 9.10.1. Binary distributions, source distributions, and documentation are available at downloads.haskell.org and via GHCup.

GHC 9.10 brings a number of new features and improvements, including:

  • The introduction of the GHC2024 language edition, building upon GHC2021 with the addition of a number of widely-used extensions.

  • Partial implementation of the GHC Proposal #281, allowing visible quantification to be used in the types of terms.

  • Extension of LinearTypes to allow linear let and where bindings

  • The implementation of the exception backtrace proposal, allowing the annotation of exceptions with backtraces, as well as other user-defined context

  • Further improvements in the info table provenance mechanism, reducing code size to allow IPE information to be enabled more widely

  • Javascript FFI support in the WebAssembly backend

  • Improvements in the fragmentation characteristics of the low-latency non-moving garbage collector.

  • … and many more

A full accounting of changes can be found in the release notes. As always, GHC’s release status, including planned future releases, can be found on the GHC Wiki status.

This is the penultimate prerelease leading to 9.10.1. In two weeks we plan to publish a release candidate, followed, if all things go well, by the final release a week later.

We would like to thank GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

by ghc-devs at April 27, 2024 12:00 AM

April 25, 2024

Tweag I/O

Re-implementing the Nix protocol in Rust

The Nix daemon uses a custom binary protocol — the nix daemon protocol — to communicate with just about everything. When you run nix build on your machine, the Nix binary opens up a Unix socket to the Nix daemon and talks to it using the Nix protocol1. When you administer a Nix server remotely using nix build --store ssh-ng://example.com [...], the Nix binary opens up an SSH connection to a remote machine and tunnels the Nix protocol over SSH. When you use remote builders to speed up your Nix builds, the local and remote Nix daemons speak the Nix protocol to one another.

Despite its importance in the Nix world, the Nix protocol has no specification or reference documentation. Besides the original implementation in the Nix project itself, the hnix-store project contains a re-implementation of the client end of the protocol. The gorgon project contains a partial re-implementation of the protocol in Rust, but we didn’t know about it when we started. We do not know of any other implementations. (The Tvix project created its own gRPC-based protocol instead of re-implementing a Nix-compatible one.)

So we re-implemented the Nix protocol, in Rust. We started it mainly as a learning exercise, but we’re hoping to do some useful things along the way:

  • Document and demystify the protocol. (That’s why we wrote this blog post! 👋)
  • Enable new kinds of debugging and observability. (We tested our implementation with a little Nix proxy that transparently forwards the Nix protocol while also writing a log.)
  • Empower other third-party Nix clients and servers. (We wrote an experimental tool that acts as a Nix remote builder, but proxies the actual build over the Bazel Remote Execution protocol.)

Unlike the hnix-store re-implementation, we’ve implemented both ends of the protocol. This was really helpful for testing, because it allowed our debugging proxy to verify that a serialization/deserialization round-trip gave us something byte-for-byte identical to the original. And thanks to Rust’s procedural macros and the serde crate, our implementation is declarative, meaning that it also serves as concise documentation of the protocol.

Structure of the Nix protocol

A Nix communication starts with the exchange of a few magic bytes, followed by some version negotiation. Both the client and server maintain compatibility with older versions of the protocol, and they always agree to speak the newest version supported by both.

The main protocol loop is initiated by the client, which sends a “worker op� consisting of an opcode and some data. The server gets to work on carrying out the requested operation. While it does so, it enters a “stderr streaming� mode in which it sends a stream of logging or tracing messages back to the client (which is how Nix’s progress messages make their way to your terminal when you run a nix build). The stream of stderr messages is terminated by a special STDERR_LAST message. After that, the server sends the operation’s result back to the client (if there is one), and waits for the next worker op to come along.

The Nix wire format

Nix’s wire format starts out simple. It has two basic types:

  • unsigned 64-bit integers, encoded in little-endian order; and
  • byte buffers, written as a length (a 64-bit integer) followed by the bytes in the buffer. If the length of the buffer is not a multiple of 8, it is zero-padded to a multiple of 8 bytes. Strings on the wire are just byte buffers, with no specific encoding.

Compound types are built up in terms of these two pieces:

  • Variable-length collections like lists, sets, or maps are represented by the number of elements they contain (as a 64-bit integer) followed by their contents.
  • Product types (i.e. structs) are represented by listing out their fields one-by-one.
  • Sum types (i.e. unions) are serialized with a tag followed by the contents.

For example, a “valid path info� consists of a deriver (a byte buffer), a hash (a byte buffer), a set of references (a sequence of byte buffers), a registration time (an integer), a nar size (an integer), a boolean (represented as an integer in the protocol), a set of signatures (a sequence of byte buffers), and finally a content address (a byte buffer). On the wire, it looks like:

3c 00 00 00 00 00 00 00 2f 6e 69 78 2f 73 74 6f 72 65 ... 2e 64 72 76 00 00 00 00  <- deriver
╰──── length (60) ────╯ ╰─── /nix/store/c3fh...-hello-2.12.1.drv ───╯ ╰ padding ╯

40 00 00 00 00 00 00 00 66 39 39 31 35 63 38 37 36 32 ... 30 33 38 32 39 30 38 66  <- hash
╰──── length (64) ────╯ ╰───────────────────── sha256 hash ─────────────────────╯

02 00 00 00 00 00 00 00                                                            â•®
╰── # elements (2) ───╯                                                            │
                                                                                   │
   39 00 00 00 00 00 00 00 2f 6e 69 78 ... 2d 32 2e 33 38 2d 32 37 00 00 .. 00 00  │
   ╰──── length (57) ────╯ ╰── /nix/store/9y8p...glibc-2.38-27 ──╯ ╰─ padding ──╯  │ references
                                                                                   │
   38 00 00 00 00 00 00 00 2f 6e 69 78 ... 2d 68 65 6c 6c 6f 2d 32 2e 31 32 2e 31  │
   ╰──── length (56) ────╯ ╰───────── /nix/store/zhl0...hello-2.12.1 ───────────╯  ╯

1c db e8 65 00 00 00 00 f8 74 03 00 00 00 00 00 00 00 00 00 00 00 00 00            <- numbers
╰ 2024-03-06 21:07:40 ╯ ╰─ 226552 (nar size) ─╯ ╰─────── false ───────╯

01 00 00 00 00 00 00 00                                                            â•®
╰── # elements (1) ───╯                                                            │
                                                                                   │ signatures
   6a 00 00 00 00 00 00 00 63 61 63 68 65 2e 6e 69 ... 51 3d 3d 00 00 00 00 00 00  │
   ╰──── length (106) ───╯ ╰─── cache.nixos.org-1:a7...oBQ== ────╯ ╰─ padding ──╯  ╯

00 00 00 00 00 00 00 00                                                            <- content address
╰──── length (0) ─────╯

This wire format is not self-describing: in order to read it, you need to know in advance which data-type you’re expecting. If you get confused or misaligned somehow, you’ll end up reading complete garbage. In my experience, this usually leads to reading a “length� field that isn’t actually a length, followed by an attempt to allocate exabytes of memory. For example, suppose we were trying to read the “valid path info� written above, but we were expecting it to be a “valid path info with path,� which is the same as a valid path info except that it has an extra path at the beginning. We’d misinterpret /nix/store/c3f-...-hello-2.12.1.drv as the path, we’d misinterpret the hash as the deriver, we’d misinterpret the number of references (2) as the number of bytes in the hash, and we’d misinterpret the length of the first reference as the hash’s data. Finally, we’d interpret /nix/sto as a 64-bit integer and promptly crash as we allocate space for more than <semantics>8×1018<annotation encoding="application/x-tex">8 \times 10^{18}</annotation></semantics>8×1018 references.

There’s one important exception to the main wire format: “framed data�. Some worker ops need to transfer source trees or build artifacts that are too large to comfortably fit in memory; these large chunks of data need to be handled differently than the rest of the protocol. Specifically, they’re transmitted as a sequence of length-delimited byte buffers, the idea being that you can read one buffer at a time, and stream it back out or write it to disk before reading the next one. Two features make this framed data unusual: the sequence of buffers are terminated by an empty buffer instead of being length-delimited like most of the protocol, and the individual buffers are not padded out to a multiple of 8 bytes.

Serde

Serde is the de-facto standard for serialization and deserialization in Rust. It defines an interface between serialization formats (like JSON, or the Nix wire protocol) on the one hand and serializable data types on the other. This divides our work into two parts: first, we implement the serialization format, by specifying the correspondence between Serde’s data model and the Nix wire format we described above. Then we describe how the Nix protocol’s messages map to the Serde data model.

The best part about using Serde for this task is that the second step becomes straightforward and completely declarative. For example, the AddToStore worker op is implemented like

#[derive(serde::Deserialize, serde::Serialize)]
pub struct AddToStore {
    pub name: StorePath,
    pub cam_str: StorePath,
    pub refs: StorePathSet,
    pub repair: bool,
    pub data: FramedData,
}

These few lines handle both serialization and deserialization of the AddToStore worker op, while ensuring that they remain in-sync.

Mismatches with the Serde data model

While Serde gives us some useful tools and shortcuts, it isn’t a perfect fit for our case. For a start, we don’t benefit much from one of Serde’s most important benefits: the decoupling between serialization formats and serializable data types. We’re interested in a specific serialization format (the Nix wire format) and a specific collection of data types (the ones used in the Nix protocol); we don’t gain much by being able to, say, serialize the Nix protocol to JSON.

The main disadvantage of using Serde is that we need to match the Nix protocol to Serde’s data model. Most things match fairly well; Serde has native support for integers, byte buffers, sequences, and structs. But there were a few mismatches that we had to work around:

  • Different kinds of sequences: Serde has native support for sequences, and it can support sequences that are either length-delimited or not. However, Serde does not make it easy to support length-delimited and non-length-delimited sequences in the same serialization format. And although most sequences in the Nix format are length-delimited, the sequence of chunks in a framed source are not. We hacked around this restriction by treating a framed source not as a sequence but as a tuple with <semantics>264<annotation encoding="application/x-tex">2^{64}</annotation></semantics>264 elements, relying on the fact that Serde doesn’t care if you terminate a tuple early.
  • The Serde data model is larger than the Nix protocol needs; for example, it supports floating point numbers, and integers of different sizes and signedness. Our Serde de/serializer raises an error at runtime if it encounters any of these data types. Our Nix protocol implementation avoids these forbidden data types, but the Serde abstraction between the serializer and the data types means that any mistakes will not be caught at compile time.
  • Sum types tagged with integers: Serde has native support for tagged unions, but it assumes that they’re tagged with either the variant name (i.e. a string) or the variant’s index within a list of all possible variants. The Nix protocol uses numeric tags, but we can’t just use the variant’s index: we need to specify specific tags for specific variants, to match the ones used by Nix. We solved this by using our own derive macro for tagged unions. Instead of using Serde’s native unions, we map a union to a Serde tuple consisting of a tag followed by its payload.

But with these mismatches resolved, our final definition of the Nix protocol is fully declarative and pretty straightforward:

#[derive(TaggedSerde)]
//       ^^ our custom procedural macro for unions tagged with integers
pub enum WorkerOp {
    #[tagged_serde = 1]
    //              ^^ this op has opcode 1
    IsValidPath(StorePath, Resp<bool>),
    //             ^^            ^^ the op's response type
    //             || the op's payload
    #[tagged_serde = 6]
    QueryReferrers(StorePath, Resp<StorePathSet>),
    #[tagged_serde = 7]
    AddToStore(AddToStore, Resp<ValidPathInfoWithPath>),
    #[tagged_serde = 9]
    BuildPaths(BuildPaths, Resp<u64>),
    #[tagged_serde = 10]
    EnsurePath(StorePath, Resp<u64>),
    #[tagged_serde = 11]
    AddTempRoot(StorePath, Resp<u64>),
    #[tagged_serde = 14]
    FindRoots((), Resp<FindRootsResponse>),
    // ... another dozen or so ops
}

Next steps

Our implementation is still a work in progress; most notably the API needs a lot of polish. It also only supports protocol version 34, meaning it cannot interact with old Nix implementations (before 2.8.0, which was released in 2022) and will lack support for features introduced in newer versions of the protocol.

Since in its current state our Nix protocol implementation can already do some useful things, we’ve made the crate available on crates.io. If you have a use-case that isn’t supported yet, let us know! We’re still trying to figure out what can be done with this.

In the meantime, now that we can handle the Nix remote protocol itself we’ve shifted our experimental hacking over to integrating with Bazel remote execution. We’re writing a program that presents itself as a Nix remote builder, but instead of executing the builds itself it sends them via the Bazel Remote Execution API to some other build infrastructure. And then when the build is done, our program sends it back to the requester as though it were just a normal Nix remote builder.

But that’s just our plan, and we think there must be more applications of this. If you could speak the Nix remote protocol, what would you do with it?


  1. Unless you’re running as a user that has read/write access to the nix store, in which case nix build will just modify the store directly instead of talking to the Nix daemon.↩

April 25, 2024 12:00 AM

April 24, 2024

Well-Typed.Com

Improvements to the ghc-debug terminal interface

ghc-debug is a debugging tool for performing precise heap analysis of Haskell programs (check out our previous post introducing it). While working on Eras Profiling, we took the opportunity to make some much needed improvements and quality of life fixes to both the ghc-debug library and the ghc-debug-brick terminal user interface.

To summarise,

  • ghc-debug now works seamlessly with profiled executables.
  • The ghc-debug-brick UI has been redesigned around a composable, filter based workflow.
  • Cost centers and other profiling metadata can now be inspected using both the library interface and the TUI.
  • More analysis modes have been integrated into the terminal interface such as the 2-level profile.

This post explores the changes and the new possibilities for inspecting the heap of Haskell processes that they enable. These changes are available by using the 0.6.0.0 version of ghc-debug-stub and ghc-debug-brick.

Recap: using ghc-debug

There are typically two processes involved when using ghc-debug on a live program. The first is the debuggee process, which is the process whose heap you want to inspect. The debuggee process is linked against the ghc-debug-stub package. The ghc-debug-stub package provides a wrapper function

withGhcDebug :: IO a -> IO a

that you wrap around your main function to enable the use of ghc-debug. This wrapper opens a unix socket and answers queries about the debuggee process’ heap, including transmitting various metadata about the debuggee, like the ghc version it was compiled with, and the actual bits that make up various objects on the heap.

The second is the debugger process, which queries the debuggee via the socket mechanism and decodes the responses to reconstruct a view of the debuggee’s Haskell heap. The most common debugger which people use is ghc-debug-brick, which provides a TUI for interacting with the debuggee process.

It is an important principle of ghc-debug that the debugger and debuggee don’t need to be compiled with the same version of GHC as each other. In other words, a debugger compiled once is flexible to work with many different debuggees. With our most recent changes debuggers now work seamlessly with profiled executables.

TUI improvements

Exploring Cost Center Stacks in the TUI

For debugging profiled executables, we added support for decoding profiling information in the ghc-debug library. Once decoding support was added, it’s easy to display the associated cost center stack information for each closure in the TUI, allowing you to interactively explore that chain of cost centers with source locations that lead to a particular closure being allocated. This gives you the same information as calling the GHC.Stack.whoCreated function on a closure, but for every closure on the heap! Additionally, ghc-debug-brick allows you to search for closures that have been allocated under a specific cost center.

Viewing the cost center stack for a specific list closure
Viewing the cost center stack for a specific list closure

As we already discussed in the eras profiling blog post, object addresses are coloured according to the era they were allocated in.

Eras profiling, each era is given a different colour
Eras profiling, each era is given a different colour

If other profiling modes like retainer profiling or biographical profiling are enabled, then the extra word tracked by those modes is used to mark used closures with a green line.

Biographical profiling, used closures are marked green
Biographical profiling, used closures are marked green

A filter based workflow

Typical ghc-debug-brick workflows would involve connecting to the client process or a snapshot and then running queries like searches to track down the objects that you are interested in. This took the form of various search commands available in the UI:

The commands now available in ghc-debug-brick
The commands now available in ghc-debug-brick

However, sometimes you would like to combine multiple search commands, in order to more precisely narrow down the exact objects you are interested in. Earlier you would have to do this by either writing custom queries with the ghc-debug Haskell API or modify the ghc-debug-brick code itself to support your custom queries.

Filters provide a composable workflow in order to perform more advanced queries. You can select a filter to apply from a list of possible filters, like the constructor name, closure size, era etc. and add it to the current filter stack to make custom search queries. Each filter can also be inverted.

The filters available in ghc-debug-brick
The filters available in ghc-debug-brick

We were motivated to add this feature after implementing support for eras profiling as it was often useful to combine existing queries with a filter by era. With these filters it’s easy to express your own domain specific queries, for example:

  • Find the Foo constructors which were allocated in a certain era.
  • Find all ARR_WORDS closures which are bigger than 1000 bytes.
  • Show me everything retained in this era, apart from ARR_WORDS and GRE constructors.

Here is a complete list of filters which are currently available:

Name Input Example Action
Address Closure Address 0x421c3d93c0 Find the closure with the specific address
Info Table Info table address 0x1664ad70 Find all closures with the specific info table
Constructor Name Constructor name Bin Find all closures with the given constructor name
Closure Name Name of closure sat_sHuJ_info Find all closures with the specific closure name
Era <era>/<start-era>-<end-era> 13 or 9-12 Find all closures allocated in the given era range
Cost centre ID A cost centre ID 107600 Finds all closures allocated (directly or indirectly) under this cost centre ID
Closure Size Int 1000 Find all closures larger than a certain size
Closure Type A closure type description ARR_WORDS Find all ARR_WORDS closures

All these queries are retainer queries which will not only show you the closures in question but also the retainer stack which explains why they are retained.

Improvements to profiling commands

ghc-debug-brick has long provided a profile command which performs a heap traversal and provides a summary like a single sample from a -hT profile. The result of this query is now displayed interactively in the terminal interface. For each entry, the left column in the header shows the type of closure in question, the total number of this closure type which are allocated, the number of bytes on the heap taken up by this closure, the maximum size of each of these closures and the average size of each allocated closure. The right column shows the same statistics, but taken over all closures in the current heap sample.

A 1-level profile, displayed in ghc-debug-brick
A 1-level profile, displayed in ghc-debug-brick

Each entry can be expanded, five sample points from each band are saved so you can inspect some closures which contributed to the size of the band. For example, here we expand the THUNK closure and can see a sample of 5 thunks which contribute to the 210,000 thunks which are live on this heap.

Expanding the THUNK entry in a 1-level profile
Expanding the THUNK entry in a 1-level profile

Support for the 2-level closure type profile has also been added to the TUI. The 2-level profile is more fine-grained than the 1-level profile as the profile key also contains the pointer arguments for the closure rather than just the closure itself. The key :[(,), :] means the list cons constructor, where the head argument is a 2-tuple, and the tail argument is another list cons.

A 2-level profile displayed in ghc-debug-brick
A 2-level profile displayed in ghc-debug-brick

For example, in the 2-level profile, lists of different types will appear as different bands. In the profile above you can see 4 different bands resulting from lists, of 4 different types. Thunks also normally appear separately as they are also segmented based on their different arguments. The sample feature also works for the 2-level profile so it’s straightforward to understand what exactly each band corresponds to in your program.

Other UI improvements

In addition to the new features discussed above, some other recent enhancements include:

  • Improved the performance of the main view when displaying a large number of rows. This noticeably reduces input lag while scrolling.
  • The search limit was hard-coded to 100 objects, which meant that only the first few results of a search would be visible in the UI. This limit is now configurable in the UI.
  • Additional analyses are now available in the TUI, such as finding duplicate ARR_WORDS closures, which is useful for identifying cases where programs end up storing many copies of the same bytestring.

Conclusion

We hope that the improvements to ghc-debug and ghc-debug-brick will aid the workflows of anyone looking to perform detailed inspections of the heap of their Haskell processes.

This work has been performed in collaboration with Mercury. Mercury have a long-term commitment to the scalability and robustness of the Haskell ecosystem and are supporting the development of memory profiling tools to aid with these goals.

Well-Typed are always interested in projects and looking for funding to improve GHC and other Haskell tools. Please contact info@well-typed.com if we might be able to work with you!

by matthew, zubin, hannes at April 24, 2024 12:00 AM

April 21, 2024

Oleg Grenrus

A note about coercions

Posted on 2024-04-21 by Oleg Grenrus

Safe coercions in GHC are a very powerful feature. However, they are not perfect; and already many years ago I was also thinking about how we could make them more expressive.

In particular such things like "higher-order roles" have been buzzing. For the record, I don't think Proposal #233 is great; but because that proposal is almost four years old, I don't remember why; nor I have tangible counter-proposal either.

So I try to recover my thoughts.

I like to build small prototypes; and I wanted to build a small language with zero-cost coercions.

The first approach, I present here, doesn't work.

While it allows model coercions, and very powerful ones, these coercions are not zero-cost as we will see. For language like GHC Haskell where being zero-cost is non-negotiable requirement, this simple approach doesn't work.

The small "formalisation" is in Agda file https://gist.github.com/phadej/5cf29d6120cd27eb3330bc1eb8a5cfcc

Syntax

We start by defining syntax. Our language is "simple": there are types

A, B = A -> B     -- function type, "arrow"

coercions

co = refl A        -- reflexive coercion
   | sym co        -- symmetric coercions
   | arr co₁ co₂   -- coercion of arrows built from codomain and domain
                   -- type coercions

and terms

f, t, s = x         -- variable
        | f t       -- application
        | λ x . t   -- lambda abstraction
        | t ▹ co    -- cast

Obviously we'd add more stuff (in particular, I'm interested in expanding coercion syntax), but these are enough to illustrate the problem.

Because the language is simple (i.e. not dependent), we can define typing rules and small step semantics independently.

Typing

There is nothing particularly surprising in typing rules.

We'll need a "well-typed coercion" rules too though, but these are also very straigh-forward

Coercion Typing:  Δ ⊢ co : A ≡ B

------------------
Δ ⊢ refl A : A ≡ A

Δ ⊢ co : A ≡ B
------------------
Δ ⊢ sym co : B ≡ A

Δ ⊢ co₁ : C ≡ A
Δ ⊢ co₂ : D ≡ B
-------------------------------------
Δ ⊢ arr co₁ co₂ : (C -> D) ≡ (A -> B)

Terms typing rules are using two contexts, for term and coercion variables (GHC has them in one, but that is unhygienic, there's a GHC issue about that). The rules for variables, applications and lambda abstractions are as usual, the only new is the typing of the cast:

Term Typing: Γ; Δ ⊢ t : A

Γ; Δ ⊢ t : A 
   Δ ⊢ co : A ≡ B
-------------------------
Γ; Δ ⊢ t ▹ co : B 

So far everything is good.

But when playing with coercions, it's important to specify the reduction rules too. Ultimately it would be great to show that we could erase coercions either before or after reduction, and in either way we'll get the same result. So let's try to specify some reduction rules.

Reduction rules

Probably the simplest approach to reduction rules is to try to inherit most reduction rules from the system without coercions; and consider coercions and casts as another "type" and "elimination form".

An elimination of refl would compute trivially:

t ▹ refl A ~~> t

This is good.

But what to do when cast's coercion is headed by arr?

t ▹ arr co₁ co₂ ~~> ???

One "easy" solution is to eta-expand t, and split the coercion:

t ▹ arr co₁ co₂ ~~> λ x . t (x ▹ sym co₁) ▹ co₂

We cast an argument before applying it to the function, and then cast the result. This way the reduction is type preserving.

But this approach is not zero-cost.

We could not erase coercions completely, we'll still need some indicator that there were an arrow coercion, so we'll remember to eta-expand:

t ▹ ??? ~~> λ x . t x

Conclusion

Treating coercions as another type constructor with cast operation being its elimination form may be a good first idea, but is not good enough. We won't be able to completely erase such coercions.

Another idea is to complicate the system a bit. We could "delay" coercion elimination until the result is scrutinised by another elimination form, e.g. in application case:

(t ▹ arr co₁ co₂) s ~~> t (s ▹ sym co₁) ▹ co₂ 

And that is the approach taken in Safe Zero-cost Coercions for Haskell, you'll need to look into JFP version of the paper, as that one has appendices.

(We do not have space to elaborate, but a key example is the use of nth in rule S_KPUSH, presented in the extended version of this paper.)

The rule S_Push looks some what like:

---------------------------------------------- S_Push
(t ▹ co) s ~~> t (s ▹ sym (nth₁ co)) ▹ nth₂ co

where we additionally have nth coercion constructor to decompose coercions.

Incidentally there was, technically is, a proposal to remove decomposition rule, but it's a wrong solution to the known problem. The problem and a proper solution was kind of already identified in the original paper

We could similarly imagine a lattice keyed by classes whose instance definitions are to be respected; with such a lattice, we could allow the coercion of Map Int v to Map Age v precisely when Int’s and Age’s Ord instances correspond.

The original paper also identified the need for higher-order roles. And also identified that

This means that Monad instances could be defined only for types that expect a representational parameter.

which I argue should be already required for Functor (and traverseBia hack with unlawful Mag would still work if GHC had unboxed representational coercions, i.e. GADTs with baked-in representational (not only nominal) coercions).

There also the mention of unidirectional Coercible, which people asked about later and recently:

Such uni-directional version of Coercible amounts to explicit inclusive subtyping and is more complicated than our current symmetric system.

It is fascinating that authors were able to predict the relevant future work so well. And I'm thankful that GHC got Coercible implemented even it was already known to not be perfect. It's useful nevertheless. But I'm sad that there haven't been any results of future work since.

April 21, 2024 12:00 AM

April 20, 2024

Magnus Therning

Update to Hackage revisions in Nix

A few days after I published Hackage revisions in Nix I got a comment from Wolfgang W that the next release of Nix will have a callHackageDirect with support for specifying revisions.

The code in PR #284490 makes callHackageDirect accept a rev argument. Like this:

haskellPackages.callHackageDirect {
  pkg = "openapi3";
  ver = "3.2.3";
  sha256 = "sha256-0F16o3oqOB5ri6KBdPFEFHB4dv1z+Pw6E5f1rwkqwi8=";
  rev = {
    revision = "4";
    sha256 = "sha256-a5C58iYrL7eAEHCzinICiJpbNTGwiOFFAYik28et7fI=";
  };
} { }

That's a lot better than using overrideCabal!

April 20, 2024 09:04 AM

April 18, 2024

Tweag I/O

Cloud Native Computing in 2024—feeling the pulse at Kubecon

Last year, at the end of winter, we wrote our impressions of the trends and evolution of infrastructure and configuration management after attending FOSDEM and CfgMgmtCamp. We’re at it again, but with Kubecon this year, the biggest cloud native computing conference.

If you’ve never heard of cloud native computing before, it has a number of definitions online, but the simplest one is that it’s mostly about Kubernetes.

Kubecon is a huge event with thousands of attendees. The conference spanned several levels of the main convention center in Paris, with a myriad of conference rooms and a whole floor for sponsor booths. FOSDEM already felt huge compared to academic conferences, but Kubecon is even bigger.

Although the program was filled with appealing talks, we ended up spending most of our time chatting with people and visiting booths, something you can’t do as easily online.

Nix for the win

The very first morning, as we were walking around and waiting for the caffeine to kick in, we immediately spotted a Nix logo on someone’s sweatshirt. No better way to start the day than to meet with fellow Nix users!

And Nix was in general a great entry point for conversations at this year’s Kubecon. We expected it to still be an outsider at an event about container-driven technology, but the problems that “containers as a default packaging unit” can’t solve were so present that Nix’s value proposition is attuned to what everyone had on their minds anyway. In other words, Nix is now known enough as to serve as a conversation starter: the company might not use it, but many people have heard about it and were very interested to hear insights from big contributors like Tweag, the Modus Create OSPO.

Cybersecurity

Many security products were represented. Securing cloud-native applications is indeed a difficult matter, as their many layers and components expose a large attack surface.

For one, the schedule included a security track, with a fair share of talks being about SBOMs tying back into the problem of containers that ship an opaque content without inventory. Nix, as a solution to this problem, was a great conversation starter here as well, especially for fellow Nixer Matthias, who can talk for hours about how Nix is the best (and maybe only) technology for automatically deriving complete SBOMs of a piece of software, in a trustworthy manner. Our own NLNet-funded project genealogos, which does exactly that, is recently getting a lot of interest.

Besides the application code and what goes in it, another focus was avoiding misconfiguration of the cloud infrastructure layer, of the Kubernetes cluster, and anything else going into container images. Many companies propose SaaS combinations of static linters scanning the configuration files directly with various policy rules, heuristics and dynamic monitoring of secure cloud native applications. Our configuration language Nickel was very relevant here: one of its raisons d’être is to provide efficient tools (types, contracts and a powerful LSP) to detect and fix misconfigurations as early as possible. We had cool conversations around writing custom security policies as Nickel contracts with the new LSP background contract checking (introduced in 1.5) reporting non-compliance live in the editor — in that light, contracts are basically a lightweight way to program an LSP.

Internal Developer Platforms (IDPs)

IDPs were a hot topic at Kubecon. Tweag’s mission, as the OSPO of a leading software development consultancy, is to improve developer experience across the software lifecycle, which makes IDPs a natural topic for us.

An IDP is a platform - usually a web interface in practice - which glues several developer tools and services together and acts as the central entry point for most developer workflows. It’s centered around the idea of self-service, not unlike the console of cloud providers, but configurable for your exact use case and open to integrate tools across ecosystem boundaries. We already emphasized this emerging new abstraction in last year’s post. Example use cases are routinely deploying new infrastructure with just a few clicks, rather than requiring to sync and to send messages back-and-forth to the DevOps team. IDPs don’t replace other tools but offer a unified interface, usually with customized presets, to interact with repositories, internal data and infrastructure.

Backstage, an open-source IDP developed by Spotify, had its own sub-conference at Kubecon. Several products are built on top of it as well: it’s not really a ready-to-use solution but rather the engine to build a custom IDP for your company, which leaves room for turnkey offers. We feel that such integrated, centralized and simple-to-use services may become a standard in the future: think of how much GitHub (or an equivalent) is a central part of our modern workflow, but also of how many things it frustratingly can’t do (in particular infrastructure).

Cloud & AI

Many Kubernetes-based MLOps companies propose services to make it easy to deploy and manage scalable AI models in the cloud.

In the other direction, with the advent of generative AI, there was no doubt that we would see AI-based products to ease the automation of infrastructure-related tasks. We attended a demo of a multi-agent system which integrates with e.g. Slack, where you can ask a bot to perform end-to-end tasks (which includes interacting with several systems, like deploying something to the cloud, editing a Jira ticket and pushing something to a GitHub repo) or ask high-level questions, such as “which AWS users don’t have MFA enabled”.

It’s hard to tell from a demo how solid this would be in a real production system. I also don’t know if I would trust an AI agent to perform tasks without proper validation from a human (though there is a mode where confirmation is required before applying changes). There are also security concerns around having those agents run somewhere with write access to your infrastructure.

Putting those important questions aside, the demo was still quite impressive. It makes sense to automate those small boring tasks which usually require you to manually interact with several different platforms and are often quite mechanical indeed.

OSPO

We attended a Birds-of-a-Feather session on Open Source Program Offices (OSPO). While the small number of participants was a bit disappointing (between 10 and 15, compared to the size of the conference), the small group discussions were still engrossing, and we were pleased to meet people from other OSPOs as well as engineers wanting to push for an OSPO in their own company.

The generally small size OSPOs (including from very large and influential tech companies) and their low maturity from a strategic point of view was surprising to us. Many OSPOs seem to be stuck in tactical concerns, managing license and IP issues that can occur when developers open up company-owned repos. In such a situation, all OSPO members are fully occupied by the large number of requests they get. But the most interesting questions: how to share benefits and costs by working efficiently with open source communities, how to provide strategic guidance and support, and how to gain visibility in communities of interest were only addressed by few. A general concern seemed to be generally a lack of understanding by upper management about the real strategic power that an OSPO can provide. From that perspective, Tweag, although a pink unicorn as a consulting OSPO, is quite far on the maturity curve with concrete strategical and technical firepower through technical groups, and our open-source portfolio (plus the projects that we contribute to but aren’t ours).

Concluding words

Kubecon was a great experience, and we’re looking forward to the next one. We are excited about the advent of Internal Developer Platforms and the concept of self-serving infrastructure, which are important aspects of developer experience.

On the technological side, the cloud-native world seems to be dominated by Kubernetes with Helm charts and YAML, and Docker, while the technologies we believe in and are actively developing are still outsiders in the space (of course they aren’t a full replacement for what currently exists, but they could fill many gaps). I’m thinking in particular about Nix (and more generally about declarative, hermetic and reproducible builds and deployments) and Nickel (better configuration languages and management tools). But, conversation after conversation, conference after conference, we’re seeing more and more interests in new paradigms, sometimes because those technologies are best equipped - by far - to solve problems that are on everyone’s radar (e.g. software traceability through SBOMs with Nix) thanks to their different approach.

April 18, 2024 12:00 AM

Oleg Grenrus

What makes a good compiler warning?

Posted on 2024-04-18 by Oleg Grenrus

Recently I came up with a criteria for a good warning to have in a compiler:

If compiler makes a choice, or has to deal with some complication, it may well tell about that.

That made me think about warnings I implemented into GHC over the years. They are fine.

Let us first understand the criteria better. It is better explained by an example which triggers few warnings:

foo :: Char
foo = let x = 'x' in
      let x = 'y' in x

First warning is -Wname-shadowing:

Shadow.hs:3:11: warning: [-Wname-shadowing]
    This binding for ‘x’ shadows the existing binding
      bound at Shadow.hs:2:11
  |
3 |       let x = 'y' in x
  |           ^

When resolving names (i.e. figuring out what textual identifiers refer to) compilers have a choice what to do with duplicate names. The usual choice is to pick the closest reference, shadowing others. But it's not the only choice, and not the only choice GHC does in similar-ish situations. e.g. module's top-level definition do not shadow imports; instead an ambiguous name error is reported. Also \ x x -> x is rejected (treated as a non-linear pattern), but \x -> \x -> x is accepted (two separate patterns, inner one shadows). So, in a way, -Wname-shadowing reminds us what GHC does.

Another warning in the example is -Wunused-binds:

Shadow.hs:2:11: warning: [-Wunused-local-binds]
    Defined but not used: ‘x’
  |
2 | foo = let x = 'x' in
  |           ^

This a kind of warning that compiler might figure out in the optimisation passes (I'm not sure if GHC always tracks usage, but IIRC GCC had some warnings triggered only when optimisations are on). When doing usage analysis, compiler may figure out that some bindings are unused, so it doesn't need to generate code for them. At the same time it may warn the user.

More examples

Let go through few of the numerous warnings GHC can emit.

-Woverflowed-literals causes a warning to be emitted if a literal will overflow. It's not strictly a compiler choice, but a choice nevertheless in base's fromInteger implementations. For most types 1 the fromInteger is a total function with rollover behavior: 300 :: Word8 is 44 :: Word8. It could been chosen to not be total too, and IMO that would been ok if fromInteger were used only for desugaring literals.

-Wderiving-defaults: Causes a warning when both DeriveAnyClass and GeneralizedNewtypeDeriving are enabled and no explicit deriving strategy is in use. This a great example of a choice compiler makes. I actually don't remember which method GHC picks then, so it's good that compiler reminds us that it is good idea to be explicit (using DerivingStrategies).

-Wincomplete-patterns warns about places where a pattern-match might fail at runtime. This a complication compiler has to deal with. Compiler needs to generate some code to make all pattern matches complete. An easy way would been to always implicitly default cases to all pattern matches, but that would have performance implications, so GHC checks pattern-match coverage, and as a side-product may report incomplete pattern matches (or -Winaccesible-code) 2.

-Wmissing-fields warns you whenever the construction of a labelled field constructor isn’t complete, missing initialisers for one or more fields. Here compiler needs to fill the missing fields with something, so it warns when it does.

-Worphans gets an honorary mention. Orphans cause so much incidental complexity inside the compiler, that I'd argue that -Worphans should be enabled by default (and not only in -Wall).

Bad warnings

-Wmissing-import-lists warns if you use an unqualified import declaration that does not explicitly list the entities brought into scope. I don't think that there are any complications or choices compiler needs to deal with, therefore I think this warning should been left for style checkers. (I very rarely have import lists for modules from the same package or even project; and this is mostly a style&convenience choice).

-Wprepositive-qualified-module is even more of an arbitrary style check. With -Wmissing-import-lists it is generally accepted that explicit import lists are better for compatibility (and for GHCs recompilation avoidance). Whether you place qualified before or after the module name is a style choice. I think this warning shouldn't exist in GHC. (For the opposite you'd need a style checker to warn if ImportQualifiedPost is enabled anywhere).

Note, while -Wtabs is also mostly a style issue, but the compiler has to make a choice how to deal with them. Whether to always convert tabs to 8 spaces, convert to next 8 spaces boundary, require indentation to be exactly the same spaces&tabs combination. All choices are sane (and I don't know which one GHC makes), so a warning to avoid tabs is justified.

Compatibility warnings

Compatibility warnings are usually good also according to my criteria. Often it is the case that there is an old and a new way of doing things. Old way is going to be removed, but before removing it, it is deprecated.

-Wsemigroup warned about Monoid instances without Semigroup instances. (A warning which you shouldn't be able to trigger with recent GHCs). Here we could not switch to new hierarchy immediately without breaking some code, but we could check whether the preconditions are met for awhile.

-Wtype-equality-out-of-scope is somewhat similar. For now, there is some compatibility code in GHC, and GHC warns when that fallback code path is triggered.

My warnings

One of the warning I added is -Wmissing-kind-signatures. For long time GHC didn't have a way to specify kind signatures until StandaloneKindSignatures were added in GHC-8.10. Without kind signatures GHC must infer kind of a data type or type family declaration. With kind signature it could just check against given kind (which is a technically a lot easier). So while the warning isn't actually implemented so, it could be triggered when GHC notices it needs to infer a kind of a definition. In the implementation the warning is raised after the type-checking phase, so the warning can include the inferred kind. However, we can argue that when inference fails, GHC could also mention that the kind signature was missing. Adding a kind signature often results in better kind errors (c.f. adding a type signature often results in a better type error when something is wrong).

The -Wmissing-poly-kind-signatures warning seems like a simple restriction of above, but it's not exactly true. There is another problem GHC deals with. When GHC infers a kind, there might be unsolved meta-kind variables left, and GHC has to do something to them. With PolyKinds extension on, GHC generalises the kind. For example when inferring a kind of Proxy as in

data Proxy a = Proxy

GHC infers that the kind is k -> Type for some k and with PolyKinds it generalises it to type Proxy :: forall {k}. k -> Type. Another option, which GHC also may do (and does when PolyKinds are not enabled) is to default kinds to Type, i.e. type Proxy :: Type -> Type. There is no warning for kind defaulting, but arguable there should be as defaulted kinds may be wrong. (Haskell98 and Haskell2010 don't have a way to specify kind signatures; that is clear design deficiency; which was first resolved by KindSignatures and finally more elegantly by StandaloneKindSignatures).

There is defaulting for type variables, and (in some cases) GHC warns about them. You probably have seen Defaulting the type variable ‘a0’ to type ‘Integer’ warnings caused by -Wtype-defaults. Adding -Wkind-defaults to GHC makes sense, even only for uniformity between (types of) terms and types; or arguably nowadays it is a sign that you should consider enabling PolyKinds in that module.

About errors

The warning criteria also made me think about the following: the error hints are by necessity imprecise. If compiler knew exactly how to fix an issue, maybe it should just fix it and instead only raise a warning.

GHC has few of such errors. For example when using a syntax guarded by an extension. It can be argued (and IIRC was recently argued in discussions around GHC language editions) that another design approach would be simply accept new syntax, but just warn about it. The current design approach where extensions are "feature flags" providing some forward and backward compatibility is also defendable.

Conversely, if there is a case where compiler kind-of-knows what the issue is, but the language is not powerful enough for compiler to fix the problem on its own, the only solution is to raise an error. Well, there is another: (find a way to) extend the language to be more expressive, so compiler could deal with the currently erroneous case. Easier said than done, but in my opinion worth trying.

An example of above would be -Wmissing-binds . Currently writing a type signature without a corresponding binding is a hard error. But compiler could as well fill it in with a dummy one, That would complement -Wmissing-methods and -Wmissing-fields. Similarly for types, a standalone kind signature tells the compiler already a lot about the type even without an actual definition: the rest of the module can treat it as an opaque type.

Another example is briefly mentioned making module-top-level definitions shadow imports. That would make adding new exports (e.g. to implicitly imported Prelude) less affecting. While we are on topic of names, GHC could also report early when imported modules have ambiguous definitions, e.g.

import qualified Data.Text.Lazy as Lazy
import qualified Data.ByteString.Lazy as Lazy

doesn't trigger any warnings. But if you try to use Lazy.unpack you get an ambiguous occurrence error. GHC already deals with the complications of ambiguous names, it could as well have an option to report them early.

Conclusion

If compiler makes a choice, or has to deal with some complication, it may well tell about that.

Seems like a good criteria for a good compiler warning. As far as I can tell most warnings in GHC pass it; but I found few "bad" ones too. And also identified at least one warning-worthy case GHC doesn't warn about.


  1. With -XNegativeLiterals and Natural, fromInteger may result in run-time error though, for example:

    <interactive>:6:1: warning: [-Woverflowed-literals]
        Literal -1000 is negative but Natural only supports positive numbers
    *** Exception: arithmetic underflow
    ↩︎
  2. Using [-fmax-pmcheck-models] we could almost turn off GHCs pattern-match coverage checker, which will make GHC consider (almost) all pattern matches as incomplete. So -Wincomplete-patterns is kind of an example of a warning which is powered by an "optional" analysis is GHC.↩︎

April 18, 2024 12:00 AM

April 17, 2024

Haskell Interlude

47: Avi Press

Avi Press is interviewed by Joachim Breitner and Andres Löh. Avi is the founder of Scarf, which uses Haskell to analyze how open source software is used. We’ll hear about the kind of shitstorm telemetry can cause, when correctness matters less than fearless refactoring and how that can lead to statically typed Stockholm syndrome.

by Haskell Podcast at April 17, 2024 12:00 PM

April 16, 2024

Chris Reade

PenroseKiteDart User Guide

Introduction

PenroseKiteDart is a Haskell package with tools to experiment with finite tilings of Penrose’s Kites and Darts. It uses the Haskell Diagrams package for drawing tilings. As well as providing drawing tools, this package introduces tile graphs (Tgraphs) for describing finite tilings. (I would like to thank Stephen Huggett for suggesting planar graphs as a way to reperesent the tilings).

This document summarises the design and use of the PenroseKiteDart package.

PenroseKiteDart package is now available on Hackage.

The source files are available on GitHub at https://github.com/chrisreade/PenroseKiteDart.

There is a small art gallery of examples created with PenroseKiteDart here.

Index

  1. About Penrose’s Kites and Darts
  2. Using the PenroseKiteDart Package (initial set up).
  3. Overview of Types and Operations
  4. Drawing in more detail
  5. Forcing in more detail
  6. Advanced Operations
  7. Other Reading

1. About Penrose’s Kites and Darts

The Tiles

In figure 1 we show a dart and a kite. All angles are multiples of 36^{\circ} (a tenth of a full turn). If the shorter edges are of length 1, then the longer edges are of length \phi, where \phi = (1+ \sqrt{5})/ 2 is the golden ratio.

Figure 1: The Dart and Kite Tiles
Figure 1: The Dart and Kite Tiles

Aperiodic Infinite Tilings

What is interesting about these tiles is:

It is possible to tile the entire plane with kites and darts in an aperiodic way.

Such a tiling is non-periodic and does not contain arbitrarily large periodic regions or patches.

The possibility of aperiodic tilings with kites and darts was discovered by Sir Roger Penrose in 1974. There are other shapes with this property, including a chiral aperiodic monotile discovered in 2023 by Smith, Myers, Kaplan, Goodman-Strauss. (See the Penrose Tiling Wikipedia page for the history of aperiodic tilings)

This package is entirely concerned with Penrose’s kite and dart tilings also known as P2 tilings.

In figure 2 we add a temporary green line marking purely to illustrate a rule for making legal tilings. The purpose of the rule is to exclude the possibility of periodic tilings.

If all tiles are marked as shown, then whenever tiles come together at a point, they must all be marked or must all be unmarked at that meeting point. So, for example, each long edge of a kite can be placed legally on only one of the two long edges of a dart. The kite wing vertex (which is marked) has to go next to the dart tip vertex (which is marked) and cannot go next to the dart wing vertex (which is unmarked) for a legal tiling.

Figure 2: Marked Dart and Kite
Figure 2: Marked Dart and Kite

Correct Tilings

Unfortunately, having a finite legal tiling is not enough to guarantee you can continue the tiling without getting stuck. Finite legal tilings which can be continued to cover the entire plane are called correct and the others (which are doomed to get stuck) are called incorrect. This means that decomposition and forcing (described later) become important tools for constructing correct finite tilings.

2. Using the PenroseKiteDart Package

You will need the Haskell Diagrams package (See Haskell Diagrams) as well as this package (PenroseKiteDart). When these are installed, you can produce diagrams with a Main.hs module. This should import a chosen backend for diagrams such as the default (SVG) along with Diagrams.Prelude.

    module Main (main) where
    
    import Diagrams.Backend.SVG.CmdLine
    import Diagrams.Prelude

For Penrose’s Kite and Dart tilings, you also need to import the PKD module and (optionally) the TgraphExamples module.

    import PKD
    import TgraphExamples

Then to ouput someExample figure

    fig::Diagram B
    fig = someExample

    main :: IO ()
    main = mainWith fig

Note that the token B is used in the diagrams package to represent the chosen backend for output. So a diagram has type Diagram B. In this case B is bound to SVG by the import of the SVG backend. When the compiled module is executed it will generate an SVG file. (See Haskell Diagrams for more details on producing diagrams and using alternative backends).

3. Overview of Types and Operations

Half-Tiles

In order to implement operations on tilings (decompose in particular), we work with half-tiles. These are illustrated in figure 3 and labelled RD (right dart), LD (left dart), LK (left kite), RK (right kite). The join edges where left and right halves come together are shown with dotted lines, leaving one short edge and one long edge on each half-tile (excluding the join edge). We have shown a red dot at the vertex we regard as the origin of each half-tile (the tip of a half-dart and the base of a half-kite).

Figure 3: Half-Tile pieces showing join edges (dashed) and origin vertices (red dots)
Figure 3: Half-Tile pieces showing join edges (dashed) and origin vertices (red dots)

The labels are actually data constructors introduced with type operator HalfTile which has an argument type (rep) to allow for more than one representation of the half-tiles.

    data HalfTile rep 
      = LD rep -- Left Dart
      | RD rep -- Right Dart
      | LK rep -- Left Kite
      | RK rep -- Right Kite
      deriving (Show,Eq)

Tgraphs

We introduce tile graphs (Tgraphs) which provide a simple planar graph representation for finite patches of tiles. For Tgraphs we first specialise HalfTile with a triple of vertices (positive integers) to make a TileFace such as RD(1,2,3), where the vertices go clockwise round the half-tile triangle starting with the origin.

    type TileFace  = HalfTile (Vertex,Vertex,Vertex)
    type Vertex    = Int  -- must be positive

The function

    makeTgraph :: [TileFace] -> Tgraph

then constructs a Tgraph from a TileFace list after checking the TileFaces satisfy certain properties (described below). We also have

    faces :: Tgraph -> [TileFace]

to retrieve the TileFace list from a Tgraph.

As an example, the fool (short for fool’s kite and also called an ace in the literature) consists of two kites and a dart (= 4 half-kites and 2 half-darts):

    fool :: Tgraph
    fool = makeTgraph [RD (1,2,3), LD (1,3,4)   -- right and left dart
                      ,LK (5,3,2), RK (5,2,7)   -- left and right kite
                      ,RK (5,4,3), LK (5,6,4)   -- right and left kite
                      ]

To produce a diagram, we simply draw the Tgraph

    foolFigure :: Diagram B
    foolFigure = draw fool

which will produce the diagram on the left in figure 4.

Alternatively,

    foolFigure :: Diagram B
    foolFigure = labelled drawj fool

will produce the diagram on the right in figure 4 (showing vertex labels and dashed join edges).

Figure 4: Diagram of fool without labels and join edges (left), and with (right)
Figure 4: Diagram of fool without labels and join edges (left), and with (right)

When any (non-empty) Tgraph is drawn, a default orientation and scale are chosen based on the lowest numbered join edge. This is aligned on the positive x-axis with length 1 (for darts) or length \phi (for kites).

Tgraph Properties

Tgraphs are actually implemented as

    newtype Tgraph = Tgraph [TileFace]
                     deriving (Show)

but the data constructor Tgraph is not exported to avoid accidentally by-passing checks for the required properties. The properties checked by makeTgraph ensure the Tgraph represents a legal tiling as a planar graph with positive vertex numbers, and that the collection of half-tile faces are both connected and have no crossing boundaries (see note below). Finally, there is a check to ensure two or more distinct vertex numbers are not used to represent the same vertex of the graph (a touching vertex check). An error is raised if there is a problem.

Note: If the TilFaces are faces of a planar graph there will also be exterior (untiled) regions, and in graph theory these would also be called faces of the graph. To avoid confusion, we will refer to these only as exterior regions, and unless otherwise stated, face will mean a TileFace. We can then define the boundary of a list of TileFaces as the edges of the exterior regions. There is a crossing boundary if the boundary crosses itself at a vertex. We exclude crossing boundaries from Tgraphs because they prevent us from calculating relative positions of tiles locally and create touching vertex problems.

For convenience, in addition to makeTgraph, we also have

    makeUncheckedTgraph :: [TileFace] -> Tgraph
    checkedTgraph   :: [TileFace] -> Tgraph

The first of these (performing no checks) is useful when you know the required properties hold. The second performs the same checks as makeTgraph except that it omits the touching vertex check. This could be used, for example, when making a Tgraph from a sub-collection of TileFaces of another Tgraph.

Main Tiling Operations

There are three key operations on finite tilings, namely

    decompose :: Tgraph -> Tgraph
    force     :: Tgraph -> Tgraph
    compose   :: Tgraph -> Tgraph

Decompose

Decomposition (also called deflation) works by splitting each half-tile into either 2 or 3 new (smaller scale) half-tiles, to produce a new tiling. The fact that this is possible, is used to establish the existence of infinite aperiodic tilings with kites and darts. Since our Tgraphs have abstracted away from scale, the result of decomposing a Tgraph is just another Tgraph. However if we wish to compare before and after with a drawing, the latter should be scaled by a factor 1/{\phi} = \phi - 1 times the scale of the former, to reflect the change in scale.

Figure 5: fool (left) and decompose fool (right)
Figure 5: fool (left) and decompose fool (right)

We can, of course, iterate decompose to produce an infinite list of finer and finer decompositions of a Tgraph

    decompositions :: Tgraph -> [Tgraph]
    decompositions = iterate decompose

Force

Force works by adding any TileFaces on the boundary edges of a Tgraph which are forced. That is, where there is only one legal choice of TileFace addition consistent with the seven possible vertex types. Such additions are continued until either (i) there are no more forced cases, in which case a final (forced) Tgraph is returned, or (ii) the process finds the tiling is stuck, in which case an error is raised indicating an incorrect tiling. [In the latter case, the argument to force must have been an incorrect tiling, because the forced additions cannot produce an incorrect tiling starting from a correct tiling.]

An example is shown in figure 6. When forced, the Tgraph on the left produces the result on the right. The original is highlighted in red in the result to show what has been added.

Figure 6: A Tgraph (left) and its forced result (right) with the original shown red
Figure 6: A Tgraph (left) and its forced result (right) with the original shown red

Compose

Composition (also called inflation) is an opposite to decompose but this has complications for finite tilings, so it is not simply an inverse. (See Graphs,Kites and Darts and Theorems for more discussion of the problems). Figure 7 shows a Tgraph (left) with the result of composing (right) where we have also shown (in pale green) the faces of the original that are not included in the composition – the remainder faces.

Figure 7: A Tgraph (left) and its (part) composed result (right) with the remainder faces shown pale green
Figure 7: A Tgraph (left) and its (part) composed result (right) with the remainder faces shown pale green

Under some circumstances composing can fail to produce a Tgraph because there are crossing boundaries in the resulting TileFaces. However, we have established that

  • If g is a forced Tgraph, then compose g is defined and it is also a forced Tgraph.

Try Results

It is convenient to use types of the form Try a for results where we know there can be a failure. For example, compose can fail if the result does not pass the connected and no crossing boundary check, and force can fail if its argument is an incorrect Tgraph. In situations when you would like to continue some computation rather than raise an error when there is a failure, use a try version of a function.

    tryCompose :: Tgraph -> Try Tgraph
    tryForce   :: Tgraph -> Try Tgraph

We define Try as a synonym for Either String (which is a monad) in module Tgraph.Try.

type Try a = Either String a

Successful results have the form Right r (for some correct result r) and failure results have the form Left s (where s is a String describing the problem as a failure report).

The function

    runTry:: Try a -> a
    runTry = either error id

will retrieve a correct result but raise an error for failure cases. This means we can always derive an error raising version from a try version of a function by composing with runTry.

    force = runTry . tryForce
    compose = runTry . tryCompose

Elementary Tgraph and TileFace Operations

The module Tgraph.Prelude defines elementary operations on Tgraphs relating vertices, directed edges, and faces. We describe a few of them here.

When we need to refer to particular vertices of a TileFace we use

    originV :: TileFace -> Vertex -- the first vertex - red dot in figure 2
    oppV    :: TileFace -> Vertex -- the vertex at the opposite end of the join edge from the origin
    wingV   :: TileFace -> Vertex -- the vertex not on the join edge

A directed edge is represented as a pair of vertices.

    type Dedge = (Vertex,Vertex)

So (a,b) is regarded as a directed edge from a to b. In the special case that a list of directed edges is symmetrically closed [(b,a) is in the list whenever (a,b) is in the list] we can think of this as an edge list rather than just a directed edge list.

For example,

    internalEdges :: Tgraph -> [Dedge]

produces an edge list, whereas

    graphBoundary :: Tgraph -> [Dedge]

produces single directions. Each directed edge in the resulting boundary will have a TileFace on the left and an exterior region on the right. The function

    graphDedges :: Tgraph -> [Dedge]

produces all the directed edges obtained by going clockwise round each TileFace so not every edge in the list has an inverse in the list.

The above three functions are defined using

    faceDedges :: TileFace -> [Dedge]

which produces a list of the three directed edges going clockwise round a TileFace starting at the origin vertex.

When we need to refer to particular edges of a TileFace we use

    joinE  :: TileFace -> Dedge  -- shown dotted in figure 2
    shortE :: TileFace -> Dedge  -- the non-join short edge
    longE  :: TileFace -> Dedge  -- the non-join long edge

which are all directed clockwise round the TileFace. In contrast, joinOfTile is always directed away from the origin vertex, so is not clockwise for right darts or for left kites:

    joinOfTile:: TileFace -> Dedge
    joinOfTile face = (originV face, oppV face)

Patches (Scaled and Positioned Tilings)

Behind the scenes, when a Tgraph is drawn, each TileFace is converted to a Piece. A Piece is another specialisation of HalfTile using a two dimensional vector to indicate the length and direction of the join edge of the half-tile (from the originV to the oppV), thus fixing its scale and orientation. The whole Tgraph then becomes a list of located Pieces called a Patch.

    type Piece = HalfTile (V2 Double)
    type Patch = [Located Piece]

Piece drawing functions derive vectors for other edges of a half-tile piece from its join edge vector. In particular (in the TileLib module) we have

    drawPiece :: Piece -> Diagram B
    dashjPiece :: Piece -> Diagram B
    fillPieceDK :: Colour Double -> Colour Double -> Piece -> Diagram B

where the first draws the non-join edges of a Piece, the second does the same but adds a dashed line for the join edge, and the third takes two colours – one for darts and one for kites, which are used to fill the piece as well as using drawPiece.

Patch is an instances of class Transformable so a Patch can be scaled, rotated, and translated.

Vertex Patches

It is useful to have an intermediate form between Tgraphs and Patches, that contains information about both the location of vertices (as 2D points), and the abstract TileFaces. This allows us to introduce labelled drawing functions (to show the vertex labels) which we then extend to Tgraphs. We call the intermediate form a VPatch (short for Vertex Patch).

    type VertexLocMap = IntMap.IntMap (Point V2 Double)
    data VPatch = VPatch {vLocs :: VertexLocMap,  vpFaces::[TileFace]} deriving Show

and

    makeVP :: Tgraph -> VPatch

calculates vertex locations using a default orientation and scale.

VPatch is made an instance of class Transformable so a VPatch can also be scaled and rotated.

One essential use of this intermediate form is to be able to draw a Tgraph with labels, rotated but without the labels themselves being rotated. We can simply convert the Tgraph to a VPatch, and rotate that before drawing with labels.

    labelled draw (rotate someAngle (makeVP g))

We can also align a VPatch using vertex labels.

    alignXaxis :: (Vertex, Vertex) -> VPatch -> VPatch 

So if g is a Tgraph with vertex labels a and b we can align it on the x-axis with a at the origin and b on the positive x-axis (after converting to a VPatch), instead of accepting the default orientation.

    labelled draw (alignXaxis (a,b) (makeVP g))

Another use of VPatches is to share the vertex location map when drawing only subsets of the faces (see Overlaid examples in the next section).

4. Drawing in More Detail

Class Drawable

There is a class Drawable with instances Tgraph, VPatch, Patch. When the token B is in scope standing for a fixed backend then we can assume

    draw   :: Drawable a => a -> Diagram B  -- draws non-join edges
    drawj  :: Drawable a => a -> Diagram B  -- as with draw but also draws dashed join edges
    fillDK :: Drawable a => Colour Double -> Colour Double -> a -> Diagram B -- fills with colours

where fillDK clr1 clr2 will fill darts with colour clr1 and kites with colour clr2 as well as drawing non-join edges.

These are the main drawing tools. However they are actually defined for any suitable backend b so have more general types

    draw ::   (Drawable a, Renderable (Path V2 Double) b) =>
              a -> Diagram2D b
    drawj ::  (Drawable a, Renderable (Path V2 Double) b) =>
              a -> Diagram2D b
    fillDK :: (Drawable a, Renderable (Path V2 Double) b) =>
              Colour Double -> Colour Double -> a -> Diagram2D b

where

    type Diagram2D b = QDiagram b V2 Double Any

denotes a 2D diagram using some unknown backend b, and the extra constraint requires b to be able to render 2D paths.

In these notes we will generally use the simpler description of types using B for a fixed chosen backend for the sake of clarity.

The drawing tools are each defined via the class function drawWith using Piece drawing functions.

    class Drawable a where
        drawWith :: (Piece -> Diagram B) -> a -> Diagram B
    
    draw = drawWith drawPiece
    drawj = drawWith dashjPiece
    fillDK clr1 clr2 = drawWith (fillPieceDK clr1 clr2)

To design a new drawing function, you only need to implement a function to draw a Piece, (let us call it newPieceDraw)

    newPieceDraw :: Piece -> Diagram B

This can then be elevated to draw any Drawable (including Tgraphs, VPatches, and Patches) by applying the Drawable class function drawWith:

    newDraw :: Drawable a => a -> Diagram B
    newDraw = drawWith newPieceDraw

Class DrawableLabelled

Class DrawableLabelled is defined with instances Tgraph and VPatch, but Patch is not an instance (because this does not retain vertex label information).

    class DrawableLabelled a where
        labelColourSize :: Colour Double -> Measure Double -> (Patch -> Diagram B) -> a -> Diagram B

So labelColourSize c m modifies a Patch drawing function to add labels (of colour c and size measure m). Measure is defined in Diagrams.Prelude with pre-defined measures tiny, verySmall, small, normal, large, veryLarge, huge. For most of our diagrams of Tgraphs, we use red labels and we also find small is a good default size choice, so we define

    labelSize :: DrawableLabelled a => Measure Double -> (Patch -> Diagram B) -> a -> Diagram B
    labelSize = labelColourSize red

    labelled :: DrawableLabelled a => (Patch -> Diagram B) -> a -> Diagram B
    labelled = labelSize small

and then labelled draw, labelled drawj, labelled (fillDK clr1 clr2) can all be used on both Tgraphs and VPatches as well as (for example) labelSize tiny draw, or labelCoulourSize blue normal drawj.

Further drawing functions

There are a few extra drawing functions built on top of the above ones. The function smart is a modifier to add dashed join edges only when they occur on the boundary of a Tgraph

    smart :: (VPatch -> Diagram B) -> Tgraph -> Diagram B

So smart vpdraw g will draw dashed join edges on the boundary of g before applying the drawing function vpdraw to the VPatch for g. For example the following all draw dashed join edges only on the boundary for a Tgraph g

    smart draw g
    smart (labelled draw) g
    smart (labelSize normal draw) g

When using labels, the function rotateBefore allows a Tgraph to be drawn rotated without rotating the labels.

    rotateBefore :: (VPatch -> a) -> Angle Double -> Tgraph -> a
    rotateBefore vpdraw angle = vpdraw . rotate angle . makeVP

So for example,

    rotateBefore (labelled draw) (90@@deg) g

makes sense for a Tgraph g. Of course if there are no labels we can simply use

    rotate (90@@deg) (draw g)

Similarly alignBefore allows a Tgraph to be aligned using a pair of vertex numbers before drawing.

    alignBefore :: (VPatch -> a) -> (Vertex,Vertex) -> Tgraph -> a
    alignBefore vpdraw (a,b) = vpdraw . alignXaxis (a,b) . makeVP

So, for example, if Tgraph g has vertices a and b, both

    alignBefore draw (a,b) g
    alignBefore (labelled draw) (a,b) g

make sense. Note that the following examples are wrong. Even though they type check, they re-orient g without repositioning the boundary joins.

    smart (labelled draw . rotate angle) g      -- WRONG
    smart (labelled draw . alignXaxis (a,b)) g  -- WRONG

Instead use

    smartRotateBefore (labelled draw) angle g
    smartAlignBefore (labelled draw) (a,b) g

where

    smartRotateBefore :: (VPatch -> Diagram B) -> Angle Double -> Tgraph -> Diagram B
    smartAlignBefore  :: (VPatch -> Diagram B) -> (Vertex,Vertex) -> Tgraph -> Diagram B

are defined using

    restrictSmart :: Tgraph -> (VPatch -> Diagram B) -> VPatch -> Diagram B

Here, restrictSmart g vpdraw vp uses the given vp for drawing boundary joins and drawing faces of g (with vpdraw) rather than converting g to a new VPatch. This assumes vp has locations for vertices in g.

Overlaid examples (location map sharing)

The function

    drawForce :: Tgraph -> Diagram B

will (smart) draw a Tgraph g in red overlaid (using <>) on the result of force g as in figure 6. Similarly

    drawPCompose  :: Tgraph -> Diagram B

applied to a Tgraph g will draw the result of a partial composition of g as in figure 7. That is a drawing of compose g but overlaid with a drawing of the remainder faces of g shown in pale green.

Both these functions make use of sharing a vertex location map to get correct alignments of overlaid diagrams. In the case of drawForce g, we know that a VPatch for force g will contain all the vertex locations for g since force only adds to a Tgraph (when it succeeds). So when constructing the diagram for g we can use the VPatch created for force g instead of starting afresh. Similarly for drawPCompose g the VPatch for g contains locations for all the vertices of compose g so compose g is drawn using the the VPatch for g instead of starting afresh.

The location map sharing is done with

    subVP :: VPatch -> [TileFace] -> VPatch

so that subVP vp fcs is a VPatch with the same vertex locations as vp, but replacing the faces of vp with fcs. [Of course, this can go wrong if the new faces have vertices not in the domain of the vertex location map so this needs to be used with care. Any errors would only be discovered when a diagram is created.]

For cases where labels are only going to be drawn for certain faces, we need a version of subVP which also gets rid of vertex locations that are not relevant to the faces. For this situation we have

    restrictVP:: VPatch -> [TileFace] -> VPatch

which filters out un-needed vertex locations from the vertex location map. Unlike subVP, restrictVP checks for missing vertex locations, so restrictVP vp fcs raises an error if a vertex in fcs is missing from the keys of the vertex location map of vp.

5. Forcing in More Detail

The force rules

The rules used by our force algorithm are local and derived from the fact that there are seven possible vertex types as depicted in figure 8.

Figure 8: Seven vertex types
Figure 8: Seven vertex types

Our rules are shown in figure 9 (omitting mirror symmetric versions). In each case the TileFace shown yellow needs to be added in the presence of the other TileFaces shown.

Figure 9: Rules for forcing
Figure 9: Rules for forcing

Main Forcing Operations

To make forcing efficient we convert a Tgraph to a BoundaryState to keep track of boundary information of the Tgraph, and then calculate a ForceState which combines the BoundaryState with a record of awaiting boundary edge updates (an update map). Then each face addition is carried out on a ForceState, converting back when all the face additions are complete. It makes sense to apply force (and related functions) to a Tgraph, a BoundaryState, or a ForceState, so we define a class Forcible with instances Tgraph, BoundaryState, and ForceState.

This allows us to define

    force :: Forcible a => a -> a
    tryForce :: Forcible a => a -> Try a

The first will raise an error if a stuck tiling is encountered. The second uses a Try result which produces a Left string for failures and a Right a for successful result a.

There are several other operations related to forcing including

    stepForce :: Forcible a => Int -> a -> a
    tryStepForce  :: Forcible a => Int -> a -> Try a

    addHalfDart, addHalfKite :: Forcible a => Dedge -> a -> a
    tryAddHalfDart, tryAddHalfKite :: Forcible a => Dedge -> a -> Try a

The first two force (up to) a given number of steps (=face additions) and the other four add a half dart/kite on a given boundary edge.

Update Generators

An update generator is used to calculate which boundary edges can have a certain update. There is an update generator for each force rule, but also a combined (all update) generator. The force operations mentioned above all use the default all update generator (defaultAllUGen) but there are more general (with) versions that can be passed an update generator of choice. For example

    forceWith :: Forcible a => UpdateGenerator -> a -> a
    tryForceWith :: Forcible a => UpdateGenerator -> a -> Try a

In fact we defined

    force = forceWith defaultAllUGen
    tryForce = tryForceWith defaultAllUGen

We can also define

    wholeTiles :: Forcible a => a -> a
    wholeTiles = forceWith wholeTileUpdates

where wholeTileUpdates is an update generator that just finds boundary join edges to complete whole tiles.

In addition to defaultAllUGen there is also allUGenerator which does the same thing apart from how failures are reported. The reason for keeping both is that they were constructed differently and so are useful for testing.

In fact UpdateGenerators are functions that take a BoundaryState and a focus (list of boundary directed edges) to produce an update map. Each Update is calculated as either a SafeUpdate (where two of the new face edges are on the existing boundary and no new vertex is needed) or an UnsafeUpdate (where only one edge of the new face is on the boundary and a new vertex needs to be created for a new face).

    type UpdateGenerator = BoundaryState -> [Dedge] -> Try UpdateMap
    type UpdateMap = Map.Map Dedge Update
    data Update = SafeUpdate TileFace 
                | UnsafeUpdate (Vertex -> TileFace)

Completing (executing) an UnsafeUpdate requires a touching vertex check to ensure that the new vertex does not clash with an existing boundary vertex. Using an existing (touching) vertex would create a crossing boundary so such an update has to be blocked.

Forcible Class Operations

The Forcible class operations are higher order and designed to allow for easy additions of further generic operations. They take care of conversions between Tgraphs, BoundaryStates and ForceStates.

    class Forcible a where
      tryFSOpWith :: UpdateGenerator -> (ForceState -> Try ForceState) -> a -> Try a
      tryChangeBoundaryWith :: UpdateGenerator -> (BoundaryState -> Try BoundaryChange) -> a -> Try a
      tryInitFSWith :: UpdateGenerator -> a -> Try ForceState

For example, given an update generator ugen and any f:: ForceState -> Try ForceState , then f can be generalised to work on any Forcible using tryFSOpWith ugen f. This is used to define both tryForceWith and tryStepForceWith.

We also specialize tryFSOpWith to use the default update generator

    tryFSOp :: Forcible a => (ForceState -> Try ForceState) -> a -> Try a
    tryFSOp = tryFSOpWith defaultAllUGen

Similarly given an update generator ugen and any f:: BoundaryState -> Try BoundaryChange , then f can be generalised to work on any Forcible using tryChangeBoundaryWith ugen f. This is used to define tryAddHalfDart and tryAddHalfKite.

We also specialize tryChangeBoundaryWith to use the default update generator

    tryChangeBoundary :: Forcible a => (BoundaryState -> Try BoundaryChange) -> a -> Try a
    tryChangeBoundary = tryChangeBoundaryWith defaultAllUGen

Note that the type BoundaryChange contains a resulting BoundaryState, the single TileFace that has been added, a list of edges removed from the boundary (of the BoundaryState prior to the face addition), and a list of the (3 or 4) boundary edges affected around the change that require checking or re-checking for updates.

The class function tryInitFSWith will use an update generator to create an initial ForceState for any Forcible. If the Forcible is already a ForceState it will do nothing. Otherwise it will calculate updates for the whole boundary. We also have the special case

    tryInitFS :: Forcible a => a -> Try ForceState
    tryInitFS = tryInitFSWith defaultAllUGen

Efficient chains of forcing operations.

Note that (force . force) does the same as force, but we might want to chain other force related steps in a calculation.

For example, consider the following combination which, after decomposing a Tgraph, forces, then adds a half dart on a given boundary edge (d) and then forces again.

    combo :: Dedge -> Tgraph -> Tgraph
    combo d = force . addHalfDart d . force . decompose

Since decompose:: Tgraph -> Tgraph, the instances of force and addHalfDart d will have type Tgraph -> Tgraph so each of these operations, will begin and end with conversions between Tgraph and ForceState. We would do better to avoid these wasted intermediate conversions working only with ForceStates and keeping only those necessary conversions at the beginning and end of the whole sequence.

This can be done using tryFSOp. To see this, let us first re-express the forcing sequence using the Try monad, so

    force . addHalfDart d . force

becomes

    tryForce <=< tryAddHalfDart d <=< tryForce

Note that (<=<) is the Kliesli arrow which replaces composition for Monads (defined in Control.Monad). (We could also have expressed this right to left sequence with a left to right version tryForce >=> tryAddHalfDart d >=> tryForce). The definition of combo becomes

    combo :: Dedge -> Tgraph -> Tgraph
    combo d = runTry . (tryForce <=< tryAddHalfDart d <=< tryForce) . decompose

This has no performance improvement, but now we can pass the sequence to tryFSOp to remove the unnecessary conversions between steps.

    combo :: Dedge -> Tgraph -> Tgraph
    combo d = runTry . tryFSOp (tryForce <=< tryAddHalfDart d <=< tryForce) . decompose

The sequence actually has type Forcible a => a -> Try a but when passed to tryFSOp it specialises to type ForceState -> Try ForseState. This ensures the sequence works on a ForceState and any conversions are confined to the beginning and end of the sequence, avoiding unnecessary intermediate conversions.

A limitation of forcing

To avoid creating touching vertices (or crossing boundaries) a BoundaryState keeps track of locations of boundary vertices. At around 35,000 face additions in a single force operation the calculated positions of boundary vertices can become too inaccurate to prevent touching vertex problems. In such cases it is better to use

    recalibratingForce :: Forcible a => a -> a
    tryRecalibratingForce :: Forcible a => a -> Try a

These work by recalculating all vertex positions at 20,000 step intervals to get more accurate boundary vertex positions. For example, 6 decompositions of the kingGraph has 2,906 faces. Applying force to this should result in 53,574 faces but will go wrong before it reaches that. This can be fixed by calculating either

    recalibratingForce (decompositions kingGraph !!6)

or using an extra force before the decompositions

    force (decompositions (force kingGraph) !!6)

In the latter case, the final force only needs to add 17,864 faces to the 35,710 produced by decompositions (force kingGraph) !!6.

6. Advanced Operations

Guided comparison of Tgraphs

Asking if two Tgraphs are equivalent (the same apart from choice of vertex numbers) is a an np-complete problem. However, we do have an efficient guided way of comparing Tgraphs. In the module Tgraph.Rellabelling we have

    sameGraph :: (Tgraph,Dedge) -> (Tgraph,Dedge) -> Bool

The expression sameGraph (g1,d1) (g2,d2) asks if g2 can be relabelled to match g1 assuming that the directed edge d2 in g2 is identified with d1 in g1. Hence the comparison is guided by the assumption that d2 corresponds to d1.

It is implemented using

    tryRelabelToMatch :: (Tgraph,Dedge) -> (Tgraph,Dedge) -> Try Tgraph

where tryRelabelToMatch (g1,d1) (g2,d2) will either fail with a Left report if a mismatch is found when relabelling g2 to match g1 or will succeed with Right g3 where g3 is a relabelled version of g2. The successful result g3 will match g1 in a maximal tile-connected collection of faces containing the face with edge d1 and have vertices disjoint from those of g1 elsewhere. The comparison tries to grow a suitable relabelling by comparing faces one at a time starting from the face with edge d1 in g1 and the face with edge d2 in g2. (This relies on the fact that Tgraphs are connected with no crossing boundaries, and hence tile-connected.)

The above function is also used to implement

    tryFullUnion:: (Tgraph,Dedge) -> (Tgraph,Dedge) -> Try Tgraph

which tries to find the union of two Tgraphs guided by a directed edge identification. However, there is an extra complexity arising from the fact that Tgraphs might overlap in more than one tile-connected region. After calculating one overlapping region, the full union uses some geometry (calculating vertex locations) to detect further overlaps.

Finally we have

    commonFaces:: (Tgraph,Dedge) -> (Tgraph,Dedge) -> [TileFace]

which will find common regions of overlapping faces of two Tgraphs guided by a directed edge identification. The resulting common faces will be a sub-collection of faces from the first Tgraph. These are returned as a list as they may not be a connected collection of faces and therefore not necessarily a Tgraph.

Empires and SuperForce

In Empires and SuperForce we discussed forced boundary coverings which were used to implement both a superForce operation

    superForce:: Forcible a => a -> a

and operations to calculate empires.

We will not repeat the descriptions here other than to note that

    forcedBoundaryECovering:: Tgraph -> [Tgraph]

finds boundary edge coverings after forcing a Tgraph. That is, forcedBoundaryECovering g will first force g, then (if it succeeds) finds a collection of (forced) extensions to force g such that

  • each extension has the whole boundary of force g as internal edges.
  • each possible addition to a boundary edge of force g (kite or dart) has been included in the collection.

(possible here means – not leading to a stuck Tgraph when forced.) There is also

    forcedBoundaryVCovering:: Tgraph -> [Tgraph]

which does the same except that the extensions have all boundary vertices internal rather than just the boundary edges.

Combinations

Combinations such as

    compForce:: Tgraph -> Tgraph      -- compose after forcing
    allCompForce:: Tgraph -> [Tgraph] -- iterated (compose after force) while not emptyTgraph
    maxCompForce:: Tgraph -> Tgraph   -- last item in allCompForce (or emptyTgraph)

make use of theorems established in Graphs,Kites and Darts and Theorems. For example

    compForce = uncheckedCompose . force 

which relies on the fact that composition of a forced Tgraph does not need to be checked for connectedness and no crossing boundaries. Similarly, only the initial force is necessary in allCompForce with subsequent iteration of uncheckedCompose because composition of a forced Tgraph is necessarily a forced Tgraph.

Tracked Tgraphs

The type

    data TrackedTgraph = TrackedTgraph
       { tgraph  :: Tgraph
       , tracked :: [[TileFace]] 
       } deriving Show

has proven useful in experimentation as well as in producing artwork with darts and kites. The idea is to keep a record of sub-collections of faces of a Tgraph when doing both force operations and decompositions. A list of the sub-collections forms the tracked list associated with the Tgraph. We make TrackedTgraph an instance of class Forcible by having force operations only affect the Tgraph and not the tracked list. The significant idea is the implementation of

    decomposeTracked :: TrackedTgraph -> TrackedTgraph

Decomposition of a Tgraph involves introducing a new vertex for each long edge and each kite join. These are then used to construct the decomposed faces. For decomposeTracked we do the same for the Tgraph, but when it comes to the tracked collections, we decompose them re-using the same new vertex numbers calculated for the edges in the Tgraph. This keeps a consistent numbering between the Tgraph and tracked faces, so each item in the tracked list remains a sub-collection of faces in the Tgraph.

The function

    drawTrackedTgraph :: [VPatch -> Diagram B] -> TrackedTgraph -> Diagram B

is used to draw a TrackedTgraph. It uses a list of functions to draw VPatches. The first drawing function is applied to a VPatch for any untracked faces. Subsequent functions are applied to VPatches for the tracked list in order. Each diagram is beneath later ones in the list, with the diagram for the untracked faces at the bottom. The VPatches used are all restrictions of a single VPatch for the Tgraph, so will be consistent in vertex locations. When labels are used, there is also a drawTrackedTgraphRotated and drawTrackedTgraphAligned for rotating or aligning the VPatch prior to applying the drawing functions.

Note that the result of calculating empires (see Empires and SuperForce ) is represented as a TrackedTgraph. The result is actually the common faces of a forced boundary covering, but a particular element of the covering (the first one) is chosen as the background Tgraph with the common faces as a tracked sub-collection of faces. Hence we have

    empire1, empire2 :: Tgraph -> TrackedTgraph
    
    drawEmpire :: TrackedTgraph -> Diagram B

Figure 10 was also created using TrackedTgraphs.

Figure 10: Using a TrackedTgraph for drawing
Figure 10: Using a TrackedTgraph for drawing

7. Other Reading

Previous related blogs are:

  • Diagrams for Penrose Tiles – the first blog introduced drawing Pieces and Patches (without using Tgraphs) and provided a version of decomposing for Patches (decompPatch).
  • Graphs, Kites and Darts intoduced Tgraphs. This gave more details of implementation and results of early explorations. (The class Forcible was introduced subsequently).
  • Empires and SuperForce – these new operations were based on observing properties of boundaries of forced Tgraphs.
  • Graphs,Kites and Darts and Theorems established some important results relating force, compose, decompose.

by readerunner at April 16, 2024 01:04 PM

GHC Developer Blog

GHC 9.6.5 is now available

GHC 9.6.5 is now available

Zubin Duggal - 2024-04-16

The GHC developers are happy to announce the availability of GHC 9.6.5. Binary distributions, source distributions, and documentation are available on the release page.

This release is primarily a bugfix release addressing some issues found in the 9.6 series. These include:

  • Bumping the bundled process library to 1.6.19.0 to avoid a potential command injection vulnerability on Windows for clients of this library. This isn’t known to affect GHC itself, but allows users who depend on the installed version of the process to avoid the issue.
  • Fixing a bug resulting in the distributed hsc2hs wrapper using flags from the compiler build environment (#24050).
  • Disabling the -fasm-shortcutting optimisation with -O2 as it is known to result in unsoundess and incorrect runtime results in some cases (#24507).
  • Ensuring we take LDFLAGS into account when configuring a linker (#24565).
  • Fixing a bug arising from incorrect parsing of paths containing spaces in the settings file (#24265).
  • And many more fixes

A full accounting of changes can be found in the release notes. As some of the fixed issues do affect correctness users are encouraged to upgrade promptly.

We would like to thank Microsoft Azure, GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

Enjoy!

-Zubin

by ghc-devs at April 16, 2024 12:00 AM

April 15, 2024

Well-Typed.Com

Choreographing a dance with the GHC specializer (Part 1)

Specialization is an optimization technique used by GHC to eliminate the performance overhead of ad-hoc polymorphism and enable other powerful optimizations. However, specialization is not free, since it requires more work by GHC during compilation and leads to larger executables. In fact, excessive specialization can result in significant increases in compilation cost and executable size with minimal runtime performance benefits. For this reason, GHC pessimistically avoids excessive specialization by default and may leave relatively low-cost performance improvements undiscovered in doing so.

Optimistic Haskell programmers hoping to take advantage of these missed opportunities are thus faced with the difficult task of discovering and enacting an optimal set of specializations for their program while balancing any performance improvements with the increased compilation costs and executable sizes. Until now, this dance was a clunky one involving desperately wading through GHC Core dumps only to come up with a precarious, inefficient, unmotivated set of pragmas and/or GHC flags that seem to improve performance.

In this two-part series of posts, I describe the recent work we have done to improve this situation and make optimal specialization of Haskell programs more of a science and less of a dark art. In this first post, I will

  • give a comprehensive introduction to GHC’s specialization optimization,
  • explore the various facilities that GHC provides for observing and controlling it, and
  • present a simple framework for thinking about the trade-offs of specialization.

In the next post of the series, I will

  • present the new tools and techniques we have developed to diagnose performance issues resulting from ad-hoc polymorphism,
  • demonstrate how these new tools can be used to systematically identify useful specializations, and
  • make sense of their impact in terms of the framework described in this post.

The intended audience of this post includes intermediate Haskell developers who want to know more about specialization and ad-hoc polymorphism in GHC, and advanced Haskell developers who are interested in systematic approaches to specializing their applications in ways that minimize compilation cost and executable sizes while maximizing performance gains.

This work was made possible thanks to Hasura, who have supported many of Well-Typed’s successful initiatives to improve tooling for commercial Haskell users.

I presented a summary of the content in this post on The Haskell Unfolder:

The Haskell Unfolder Episode 23: specialisation

Overloaded functions are common in Haskell, but they come with a cost. Thankfully, the GHC specialiser is extremely good at removing that cost. We can therefore write high-level, polymorphic programs and be confident that GHC will compile them into very efficient, monomorphised code. In this episode, we’ll demystify the seemingly magical things that GHC is doing to achieve this.

Ad-hoc polymorphism

In Haskell, an ad-hoc polymorphic or overloaded function is one whose type contains class constraints. For example, this f is an overloaded function:

f :: (Ord a, Num a) => a -> a -> a
f x y =
    if x < y then
        x + y
    else
        x - y

For some type a such that Ord a and Num a instances are provided, f takes two values of type a and evaluates to another a.

Importantly, unlike type arguments, those class constraints are not erased at runtime! Actually, they will be passed to f just like any other value argument, meaning f at runtime is more like:

f :: Ord a -> Num a -> a -> a -> a
f ord_a num_a x y = ...

How does the definition of f change to represent this? And what do these ord_a and num_a values look like? This is how it works:

  • Instances are compiled to records, typically referred to as dictionaries, whose fields are the definitions provided in the instance.
  • Class functions (e.g. < in the body of f) become record selectors that are applied to the dictionaries to look up the appropriate definitions.

Thus, f at runtime is more like:

f :: Ord a -> Num a -> a -> a -> a
f ord_a num_a x y =
    if (<) ord_a x y then
        (+) num_a x y
    else
        (-) num_a x y

The previously-infix class operators are now applied in prefix position to select the appropriate definitions out of the dictionaries, which are then applied to the arguments.

We can see this for ourselves by compiling the definition of f in a module F.hs and emitting the intermediate representation (in GHC’s Core language):

ghc F.hs -O -dno-typeable-binds -dsuppress-all -dsuppress-uniques -ddump-ds

The -O flag enables optimizations, and the -ddump-ds flag tells GHC to dump the Core representation of the program after desugaring, before optimizations. The other flags make the output more readable.

For a comprehensive introduction to GHC Core and the flags GHC accepts for viewing it, check out The Haskell Unfolder Episode 9: GHC Core.

The above command will output the following Core for f:

f = \ @a $dOrd $dNum x y ->
      case < $dOrd x y of {
        False -> - $dNum x y;
        True -> + $dNum x y
      }

The if has been transformed into a case (Core has no if construct). The $dOrd and $dNum arguments are the Ord a and Num a instance dictionaries, respectively. The < operator is applied in prefix position (as are all operators in Core) to the $dOrd dictionary to get the appropriate implementation of <, which is further applied to x and y. The - and + operators in the branches of the case are similar.

The extra allocations required to pass these implicit dictionary arguments and apply selectors to them do result in a measurable overhead, albeit one that is insignificant for most intents and purposes. As we will see, the real cost of ad-hoc polymorphism comes from the optimizations it prevents rather than the overhead it introduces.

Specialization

In this context, specialization refers to the removal of ad-hoc polymorphism. When we specialize an overloaded expression e :: C a => S a, we create a new binding eT :: S T, where T is some concrete type for which a C T instance exists. Here eT is the specialization of e at (or to) type T.

For example, we can manually create a specialization of f at type Int. The source definition stays exactly the same, only the type changes:

fInt :: Int -> Int -> Int
fInt x y =
    if x < y then
        x + y
    else
        x - y

At the Core level, the dictionaries that were passed as value arguments to f are now used directly in the body of fInt. If we add the definition of fInt to our example module and compile it as we did before, we get the following output:

f = \ @a $dOrd $dNum x y ->
      case < $dOrd x y of {
        False -> - $dNum x y;
        True -> + $dNum x y
      }

fInt
  = \ x y ->
      case < $fOrdInt x y of {
        False -> - $fNumInt x y;
        True -> + $fNumInt x y
      }

fInt no longer accepts dictionary arguments, and instead references the global Ord Int and Num Int dictionaries directly. In fact, this definition of fInt is exactly what the GHC specializer would create if it decided to specialize f to Int. We can see this for ourselves by manually instructing GHC to do the specialization using a SPECIALIZE pragma. Our whole module is now:

module F where

{-# SPECIALIZE f :: Int -> Int -> Int #-}

f :: (Ord a, Num a) => a -> a -> a
f x y =
    if x < y then
        x + y
    else
        x - y

fInt :: Int -> Int -> Int
fInt x y =
    if x < y then
        x + y
    else
        x - y

And the -ddump-ds Core output becomes:

fInt
  = \ x y ->
      case < $fOrdInt x y of {
        False -> - $fNumInt x y;
        True -> + $fNumInt x y
      }

$sf
  = \ x y ->
      case < $fOrdInt x y of {
        False -> - $fNumInt x y;
        True -> + $fNumInt x y
      }

f = \ @a $dOrd $dNum x y ->
      case < $dOrd x y of {
        False -> - $dNum x y;
        True -> + $dNum x y
      }

The GHC generated specialization is named $sf (all specializations that GHC generates are prefixed by $s). Note that our specialization (fInt) and the GHC generated specialization ($sf) are exactly equivalent!

Why is this an optimization?

The above transformation really is all that the GHC specializer does to our programs. It may not be immediately clear why this optimization is a meaningful optimization at all. That is because specialization is an enabling optimization: The real benefit comes from the optimizations that it enables later in the pipeline, such as inlining.

Inlining is the replacement of defined (top-level or let-bound) variables with their definitions. Although f and its specialization $sf look similar, the key difference is that f includes calls to “unknown” functions passed as part of the dictionary arguments, while $sf includes calls to “known” functions contained in the $fOrdInt and $fNumInt dictionaries. Since GHC has access to the definitions of those dictionaries and the contained functions, they can be inlined, exposing yet more opportunities for optimization.

We can see this in action by comparing the fully optimized bindings of our example module to those just after desugaring. To do this, compile using the same command as above but add the -ddump-simpl flag, which tells GHC to dump the Core at the end of the Core optimization pipeline (also add -fforce-recomp to force recompilation, since we haven’t changed the code since our last compilation):

ghc F.hs -fforce-recomp -O -dno-typeable-binds -dsuppress-all -dsuppress-uniques -ddump-ds -ddump-simpl

The dumped output is:

==================== Desugar (after optimization) ====================
Result size of Desugar (after optimization)
  = {terms: 57, types: 37, coercions: 0, joins: 0/0}

fInt
  = \ x y ->
      case < $fOrdInt x y of {
        False -> - $fNumInt x y;
        True -> + $fNumInt x y
      }

$sf
  = \ x y ->
      case < $fOrdInt x y of {
        False -> - $fNumInt x y;
        True -> + $fNumInt x y
      }

f = \ @a $dOrd $dNum x y ->
      case < $dOrd x y of {
        False -> - $dNum x y;
        True -> + $dNum x y
      }

==================== Tidy Core ====================
Result size of Tidy Core
  = {terms: 44, types: 29, coercions: 0, joins: 0/0}

fInt
  = \ x y ->
      case x of { I# x1 ->
      case y of { I# y1 ->
      case <# x1 y1 of {
        __DEFAULT -> I# (-# x1 y1);
        1# -> I# (+# x1 y1)
      }
      }
      }

f = \ @a $dOrd $dNum x y ->
      case < $dOrd x y of {
        False -> - $dNum x y;
        True -> + $dNum x y
      }

------ Local rules for imported ids --------
"USPEC f @Int" forall $dNum $dOrd. f $dOrd $dNum = fInt

The output of the desugaring pass is in the “Desugar (after optimization)” section, while the fully optimized output is in the “Tidy Core” section. The name “Desugar (after optimization)” only means it is the desugared Core output after GHC’s simple optimizer has run. The simple optimizer only does very lightweight, pure transformations to the Core program. We will still refer to the Core output of this stage as “unoptimized”.

During the full optimization pipeline, GHC identified the equivalence between fInt and $sf and decided to remove $sf. The fully optimized binding for fInt is unboxing the Ints (pattern matching on the I# constructor) and using efficient primitive operations (<#, -#, +#), while the fully optimized binding for f is the same as the unoptimized binding. The optimizer simply couldn’t do anything with those opaque dictionaries in the way!

At the bottom of the output is the rewrite rule that the SPECIALIZE pragma created, which will cause any calls of f known to be at type Int to be rewritten as applications of fInt. This is what allows the rest of the program to benefit from the specialization. The rule simply discards the dictionary arguments $dNum :: Num Int and $dOrd :: Ord Int, which is safe because of global typeclass coherence: any dictionaries passed explicitly must have originally come from the same global instances.

In summary, by replacing the opaque dictionary arguments to f with references to the concrete Ord Int and Num Int dictionaries in fInt, GHC was able to do a lot more optimization later in the pipeline.

Automatic specialization

In our example module, we manually instructed GHC to generate a specialization of f at Int using a SPECIALIZE pragma. In reality, we often rely on GHC to figure out what specializations are necessary and generate them for us automatically. GHC needs to be careful though, since specialization requires the creation and optimization of more bindings, which increases compilation costs and executable sizes

GHC uses several heuristics to avoid excessive automatic specialization by default. The heuristics are very pessimistic, which means GHC can easily miss valuable specialization opportunities that programmers may wish to manually address. This is precisely the manual effort that our recent work aims to assist, so before we go any further it’s important that we understand exactly when and why GHC decides specialization should (or should not) happen.

When does automatic specialization happen?

GHC will only potentially attempt automatic specialization in exactly one scenario: An overloaded call at a concrete, statically known type is encountered (we’ll refer to such calls as “specializable” calls from now on). This means that automatic specialization will only ever be triggered at call sites, not definition sites. Even in this scenario, there are other factors to consider which the following example will demonstrate.

Let’s add a binding foo to our example module F.hs from above:

foo :: (Integer, Integer) -> Integer
foo (x, y) = f x y

foo makes a specializable call to f at the concrete type Integer, so we might expect automatic specialization to happen. However, the inliner beats the specializer to the punch here, which is evident in the -ddump-simpl output:

$wfoo
  = \ ww ww1 ->
      case integerLt ww ww1 of {
        False -> integerSub ww ww1;
        True -> integerAdd ww ww1
      }

foo = \ ds -> case ds of { (ww, ww1) -> $wfoo ww ww1 }

Instead of specializing, GHC decided to eliminate the call entirely by inlining f, thus exposing other optimization opportunities (such as worker/wrapper) which GHC took advantage of. This is intended, since f is so small and GHC knows that inlining it is very cheap and likely worth the performance outcomes.

Another way we can observe the inlining decision by GHC here is via the -ddump-inlinings flag, which causes GHC to dump the names of any bindings it decides to inline. Compiling our module with

ghc F.hs -O -fforce-recomp -ddump-inlinings

results in output indicating that GHC did decide to inline f:

Inlining done: F.f

To inline or to specialize?

GHC prefers inlining over specialization, when possible, since inlining eliminates calls and doesn’t require creation of new bindings. However, excessive inlining is often even more dangerous than excessive specialization. So, even when a specializable call is deemed too costly to inline, GHC will still attempt to specialize it.

We can artificially create such a scenario in our example by adjusting what GHC calls the “unfolding use threshold”. An “unfolding” is, roughly, the definition of a binding that GHC uses when it decides to inline or specialize calls to that binding. The unfolding use threshold governs the maximum effective size1 of unfoldings that GHC will inline, and it can be manually adjusted using the -funfolding-use-threshold flag. Let’s set the unfolding use threshold to -1, essentially making GHC think all inlining is very expensive, and check the -ddump-simpl output:

ghc F.hs -O -fforce-recomp -ddump-simpl -funfolding-use-threshold=-1

As we can see, GHC did specialize the call:

...
f_$sf1
  = \ x y ->
      case integerLt x y of {
        False -> integerSub x y;
        True -> integerAdd x y
      }

foo = \ ds -> case ds of { (ww, ww1) -> f_$sf1 ww ww1 }

------ Local rules for imported ids --------
"SPEC f @Integer" forall $dOrd $dNum. f $dOrd $dNum = f_$sf1
...

The name of the specialization (f_$sf1) and the rewrite rule indicate that GHC did successfully automatically specialize the overloaded call to f.

Interestingly, the Core terms for foo and its specialization f_$sf are alpha-equivalent to the terms we arrived at when GHC inlined the call and applied worker/wrapper instead2, with the specialization playing the same role as the worker.

Cross-module automatic specialization

We have now discussed two prerequisites for automatic specialization of a call:

  • The call must be specializable (i.e. it must be a call to an overloaded binding at a known type).
  • Other optimizations, such as inlining, that remove the call or otherwise ruin the specializability of the call must not fire before specialization can occur.

In fact, for specializable calls which occur in the definition module of the overloaded binding (as was the case in our previous example), these are the only prerequisites. When the overloaded binding is imported from another module (as is most often the case), there are additional prerequisites which we’ll discuss now.

Exposed unfoldings and the INLINABLE pragma

GHC performs separate compilation (as opposed to whole program compilation), compiling one Haskell module at a time. When GHC compiles a module, it produces not only compiled code in an object file, but also an interface file (with suffix .hi) . The interface file contains information about the module that GHC might need to reference when compiling other modules, such as the names and types of the bindings exported by the module. If certain criteria are met, GHC will include a binding’s unfolding in the module’s interface file so that it can be used later for cross-module inlining or specialization. Such unfoldings are referred to as exposed unfoldings.

Now, you might reasonably wonder: If unfoldings are used to do these powerful optimizations, why does GHC only expose unfoldings which meet some criteria? Why not expose all unfoldings? The reason is that during compilation, GHC holds the interfaces of every module in the program in memory. Thus, to keep GHC’s own default performance and memory usage reasonable, module interfaces need to be as small as possible while still producing well-optimized programs. One way that GHC achieves this is by limiting the size of unfoldings that get included in interface files so that only small unfoldings are exposed by default.

There’s another wrinkle here that impacts cross-module specialization: Even if GHC decides to expose an overloaded binding’s unfolding, and a specializable call to that binding occurs in another module, GHC will still never automatically specialize that call unless it has been given explicit permission to create the specialization. Such explicit permission can only be given in one of the following ways:

Let’s explore this fact by continuing with our example. Move foo, which makes a specializable call to f, to another module Foo.hs that has -funfolding-use-threshold set to -1 to fool the inliner as before:

{-# OPTIONS_GHC -funfolding-use-threshold=-1 #-}
module Foo where

import F

foo :: (Integer, Integer) -> Integer
foo (x, y) = f x y

Also remove everything from F.hs except f, for good measure:

module F where

f :: (Ord a, Num a) => a -> a -> a
f x y =
    if x < y then
        x + y
    else
        x - y

Since f is so small, we might expect GHC to expose its unfolding in the F.hi module interface by default. If we compile with just

ghc F.hs

we get the object file F.o and the interface file F.hi. We can determine whether GHC decided to expose the unfolding of f by viewing the contents of the interface file using GHC’s --show-iface option:

ghc --show-iface F.hi -dsuppress-all

Specific information for each binding in the module is listed towards the bottom of the output. The GHC Core of any exposed unfoldings will be displayed under their respective bindings. In this case, the information for f looks like this:

bcb4b04f3cbb5e6aa2f776d6226a0930
  f :: (Ord a, Num a) => a -> a -> a
  []

It only includes the type, no unfolding! This is because at GHC’s default optimization level of -O0, the -fomit-interface-pragmas and -fignore-interface-pragmas flags are enabled which prevent unfoldings (among other things) from being included in and read from the module interfaces. Recompile with optimizations enabled and check the module interface again:

ghc -O F.hs
ghc --show-iface F.hi -dsuppress-all

This time, GHC did expose the unfolding:

152dd20f273a86bea689edd6a298afe6
  f :: (Ord a, Num a) => a -> a -> a
  [...,
   Unfolding: Core: <vanilla>
              \ @a
                ($dOrd['Many] :: Ord a)
                ($dNum['Many] :: Num a)
                (x['Many] :: a)
                (y['Many] :: a) ->
              case < @a $dOrd x y of wild {
                False -> - @a $dNum x y True -> + @a $dNum x y }]

Remember, we still haven’t given GHC explicit permission to specialize calls to f across modules, so we should expect the fully optimized Core of Foo.hs to still include the overloaded call to f. Let’s check:

ghc Foo.hs -O -dno-typeable-binds -dsuppress-all -dsuppress-uniques -ddump-simpl

The dumped Core includes:

$wfoo = \ ww ww1 -> f $fOrdInteger $fNumInteger ww ww1

foo = \ ds -> case ds of { (ww, ww1) -> $wfoo ww ww1 }

Indeed, GHC applied the worker/wrapper transformation to foo, but was not able to specialize the call to f, despite it meeting our previously discussed prerequisites for automatic specialization.

There is a warning flag in GHC that can notify us of such a case: -Wall-missed-specializations. Compile Foo.hs again, including this flag:

ghc Foo.hs -O -fforce-recomp -Wall-missed-specializations

This will output the following warning:

Foo.hs: warning: [-Wall-missed-specialisations]
    Could not specialise imported function ‘f’
    Probable fix: add INLINABLE pragma on ‘f’

If we do what the warning says by adding an INLINABLE pragma on f, and dump the core of Foo.hs, we’ll see that automatic specialization succeeds:

$sf
  = \ x y ->
      case integerLt x y of {
        False -> integerSub x y;
        True -> integerAdd x y
      }

foo = \ ds -> case ds of { (ww, ww1) -> $sf ww ww1 }

------ Local rules for imported ids --------
"SPEC/Foo f @Integer" forall $dOrd $dNum. f $dOrd $dNum = $sf

Removing the INLINABLE pragma on f and instead enabling -fspecialize-aggressively has the same result.

The automatic specialization decision graph

We have now covered all the major prerequisites for automatic specialization. To summarize them, here is a decision graph illustrating the various ways that an arbitrary function call can trigger automatic specialization:

The automatic specialization decision graph
The automatic specialization decision graph

Now that we fully understand how, why, and when the GHC specializer works, we can move on to discussing the real problems that result from its behavior. Most of this discussion will be left for the next post in this series, but before concluding, I want to introduce something I call “the specialization spectrum”.

The specialization spectrum

Specialization is a very valuable compiler optimization, but I’ve mentioned many times throughout this post that excessive specialization can be a bad thing. This prompts a question: How do we know if we are appropriately benefitting from specialization? The meaning of “appropriately” here depends on application-specific requirements that dictate the desired size of our executables, how much we care about compilation costs, and how much we care about performance.

For example, if we want to maximize performance at all costs, we should make sure that we are generating and using the set of specializations that maximize the performance metrics we’re interested in, disregarding the increase in compilation costs and executable sizes.

Essentially, our goal is to find our ideal spot in the specialization spectrum.

The Specialization Spectrum
Idealized graph of performance against code size and compilation cost.
Idealized graph of performance against code size and compilation cost.

This is our search space, with performance on one axis and code size and compilation cost on the other. The plotted points represent important application-agnostic points in the spectrum. Those points are:

  • Baseline: Lowest performance and lowest cost. This point represents GHC’s default behavior where its heuristics will result in smaller code size and lower compilation cost but potentially miss specializations that would result in big performance wins.
  • Ideal: As the application authors, we get to choose the location of this point based on our priorities. Typically, we want this as “high and to the left” as possible.
  • Max performance: This point represents the optimal set of specializations, which will result in better runtime performance than any other set of specializations.
  • Max specialization: This point is the result of generating every3 possible specialization by enabling -fexpose-all-unfoldings and -fspecialize-aggressively. Importantly, this is not always equivalent to max performance! If we generate useless specializations that result in little to no performance improvements but do grow the code size, we can end up losing performance due to more code swapping in and out of CPU caches.

The dotted line illustrates an approximate “optimal path” representing the results we might see as we generate all specializations in order of decreasing performance improvement.

This framework makes it clear that this really is just an optimization problem, with all the normal issues of traditional optimization problems at play. Unfortunately, in the absence of good tools for exploring this spectrum, it is particularly easy for programmers to get lost and go down treacherous, unoptimal paths like this:

Typical path leading to high compilation cost but only medium performance.
Typical path leading to high compilation cost but only medium performance.

Such cases are deceptive, making the programmer think they have landed in a good spot when they are actually in a poor-performing local optimum. Fortunately, the tools and techniques we’ll discuss in the next post of this series will greatly simplify optimal search of the specialization spectrum.

Summary

This concludes our introductory exploration of specialization. Here’s what we have learned:

  • Calls to overloaded functions are compiled by passing dictionary values with a record of functions for each type class constraint.
  • Specialization removes type class dictionary arguments from an overloaded function and replaces references to them with references to a concrete dictionary instead.
  • Almost all of the benefit of specialization comes from the optimizations that it enables by replacing the opaque dictionary arguments with concrete dictionaries whose contents can be inlined.
  • GHC will only automatically specialize calls if a specific set of conditions holds. See the automatic specialization decision graph.
  • The specialization spectrum is a convenient framework for conceptualizing the impact of specialization on a program’s compilation cost and runtime performance.

In the next post of this series, we will apply all of what we have learned so far on some example applications, and demonstrate how the new tools we have developed can help us achieve optimal specialization and performance.

Footnotes

  1. The effective size of an unfolding can be thought of as the number of terms in the Core representation of the unfolding, plus or minus some discounts that are applied depending on where GHC is considering inlining the unfolding.↩︎

  2. This hints at a weak confluence of GHC Core and the reductions (i.e. optimizations) that the GHC optimizer applies to it.↩︎

  3. Even with something like this in a cabal.project file:

    package *
      ghc-options: -fexpose-all-unfoldings -fspecialize-aggressively

    Some overloaded calls may still not get specialized! This can occur if a chain of calls to overloaded functions includes a call to an overloaded function in a GHC boot library that cannot be reinstalled by Cabal, e.g. base, which does not have its unfolding exposed. The only way to specialize such calls is to build boot libraries from source with -fexpose-all-unfoldings and -fspecialize-aggressively, and include the snippet above in a cabal.project file.

    Additionally, some specific scenarios can cause overloaded calls to appear late in the optimization pipeline. To specialize those calls, -flate-specialise (British spelling required) is necessary, which runs another specialization pass at the end of GHC’s Core optimization pipeline.

    Further, even after the above, some overloaded calls may still survive without -fpolymorphic-specialisation (British spelling required), which is known to be unsound at the time of writing. Unfortunately, in complex applications, total elimination of overloaded calls is still quite a difficult goal to achieve.↩︎

by finley at April 15, 2024 12:00 AM

GHC Developer Blog

GHC 9.10.1-alpha3 is now available

GHC 9.10.1-alpha3 is now available

bgamari - 2024-04-15

The GHC developers are very pleased to announce the availability of the third alpha release of GHC 9.10.1. Binary distributions, source distributions, and documentation are available at downloads.haskell.org.

We hope to have this release available via ghcup shortly.

GHC 9.10 will bring a number of new features and improvements, including:

  • The introduction of the GHC2024 language edition, building upon GHC2021 with the addition of a number of widely-used extensions.

  • Partial implementation of the GHC Proposal #281, allowing visible quantification to be used in the types of terms.

  • Extension of LinearTypes to allow linear let and where bindings

  • The implementation of the exception backtrace proposal, allowing the annotation of exceptions with backtraces, as well as other user-defined context

  • Further improvements in the info table provenance mechanism, reducing code size to allow IPE information to be enabled more widely

  • Javascript FFI support in the WebAssembly backend

  • Improvements in the fragmentation characteristics of the low-latency non-moving garbage collector.

  • … and many more

A full accounting of changes can be found in the release notes. As always, GHC’s release status, including planned future releases, can be found on the GHC Wiki status.

This alpha is the penultimate prerelease leading to 9.10.1. In two weeks we plan to publish a release candidate, followed, if all things go well, by the final release a week later.

We would like to thank GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release.

As always, do give this release a try and open a ticket if you see anything amiss.

by ghc-devs at April 15, 2024 12:00 AM

April 12, 2024

Oleg Grenrus

Core Inspection

Posted on 2024-04-12 by Oleg Grenrus

inspection-testing was created over five years ago. You may want to glance over Joachim Breitner A promise checked is a promise kept: inspection testing) Haskell Symposium paper introducing it.

Already in 2018 I thought it's a fine tool, but it's more geared towards /library/ writers. They can check on (some) examples that the promises they make about the libraries they write work at least on some examples.

What we cannot do with current inspection-testing is check that the actual "real-life" use of the library works as intended.

Luckily, relatively recently, GHC got a feature to include all Core bindings in the interface files. While the original motivation is different (to make Template Haskell run fast), the -fwrite-if-simplified-core enables us to inspect (as in inspection testing) the "production" Core (not the test examples).

The cabal-core-inspection is a very quick & dirty proof-of-concept of this idea.

Let me illustrate this with two examples.

In neither example I need to do any test setup, other than configuring cabal-core-inspection (though configuration is now hardcoded). Compare that to configuring e.g. HLint (HLint has user definable rules, and these are actually powerful tool). In fact, cabal-core-inspection is nothing more than a linter for Core.

countChars

First example is countChars as in Haskell Symposium Paper.

countChars :: ByteString -> Int
countChars = T.length . T.toUpper . TE.decodeUtf8

The promise is (actually: was) that no intermediate Text values are created.

As far as I know, we cannot use inspection-testing in its current form to check anything about non-local bindings, so if countChars is defined in an application, we would need to duplicate its definition in the test-suite to inspect it. That is not great.

With Core inspection, we can look at the actual Core of the module (as it is in the compiler interface file).

The prototype doesn't have any configuration, but if we imagine it has we could ask it to check that Example.countChars should not contain type Text. The prototype prints

Text value created with decodeUtf8With1 in countChars

So that's not the case. The intermediate Text value is created. In fact, nowadays text doesn't promise that toUpper fuses with anything.

A nice thing about cabal-core-inspection that (in theory) it could check any definition in any module as long as it's compiled with -fwrite-if-simplified-core. So we could check things for our friends, if we care about something specific.

no Generics

Second example is about GHC.Generics. I use a simple generic equality, but this could apply to any GHC.Generics based deriving. (You should rather use deriving stock Eq, but generic equality is a simplest example which I remembered for now).

The generic equality might be defined in a library. And library author may actually have tested it with inspection-testing. But does it work on our type?

If we have

data T where
    T1 :: Int -> Char -> T
    T2 :: Bool -> Double -> T
  deriving Generic

instance Eq T where
    (==) = genericEq

it does. The cabal-core-inspection doesn't complain.

But if we add a third constructor

data T where
    T1 :: Int -> Char -> T
    T2 :: Bool -> Double -> T
    T3 :: ByteString -> T.Text -> T

cabal-core-inspection barfs:

Found L1 from GHC.Generics
Found :*: from GHC.Generics
Found R1 from GHC.Generics

The T becomes too large for GHC to want inline all the generics stuff.

It won't be fair to blame the library author, for example for

data T where
    T1 :: Int -> T
    T2 :: Bool -> T
    T3 :: Char -> T
    T4 :: Double -> T
  deriving Generic

generic equality still optimises well, and doesn't have any traces of GHC.Generics. We may actually need to (and may be adviced to) tune some GHC optimisation parameters. But we need a way to check whether they are enough. inspection-testing doesn't help, but a proper version of core inspection would be perfect for that task.

Conclusion

The -fwrite-if-simplified-core enables us to automate inspection of actual Core. That is a huge win. The cabal-core-inspection is just a proof-of-concept, and I might try to make it into a real thing, but right now I don't have a real use case for it.

I'm also worried about Note [Interface File with Core: Sharing RHSs] in GHC. It says

In order to avoid duplicating definitions for bindings which already have unfoldings we do some minor headstands to avoid serialising the RHS of a definition if it has *any* unfolding.

  • Only global things have unfoldings, because local things have had their unfoldings stripped.
  • For any global thing which has an unstable unfolding, we just use that.

Currently this optimisation is disabled, so cabal-core-inspection works, but if it's enabled as is; then INLINEd bindings won't have their simplified unfoldings preserved (but rather only "inline-RHS"), and that would destroy Core inspection possibility.

But until then, cabal-core-inspection idea works.

April 12, 2024 12:00 AM

April 07, 2024

Abhinav Sarkar

Solving Advent of Code ’23 “Aplenty” by Compiling

Every year I try to solve some problems from the Advent of Code (AoC) competition in a not straightforward way. Let’s solve the part one of the day 19 problem Aplenty by compiling the problem input to an executable file.

This post was originally published on abhinavsarkar.net.

The Problem

What the problem presents as input is essentially a program. Here is the example input:

px{a<2006:qkq,m>2090:A,rfg}
pv{a>1716:R,A}
lnx{m>1548:A,A}
rfg{s<537:gd,x>2440:R,A}
qs{s>3448:A,lnx}
qkq{x<1416:A,crn}
crn{x>2662:A,R}
in{s<1351:px,qqz}
qqz{s>2770:qs,m<1801:hdj,R}
gd{a>3333:R,R}
hdj{m>838:A,pv}

{x=787,m=2655,a=1222,s=2876}
{x=1679,m=44,a=2067,s=496}
{x=2036,m=264,a=79,s=2244}
{x=2461,m=1339,a=466,s=291}
{x=2127,m=1623,a=2188,s=1013}
exinput.txt

Each line in the first section of the input is a code block. The bodies of the blocks have statements of these types:

  • Accept (A) or Reject (R) that terminate the program.
  • Jumps to other blocks by their names, for example: rfg as the last statement of the px block in the first line.
  • Conditional statements that have a condition and what to do if the condition is true, which can be only Accept/Reject or a jump to another block.

The problem calls the statements “rules”, the blocks “workflows”, and the program “system”.

All blocks of the program operates on a set of four values: x, m, a, and s. The problem calls them “ratings”, and each set of ratings is for/forms a “part”. The second section of the input specifies a bunch of these parts to run the system against.

This seems to map very well to a C program, with Accept and Reject returning true and false respectively, and jumps accomplished using gotos. So that’s what we’ll do: we’ll compile the problem input to a C program, then compile that to an executable, and run it to get the solution to the problem.

And of course, we’ll do all this in Haskell. First some imports:

{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE StrictData #-}

module Main where

import qualified Data.Array as Array
import Data.Char (digitToInt, isAlpha, isDigit)
import Data.Foldable (foldl', foldr')
import Data.Function (fix)
import Data.Functor (($>))
import qualified Data.Graph as Graph
import Data.List (intercalate, (\\))
import qualified Data.Map.Strict as Map
import System.Environment (getArgs)
import qualified Text.ParserCombinators.ReadP as P
import Prelude hiding (GT, LT)

The Parser

First, we parse the input program to Haskell data types. We use the ReadP parser library built into the Haskell standard library.

data Part = Part
  { partX :: Int,
    partM :: Int,
    partA :: Int,
    partS :: Int
  } deriving (Show)

data Rating = X | M | A | S deriving (Show, Eq)

emptyPart :: Part
emptyPart = Part 0 0 0 0

addRating :: Part -> (Rating, Int) -> Part
addRating p (r, v) = case r of
  X -> p {partX = v}
  M -> p {partM = v}
  A -> p {partA = v}
  S -> p {partS = v}

partParser :: P.ReadP Part
partParser =
  foldl' addRating emptyPart
    <$> P.between (P.char '{') (P.char '}')
          (partRatingParser `P.sepBy1` P.char ',')

partRatingParser :: P.ReadP (Rating, Int)
partRatingParser =
  (,) <$> ratingParser <*> (P.char '=' *> intParser)

ratingParser :: P.ReadP Rating
ratingParser =
  P.get >>= \case
    'x' -> pure X
    'm' -> pure M
    'a' -> pure A
    's' -> pure S
    _ -> P.pfail

intParser :: P.ReadP Int
intParser =
  foldl' (\n d -> n * 10 + d) 0 <$> P.many1 digitParser

digitParser :: P.ReadP Int
digitParser = digitToInt <$> P.satisfy isDigit

parse :: (Show a) => P.ReadP a -> String -> Either String a
parse parser text = case P.readP_to_S (parser <* P.eof) text of
  [(res, "")] -> Right res
  [(_, s)] -> Left $ "Leftover input: " <> s
  out -> Left $ "Unexpected output: " <> show out

Part is a Haskell data type representing parts, and Rating is an enum for, well, ratings1. Following that are parsers for parts and ratings, written in Applicative and Monadic styles using the basic parsers and combinators provided by the ReadP library.

Finally, we have the parse function to run a parser on an input. We can try parsing parts in GHCi:

> parse partParser "{x=2127,m=1623,a=2188,s=1013}"
Right (Part {partX = 2127, partM = 1623, partA = 2188, partS = 1013})

Next, we represent and parse the program, I mean, the system:

newtype System =
  System (Map.Map WorkflowName Workflow)
  deriving (Show, Eq)

data Workflow = Workflow
  { wName :: WorkflowName,
    wRules :: [Rule]
  } deriving (Show, Eq)

type WorkflowName = String

data Rule
  = AtomicRule AtomicRule
  | If Condition AtomicRule
  deriving (Show, Eq)

data AtomicRule
  = Jump WorkflowName
  | Accept
  | Reject
  deriving (Show, Eq, Ord)

data Condition
  = Comparison Rating CmpOp Int
  deriving (Show, Eq)

data CmpOp = LT | GT deriving (Show, Eq)

A System is a map of workflows by their names. A Workflow has a name and a list of rules. A Rule is either an AtomicRule, or an If rule. An AtomicRule is either a Jump to another workflow by name, or an Accept or Reject rule. The Condition of an If rule is a less that (LT) or a greater than (GT) Comparison of some Rating of an input part with an integer value.

Now, it’s time to parse the system:

systemParser :: P.ReadP System
systemParser =
  System
    . foldl' (\m wf -> Map.insert (wName wf) wf m) Map.empty
    <$> workflowParser `P.endBy1` P.char '\n'

workflowParser :: P.ReadP Workflow
workflowParser =
  Workflow
    <$> P.many1 (P.satisfy isAlpha)
    <*> P.between (P.char '{') (P.char '}')
          (ruleParser `P.sepBy1` P.char ',')

ruleParser :: P.ReadP Rule
ruleParser =
  (AtomicRule <$> atomicRuleParser) P.<++ ifRuleParser

ifRuleParser :: P.ReadP Rule
ifRuleParser =
  If
    <$> (Comparison <$> ratingParser <*> cmpOpParser <*> intParser)
    <*> (P.char ':' *> atomicRuleParser)

atomicRuleParser :: P.ReadP AtomicRule
atomicRuleParser = do
  c : _ <- P.look
  case c of
    'A' -> P.char 'A' $> Accept
    'R' -> P.char 'R' $> Reject
    _ -> (Jump .) . (:) <$> P.char c <*> P.many1 (P.satisfy isAlpha)

cmpOpParser :: P.ReadP CmpOp
cmpOpParser = P.choice [P.char '<' $> LT, P.char '>' $> GT]

Parsing is straightforward as there are no recursive data types or complicated precedence or associativity rules here. We can exercise it in GHCi (output formatted for clarity):

> parse workflowParser "px{a<2006:qkq,m>2090:A,rfg}"
Right (
  Workflow {
    wName = "px",
    wRules = [
      If (Comparison A LT 2006) (Jump "qkq"),
      If (Comparison M GT 2090) Accept,
      AtomicRule (Jump "rfg")
    ]
  }
)

Excellent! We can now combine the part parser and the system parser to parse the problem input:

data Input = Input System [Part] deriving (Show)

inputParser :: P.ReadP Input
inputParser =
  Input
    <$> systemParser
    <*> (P.char '\n' *> partParser `P.endBy1` P.char '\n')

Before moving on to translating the system to C, let’s write an interpreter so that we can compare the output of our final C program against it for validation.

The Interpreter

Each system has a workflow named “in”, where the execution of the system starts. Running the system results in True if the run ends with an Accept rule, or in False if the run ends with a Reject rule. With this in mind, let’s cook up the interpreter:

runSystem :: System -> Part -> Bool
runSystem (System system) part = runRule $ Jump "in"
  where
    runRule = \case
      Accept -> True
      Reject -> False
      Jump wfName -> jump wfName

    jump wfName = case Map.lookup wfName system of
      Just workflow -> runRules $ wRules workflow
      Nothing ->
        error $ "Workflow not found in system: " <> wfName

    runRules = \case
      (rule : rest) -> case rule of
        AtomicRule aRule -> runRule aRule
        If cond aRule ->
          if evalCond cond
            then runRule aRule
            else runRules rest
      _ -> error "Workflow ended without accept/reject"

    evalCond = \case
      Comparison r LT value -> rating r < value
      Comparison r GT value-> rating r > value

    rating = \case
      X -> partX part
      M -> partM part
      A -> partA part
      S -> partS part

The interpreter starts by running the rule to jump to the “in” workflow. Running a rule returns True or False for Accept or Reject rules respectively, or jumps to a workflow for Jump rules. Jumping to a workflow looks it up in the system’s map of workflows, and sequentially runs each of its rules.

An AtomicRule is run as previously mentioned. An If rule evaluates its condition, and either runs the consequent rule if the condition is true, or moves on to running the rest of the rules in the workflow.

That’s it for the interpreter. We can run it on the example input:

> inputText <- readFile "input.txt"
> Right (Input system parts) = parse inputParser inputText
> runSystem system (parts !! 0)
True
> runSystem system (parts !! 1)
False

The AoC problem requires us to return the sum total of the ratings of the parts that are accepted by the system:

solve :: Input -> Int
solve (Input system parts) =
  sum
  . map (\(Part x m a s) -> x + m + a + s)
  . filter (runSystem system)
  $ parts

Let’s run it for the example input:

> Right input <- parse inputParser <$> readFile "exinput.txt"
> solve input
19114

It returns the correct answer! Next up, we generate some C code.

The Control-flow Graph

But first, a quick digression to graphs. A Control-flow graph or CFG, is a graph of all possible paths that can be taken through a program during its execution. It has many uses in compilers, but for now, we use it to generate more readable C code.

Using the Data.Graph module from the containers package, we write the function to create a control-flow graph for our system/program, and use it to topologically sort the workflows:

type Graph' a =
  (Graph.Graph, Graph.Vertex -> (a, [a]), a -> Maybe Graph.Vertex)

cfGraph :: Map.Map WorkflowName Workflow -> Graph' WorkflowName
cfGraph system =
  graphFromMap
    . Map.toList
    . flip Map.map system
    $ \(Workflow _ rules) ->
      flip concatMap rules $ \case
        AtomicRule (Jump wfName) -> [wfName]
        If _ (Jump wfName) -> [wfName]
        _ -> []
  where
    graphFromMap :: (Ord a) => [(a, [a])] -> Graph' a
    graphFromMap m =
      let (graph, nLookup, vLookup) =
            Graph.graphFromEdges $ map (\(f, ts) -> (f, f, ts)) m
       in (graph, \v -> let (x, _, xs) = nLookup v in (x, xs), vLookup)

toposortWorkflows :: Map.Map WorkflowName Workflow -> [WorkflowName]
toposortWorkflows system =
  let (cfg, nLookup, _) = cfGraph system
   in map (fst . nLookup) $ Graph.topSort cfg

Graph' is a simpler type for a graph of nodes of type a. The cfGraph function takes a the map from workflow names to workflows — that is, a system — and returns a control-flow graph of workflow names. It does this by finding jumps from workflows to other workflows, and connecting them.

Then, the toposortWorkflows function uses the created CFG to topologically sort the workflows. We’ll see this in action in a bit. Moving on to …

The Compiler

The compiler, for now, simply generates the C code for a given system. We write a ToC typeclass for convenience:

class ToC a where
  toC :: a -> String

instance ToC Part where
  toC (Part x m a s) =
    "{" <> intercalate ", " (map show [x, m, a, s]) <> "}"

instance ToC CmpOp where
  toC = \case
    LT -> "<"
    GT -> ">"

instance ToC Rating where
  toC = \case
    X -> "x"
    M -> "m"
    A -> "a"
    S -> "s"

instance ToC AtomicRule where
  toC = \case
    Accept -> "return true;"
    Reject -> "return false;"
    Jump wfName -> "goto " <> wfName <> ";"

instance ToC Condition where
  toC = \case
    Comparison rating op val ->
      toC rating <> " " <> toC op <> " " <> show val

instance ToC Rule where
  toC = \case
    AtomicRule aRule -> toC aRule
    If cond aRule ->
      "if (" <> toC cond <> ") { " <> toC aRule <> " }"

instance ToC Workflow where
  toC (Workflow wfName rules) =
    wfName
      <> ":\n"
      <> intercalate "\n" (map (("  " <>) . toC) rules)

instance ToC System where
  toC (System system) =
    intercalate
      "\n"
      [ "bool runSystem(int x, int m, int a, int s) {",
        "  goto in;",
        intercalate
          "\n"
          (map (toC . (system Map.!)) $ toposortWorkflows system),
        "}"
      ]

instance ToC Input where
  toC (Input system parts) =
    intercalate
      "\n"
      [ "#include <stdbool.h>",
        "#include <stdio.h>\n",
        toC system,
        "int main() {",
        "  int parts[][4] = {",
        intercalate ",\n" (map (("    " <>) . toC) parts),
        "  };",
        "  int totalRating = 0;",
        "  for(int i = 0; i < " <> show (length parts) <> "; i++) {",
        "    int x = parts[i][0];",
        "    int m = parts[i][1];",
        "    int a = parts[i][2];",
        "    int s = parts[i][3];",
        "    if (runSystem(x, m, a, s)) {",
        "      totalRating += x + m + a + s;",
        "    }",
        "  }",
        "  printf(\"%d\", totalRating);",
        "  return 0;",
        "}"
      ]

As mentioned before, Accept and Reject rules are converted to return true and false respectively, and Jump rules are converted to gotos. If rules become if statements, and Workflows become block labels followed by block statements.

A System is translated to a function runSystem that takes four parameters, x, m, a and s, and runs the workflows translated to blocks by executing goto in.

Finally, an Input is converted to a C file with the required includes, and a main function that solves the problem by calling the runSystem function for all parts.

Let’s throw in a main function to put everything together.

main :: IO ()
main = do
  file <- head <$> getArgs
  code <- readFile file
  case parse inputParser code of
    Right input -> putStrLn $ toC input
    Left err -> error err

The main function reads the input from the file provided as the command line argument, parses it and outputs the generated C code. Let’s run it now.

The Compiler Output

We compile our compiler and run it to generate the C code for the example problem:

$ ghc --make aplenty.hs
$ ./aplenty exinput.txt > aplenty.c

This is the C code it generates:

#include <stdbool.h>
#include <stdio.h>

bool runSystem(int x, int m, int a, int s) {
  goto in;
in:
  if (s < 1351) { goto px; }
  goto qqz;
qqz:
  if (s > 2770) { goto qs; }
  if (m < 1801) { goto hdj; }
  return false;
qs:
  if (s > 3448) { return true; }
  goto lnx;
lnx:
  if (m > 1548) { return true; }
  return true;
px:
  if (a < 2006) { goto qkq; }
  if (m > 2090) { return true; }
  goto rfg;
rfg:
  if (s < 537) { goto gd; }
  if (x > 2440) { return false; }
  return true;
qkq:
  if (x < 1416) { return true; }
  goto crn;
hdj:
  if (m > 838) { return true; }
  goto pv;
pv:
  if (a > 1716) { return false; }
  return true;
gd:
  if (a > 3333) { return false; }
  return false;
crn:
  if (x > 2662) { return true; }
  return false;
}
int main() {
  int parts[][4] = {
    {787, 2655, 1222, 2876},
    {1679, 44, 2067, 496},
    {2036, 264, 79, 2244},
    {2461, 1339, 466, 291},
    {2127, 1623, 2188, 1013}
  };
  int totalRating = 0;
  for(int i = 0; i < 5; i++) {
    int x = parts[i][0];
    int m = parts[i][1];
    int a = parts[i][2];
    int s = parts[i][3];
    if (runSystem(x, m, a, s)) {
      totalRating += x + m + a + s;
    }
  }
  printf("%d", totalRating);
  return 0;
}

We see the toposortWorkflows function in action, sorting the blocks in the topological order of jumps between them, as opposed to the original input. Does this work? Only one way to know:

$ gcc aplenty.c -o solution
$ ./solution
19114

Perfect! The solution matches the interpreter output.

The Bonus: Optimizations

By studying the output C code, we spot some possibilities for optimizing the compiler output. Notice how the lnx block returns same value (true) regardless of which branch it takes:

lnx:
  if (m > 1548) { return true; }
  return true;

So, we should be able to replace it with:

lnx:
  return true;

If we do this, the lnx block becomes degenerate, and hence the jumps to the block can be inlined, turning the qs block from:

qs:
  if (s > 3448) { return true; }
  goto lnx;

to:

qs:
  if (s > 3448) { return true; }
  return true;

which makes the if statement in the qs block redundant as well. Hence, we can repeat the previous optimization and further reduce the generated code.

Another possible optimization is to inline the blocks to which there are only single jumps from the rest of the blocks, for example the qqz block.

Let’s write these optimizations.

Simplify Workflows

simplifyWorkflows :: System -> System
simplifyWorkflows (System system) =
  System $ Map.map simplifyWorkflow system
  where
    simplifyWorkflow (Workflow name rules) =
      Workflow name
        $ foldr'
          ( \r rs -> case rs of
              [r'] | ruleOutcome r == ruleOutcome r' -> rs
              _ -> r : rs
          )
          [last rules]
        $ init rules

    ruleOutcome = \case
      If _ aRule -> aRule
      AtomicRule aRule -> aRule

simplifyWorkflows goes over all workflows and repeatedly removes the statements from the end of the blocks that has same outcome as the statement previous to them.

Inline Redundant Jumps

inlineRedundantJumps :: System -> System
inlineRedundantJumps (System system) =
  System $
    foldl' (flip Map.delete) (Map.map inlineJumps system) $
      Map.keys redundantJumps
  where
    redundantJumps =
      Map.map (\wf -> let ~(AtomicRule rule) = head $ wRules wf in rule)
        . Map.filter (\wf -> length (wRules wf) == 1)
        $ system

    inlineJumps (Workflow name rules) =
      Workflow name $ map inlineJump rules

    inlineJump = \case
      AtomicRule (Jump wfName)
        | Map.member wfName redundantJumps ->
            AtomicRule $ redundantJumps Map.! wfName
      If cond (Jump wfName)
        | Map.member wfName redundantJumps ->
            If cond $ redundantJumps Map.! wfName
      rule -> rule

inlineRedundantJumps find the jumps to degenerate workflows and inlines them. It does this by first going over all workflows and creating a map of degenerate workflow names to the only rule in them, and then replacing the jumps to such workflows with the only rules.

Remove Jumps

removeJumps :: System -> System
removeJumps (System system) =
  let system' =
        foldl' (flip $ Map.adjust removeJumpsWithSingleJumper) system $
          toposortWorkflows system
   in System
        . foldl' (flip Map.delete) system'
        . (\\ ["in"])
        $ workflowsWithNJumpers 0 system'
  where
    removeJumpsWithSingleJumper (Workflow name rules) =
      Workflow name $
        init rules <> case last rules of
          AtomicRule (Jump wfName)
            | wfName `elem` workflowsWithSingleJumper ->
                let (Workflow _ rules') = system Map.! wfName
                 in rules'
          rule -> [rule]

    workflowsWithSingleJumper = workflowsWithNJumpers 1 system

    workflowsWithNJumpers n sys =
      let (cfg, nLookup, _) = cfGraph sys
       in map (fst . nLookup . fst)
            . filter (\(_, d) -> d == n)
            . Array.assocs
            . Graph.indegree
            $ cfg

removeJumps does two things: first, it finds blocks with only one jumper, and inlines their statements to the jump location. Then it finds blocks to which there are no jumps, and removes them entirely from the program. It uses the workflowsWithNJumpers helper function that uses the control-flow graph of the system to find all workflows to which there are n number of jumps, where n is provided as an input to the function. Note the usage of the toposortWorkflows function here, which makes sure that we remove the blocks in topological order, accumulating as many statements as possible in the final program.

With these functions in place, we write the optimize function:

optimize :: System -> System
optimize =
  applyTillUnchanged
    (removeJumps . inlineRedundantJumps . simplifyWorkflows)
  where
    applyTillUnchanged :: (Eq a) => (a -> a) -> a -> a
    applyTillUnchanged f =
      fix (\recurse x -> if f x == x then x else recurse (f x))

We execute the three optimization functions repeatedly till a fixed point is reached for the resultant System, that is, till there are no further possibilities of optimization.

Finally, we change our main function to apply the optimizations:

main :: IO ()
main = do
  file <- head <$> getArgs
  code <- readFile file
  case parse inputParser code of
    Right (Input system parts) ->
      putStrLn . toC $ Input (optimize system) parts
    Left err -> error err

Compiling the optimized compiler and running it as earlier, generates this C code for the runSystem function now:

bool runSystem(int x, int m, int a, int s) {
  goto in;
in:
  if (s < 1351) { goto px; }
  if (s > 2770) { return true; }
  if (m < 1801) { goto hdj; }
  return false;
px:
  if (a < 2006) { goto qkq; }
  if (m > 2090) { return true; }
  if (s < 537) { return false; }
  if (x > 2440) { return false; }
  return true;
qkq:
  if (x < 1416) { return true; }
  if (x > 2662) { return true; }
  return false;
hdj:
  if (m > 838) { return true; }
  if (a > 1716) { return false; }
  return true;
}

It works well2. We now have 1.7x fewer lines of code as compared to before3.

The Conclusion

This was another attempt to solve Advent of Code problems in somewhat unusual ways. This year we learned some basics of compilation. Swing by next year for more weird ways to solve simple problems.

The full code for this post is available here.


  1. I love how I have to write XMAS horizontally and vertically a couple of time.↩︎

  2. I’m sure many more optimizations are possible yet. After all, this program is essentially a decision tree.↩︎

  3. For the actual problem input with 522 blocks, the optimizations reduce the LoC by 1.5x.↩︎

If you liked this post, please leave a comment.

by Abhinav Sarkar (abhinav@abhinavsarkar.net) at April 07, 2024 12:00 AM

April 02, 2024

Haskell Interlude

46: effectfully

Roman, known better online as effectfully, is interviewed by Wouter and Joachim. On his path to becoming a Plutus language developer at IOG, he learned English to read Software Foundations,   has encountered many spaceleaks, and used Haskell to prevent robots from killing people.

April 02, 2024 07:00 AM

April 01, 2024

Chris Reade

Graphs, Kites and Darts

Graphs, Kites and Darts

Figure 1: Three Coloured Patches
Figure 1: Three Coloured Patches

Non-periodic tilings with Penrose’s kites and darts

(An updated version, since original posting on Jan 6, 2022)

We continue our investigation of the tilings using Haskell with Haskell Diagrams. What is new is the introduction of a planar graph representation. This allows us to define more operations on finite tilings, in particular forcing and composing.

Previously in Diagrams for Penrose Tiles we implemented tools to create and draw finite patches of Penrose kites and darts (such as the samples depicted in figure 1). The code for this and for the new graph representation and tools described here can be found on GitHub https://github.com/chrisreade/PenroseKiteDart.

To describe the tiling operations it is convenient to work with the half-tiles: LD (left dart), RD (right dart), LK (left kite), RK (right kite) using a polymorphic type HalfTile (defined in a module HalfTile)

data HalfTile rep 
 = LD rep | RD rep | LK rep | RK rep   deriving (Show,Eq)

Here rep is a type variable for a representation to be chosen. For drawing purposes, we chose two-dimensional vectors (V2 Double) and called these Pieces.

type Piece = HalfTile (V2 Double)

The vector represents the join edge of the half tile (see figure 2) and thus the scale and orientation are determined (the other tile edges are derived from this when producing a diagram).

Figure 2: The (half-tile) pieces showing join edges (dashed) and origin vertices (red dots)
Figure 2: The (half-tile) pieces showing join edges (dashed) and origin vertices (red dots)

Finite tilings or patches are then lists of located pieces.

type Patch = [Located Piece]

Both Piece and Patch are made transformable so rotate, and scale can be applied to both and translate can be applied to a Patch. (Translate has no effect on a Piece unless it is located.)

In Diagrams for Penrose Tiles we also discussed the rules for legal tilings and specifically the problem of incorrect tilings which are legal but get stuck so cannot continue to infinity. In order to create correct tilings we implemented the decompose operation on patches.

The vector representation that we use for drawing is not well suited to exploring properties of a patch such as neighbours of pieces. Knowing about neighbouring tiles is important for being able to reason about composition of patches (inverting a decomposition) and to find which pieces are determined (forced) on the boundary of a patch.

However, the polymorphic type HalfTile allows us to introduce our alternative graph representation alongside Pieces.

Tile Graphs

In the module Tgraph.Prelude, we have the new representation which treats half tiles as triangular faces of a planar graph – a TileFace – by specialising HalfTile with a triple of vertices (clockwise starting with the tile origin). For example

LD (1,3,4)       RK (6,4,3)
type Vertex = Int
type TileFace = HalfTile (Vertex,Vertex,Vertex)

When we need to refer to particular vertices from a TileFace we use originV (the first vertex – red dot in figure 2), oppV (the vertex at the opposite end of the join edge – dashed edge in figure 2), wingV (the remaining vertex not on the join edge).

originV, oppV, wingV :: TileFace -> Vertex

Tgraphs

The Tile Graphs implementation uses a newtype Tgraph which is a list of tile faces.

newtype Tgraph = Tgraph [TileFace]
                 deriving (Show)

faces :: Tgraph -> [TileFace]
faces (Tgraph fcs) = fcs

For example, fool (short for a fool’s kite) is a Tgraph with 6 faces (and 7 vertices), shown in figure 3.

fool = Tgraph [RD (1,2,3),LD (1,3,4),RK (6,2,5)
              ,LK (6,3,2),RK (6,4,3),LK (6,7,4)
              ]

(The fool is also called an ace in the literature)

Figure 3: fool
Figure 3: fool

With this representation we can investigate how composition works with whole patches. Figure 4 shows a twice decomposed sun on the left and a once decomposed sun on the right (both with vertex labels). In addition to decomposing the right Tgraph to form the left Tgraph, we can also compose the left Tgraph to get the right Tgraph.

Figure 4: sunD2 and sunD
Figure 4: sunD2 and sunD

After implementing composition, we also explore a force operation and an emplace operation to extend tilings.

There are some constraints we impose on Tgraphs.

  • No spurious vertices. The vertices of a Tgraph are the vertices that occur in the faces of the Tgraph (and maxV is the largest number occurring).
  • Connected. The collection of faces must be a single connected component.
  • No crossing boundaries. By this we mean that vertices on the boundary are incident with exactly two boundary edges. The boundary consists of the edges between the Tgraph faces and exterior region(s). This is important for adding faces.
  • Tile connected. Roughly, this means that if we collect the faces of a Tgraph by starting from any single face and then add faces which share an edge with those already collected, we get all the Tgraph faces. This is important for drawing purposes.

In fact, if a Tgraph is connected with no crossing boundaries, then it must be tile connected. (We could define tile connected to mean that the dual graph excluding exterior regions is connected.)

Figure 5 shows two excluded graphs which have crossing boundaries at 4 (left graph) and 13 (right graph). The left graph is still tile connected but the right is not tile connected (the two faces at the top right do not have an edge in common with the rest of the faces.)

Although we have allowed for Tgraphs with holes (multiple exterior regions), we note that such holes cannot be created by adding faces one at a time without creating a crossing boundary. They can be created by removing faces from a Tgraph without necessarily creating a crossing boundary.

Important We are using face as an abbreviation for half-tile face of a Tgraph here, and we do not count the exterior of a patch of faces to be a face. The exterior can also be disconnected when we have holes in a patch of faces and the holes are not counted as faces either. In graph theory, the term face would generally include these other regions, but we will call them exterior regions rather than faces.

Figure 5: A tile-connected graph with crossing boundaries at 4, and a non tile-connected graph
Figure 5: A tile-connected graph with crossing boundaries at 4, and a non tile-connected graph

In addition to the constructor Tgraph we also use

checkedTgraph:: [TileFace] -> Tgraph

which creates a Tgraph from a list of faces, but also performs checks on the required properties of Tgraphs. We can then remove or select faces from a Tgraph and then use checkedTgraph to ensure the resulting Tgraph still satisfies the required properties.

selectFaces, removeFaces  :: [TileFace] -> Tgraph -> Tgraph
selectFaces fcs g = checkedTgraph (faces g `intersect` fcs)
removeFaces fcs g = checkedTgraph (faces g \\ fcs)

Edges and Directed Edges

We do not explicitly record edges as part of a Tgraph, but calculate them as needed. Implicitly we are requiring

  • No spurious edges. The edges of a Tgraph are the edges of the faces of the Tgraph.

To represent edges, a pair of vertices (a,b) is regarded as a directed edge from a to b. A list of such pairs will usually be regarded as a directed edge list. In the special case that the list is symmetrically closed [(b,a) is in the list whenever (a,b) is in the list] we will refer to this as an edge list rather than a directed edge list.

The following functions on TileFaces all produce directed edges (going clockwise round a face).

type Dedge = (Vertex,Vertex)

joinE  :: TileFace -> Dedge  -- join edge - dashed in figure 2
shortE :: TileFace -> Dedge  -- the short edge which is not a join edge
longE  :: TileFace -> Dedge  -- the long edge which is not a join edge
faceDedges :: TileFace -> [Dedge]
  -- all three directed edges clockwise from origin

For the whole Tgraph, we often want a list of all the directed edges of all the faces.

graphDedges :: Tgraph -> [Dedge]
graphDedges = concatMap faceDedges . faces

Because our graphs represent tilings they are planar (can be embedded in a plane) so we know that at most two faces can share an edge and they will have opposite directions of the edge. No two faces can have the same directed edge. So from graphDedges g we can easily calculate internal edges (edges shared by 2 faces) and boundary directed edges (directed edges round the external regions).

internalEdges, boundaryDedges :: Tgraph -> [Dedge]

The internal edges of g are those edges which occur in both directions in graphDedges g. The boundary directed edges of g are the missing reverse directions in graphDedges g.

We also refer to all the long edges of a Tgraph (including kite join edges) as phiEdges (both directions of these edges).

phiEdges :: Tgraph -> [Dedge]

This is so named because, when drawn, these long edges are phi times the length of the short edges (phi being the golden ratio which is approximately 1.618).

Drawing Tgraphs (Patches and VPatches)

The module Tgraph.Convert contains functions to convert a Tgraph to our previous vector representation (Patch) defined in TileLib so we can use the existing tools to produce diagrams.

However, it is convenient to have an intermediate stage (a VPatch = Vertex Patch) which contains both faces and calculated vertex locations (a finite map from vertices to locations). This allows vertex labels to be drawn and for faces to be identified and retained/excluded after the location information is calculated.

data VPatch = VPatch { vLocs :: VertexLocMap
                     , vpFaces::[TileFace]
                     } deriving Show

The conversion functions include

makeVP   :: Tgraph -> VPatch

For drawing purposes we introduced a class Drawable which has a means to create a diagram when given a function to draw Pieces.

class Drawable a where
  drawWith :: (Piece -> Diagram B) -> a -> Diagram B

This allows us to make Patch, VPatch and Tgraph instances of Drawable, and we can define special cases for the most frequently used drawing tools.

draw :: Drawable a => a -> Diagram B
draw = drawWith drawPiece

drawj :: Drawable a => a -> Diagram B
drawj = drawWith dashjPiece

We also need to be able to create diagrams with vertex labels, so we use a draw function modifier

class DrawableLabelled a where
  labelSize :: Measure Double -> (VPatch -> Diagram B) -> a -> Diagram B

Both VPatch and Tgraph are made instances (but not Patch as this no longer has vertex information). The type Measure is defined in Diagrams, but we generally use a default measure for labels to define

labelled :: DrawableLabelled a => (VPatch -> Diagram B) -> a -> Diagram B
labelled = labelSize (normalized 0.018)

This allows us to use, for example (where g is a Tgraph or VPatch)

labelled draw g
labelled drawj g

One consequence of using abstract graphs is that there is no unique predefined way to orient or scale or position the VPatch (and Patch) arising from a Tgraph representation. Our implementation selects a particular join edge and aligns it along the x-axis (unit length for a dart, philength for a kite) and tile-connectedness ensures the rest of the VPatch (and Patch) can be calculated from this.

We also have functions to re-orient a VPatch and lists of VPatchs using chosen pairs of vertices. [Simply doing rotations on the final diagrams can cause problems if these include vertex labels. We do not, in general, want to rotate the labels – so we need to orient the VPatch before converting to a diagram]

Decomposing Graphs

We previously implemented decomposition for patches which splits each half-tile into two or three smaller scale half-tiles.

decompPatch :: Patch -> Patch

We now have a Tgraph version of decomposition in the module Tgraph.Decompose:

decompose :: Tgraph -> Tgraph

Graph decomposition is particularly simple. We start by introducing one new vertex for each long edge (the phiEdges) of the Tgraph. We then build the new faces from each old face using the new vertices.

As a running example we take fool (mentioned above) and its decomposition foolD

*Main> foolD = decompose fool

*Main> foolD
Tgraph [LK (1,8,3),RD (2,3,8),RK (1,3,9)
       ,LD (4,9,3),RK (5,13,2),LK (5,10,13)
       ,RD (6,13,10),LK (3,2,13),RK (3,13,11)
       ,LD (6,11,13),RK (3,14,4),LK (3,11,14)
       ,RD (6,14,11),LK (7,4,14),RK (7,14,12)
       ,LD (6,12,14)
       ]

which are best seen together (fool followed by foolD) in figure 6.

Figure 6: fool and foolD (= decomposeG fool)
Figure 6: fool and foolD (= decompose fool)

Composing Tgraphs, and Unknowns

Composing is meant to be an inverse to decomposing, and one of the main reasons for introducing our graph representation. In the literature, decomposition and composition are defined for infinite tilings and in that context they are unique inverses to each other. For finite patches, however, we will see that composition is not always uniquely determined.

In figure 7 (Two Levels) we have emphasised the larger scale faces on top of the smaller scale faces.

Figure 7: Two Levels
Figure 7: Two Levels

How do we identify the composed tiles? We start by classifying vertices which are at the wing tips of the (smaller) darts as these determine how things compose. In the interior of a graph/patch (e.g in figure 7), a dart wing tip always coincides with a second dart wing tip, and either

  1. the 2 dart halves share a long edge. The shared wing tip is then classified as a largeKiteCentre and is at the centre of a larger kite. (See left vertex type in figure 8), or
  2. the 2 dart halves touch at their wing tips without sharing an edge. This shared wing tip is classified as a largeDartBase and is the base of a larger dart. (See right vertex type in figure 8)
Figure 8: largeKiteCentre (left) and largeDartBase (right)
Figure 8: largeKiteCentre (left) and largeDartBase (right)

[We also call these (respectively) a deuce vertex type and a jack vertex type later in figure 10]

Around the boundary of a Tgraph, the dart wing tips may not share with a second dart. Sometimes the wing tip has to be classified as unknown but often it can be decided by looking at neighbouring tiles. In this example of a four times decomposed sun (sunD4), it is possible to classify all the dart wing tips as a largeKiteCentre or a largeDartBase so there are no unknowns.

If there are no unknowns, then we have a function to produce the unique composed Tgraph.

compose:: Tgraph -> Tgraph

Any correct decomposed Tgraph without unknowns will necessarily compose back to its original. This makes compose a left inverse to decompose provided there are no unknowns.

For example, with an (n times) decomposed sun we will have no unknowns, so these will all compose back up to a sun after n applications of compose. For n=4 (sunD4 – the smaller scale shown in figure 7) the dart wing classification returns 70 largeKiteCentres, 45 largeDartBases, and no unknowns.

Similarly with the simpler foolD example, if we classsify the dart wings we get

largeKiteCentres = [14,13]
largeDartBases = [3]
unknowns = []

In foolD (the right hand Tgraph in figure 6), nodes 14 and 13 are new kite centres and node 3 is a new dart base. There are no unknowns so we can use compose safely

*Main> compose foolD
Tgraph [RD (1,2,3),LD (1,3,4),RK (6,2,5)
       ,RK (6,4,3),LK (6,3,2),LK (6,7,4)
       ]

which reproduces the original fool (left hand Tgraph in figure 6).

However, if we now check out unknowns for fool we get

largeKiteCentres = []
largeDartBases = []
unknowns = [4,2]    

So both nodes 2 and 4 are unknowns. It had looked as though fool would simply compose into two half kites back-to-back (sharing their long edge not their join), but the unknowns show there are other possible choices. Each unknown could become a largeKiteCentre or a largeDartBase.

The question is then what to do with unknowns.

Partial Compositions

In fact our compose resolves two problems when dealing with finite patches. One is the unknowns and the other is critical missing faces needed to make up a new face (e.g the absence of any half dart).

It is implemented using an intermediary function for partial composition

partCompose:: Tgraph -> ([TileFace],Tgraph) 

partCompose will compose everything that is uniquely determined, but will leave out faces round the boundary which cannot be determined or cannot be included in a new face. It returns the faces of the argument Tgraph that were not used, along with the composed Tgraph.

Figure 9 shows the result of partCompose applied to two graphs. [These are force kiteD3 and force dartD3 on the left. Force is described later]. In each case, the excluded faces of the starting Tgraph are shown in pale green, overlaid by the composed Tgraph on the right.

Figure 9: partCompose for two graphs (force kiteD3 top row and force dartD3 bottom row)
Figure 9: partCompose for two graphs (force kiteD3 top row and force dartD3 bottom row)

Then compose is simply defined to keep the composed faces and ignore the unused faces produced by partCompose.

compose:: Tgraph -> Tgraph
compose = snd . partCompose 

This approach avoids making a decision about unknowns when composing, but it may lose some information by throwing away the uncomposed faces.

For correct Tgraphs g, if decompose g has no unknowns, then compose is a left inverse to decompose. However, if we take g to be two kite halves sharing their long edge (not their join edge), then these decompose to fool which produces an empty Tgraph when recomposed. Thus we do not have g = compose (decompose g) in general. On the other hand we do have g = compose (decompose g) for correct whole-tile Tgraphs g (whole-tile means all half-tiles of g have their matching half-tile on their join edge in g)

Later (figure 21) we show another exception to g = compose (decompose g) with an incorrect tiling.

We make use of

selectFacesVP    :: [TileFace] -> VPatch -> VPatch
removeFacesVP    :: [TileFace] -> VPatch -> VPatch

for creating VPatches from selected tile faces of a Tgraph or VPatch. This allows us to represent and draw a list of faces which need not be connected nor satisfy the no crossing boundaries property provided the Tgraph it was derived from had these properties.

Forcing

When building up a tiling, following the rules, there is often no choice about what tile can be added alongside certain tile edges at the boundary. Such additions are forced by the existing patch of tiles and the rules. For example, if a half tile has its join edge on the boundary, the unique mirror half tile is the only possibility for adding a face to that edge. Similarly, the short edge of a left (respectively, right) dart can only be matched with the short edge of a right (respectively, left) kite. We also make use of the fact that only 7 types of vertex can appear in (the interior of) a patch, so on a boundary vertex we sometimes have enough of the faces to determine the vertex type. These are given the following names in the literature (shown in figure 10): sun, star, jack (=largeDartBase), queen, king, ace (=fool), deuce (=largeKiteCentre).

Figure 10: Vertex types
Figure 10: Vertex types

The function

force :: Tgraph -> Tgraph

will add some faces on the boundary that are forced (i.e new faces where there is exactly one possible choice). For example:

  • When a join edge is on the boundary – add the missing half tile to make a whole tile.
  • When a half dart has its short edge on the boundary – add the half kite that must be on the short edge.
  • When a vertex is both a dart origin and a kite wing (it must be a queen or king vertex) – if there is a boundary short edge of a kite half at the vertex, add another kite half sharing the short edge, (this converts 1 kite to 2 and 3 kites to 4 in combination with the first rule).
  • When two half kites share a short edge their common oppV vertex must be a deuce vertex – add any missing half darts needed to complete the vertex.

Figure 11 shows foolDminus (which is foolD with 3 faces removed) on the left and the result of forcing, ie force foolDminus on the right which is the same Tgraph we get from force foolD (modulo vertex renumbering).

foolDminus = 
    removeFaces [RD(6,14,11), LD(6,12,14), RK(5,13,2)] foolD
Figure 11: foolDminus and force foolDminus = force foolD
Figure 11: foolDminus and force foolDminus = force foolD

Figures 12, 13 and 14 illustrate the result of forcing a 5-times decomposed kite, a 5-times decomposed dart, and a 5-times decomposed sun (respectively). The first two figures reproduce diagrams from an article by Roger Penrose illustrating the extent of influence of tiles round a decomposed kite and dart. [Penrose R Tilings and quasi-crystals; a non-local growth problem? in Aperiodicity and Order 2, edited by Jarich M, Academic Press, 1989. (fig 14)].

Figure 12: force kiteD5 with kiteD5 shown in red
Figure 12: force kiteD5 with kiteD5 shown in red
Figure 13: force dartD5 with dartD5 shown in red
Figure 13: force dartD5 with dartD5 shown in red
Figure 14: force sunD5 with sunD5 shown in red
Figure 14: force sunD5 with sunD5 shown in red

In figure 15, the bottom row shows successive decompositions of a dart (dashed blue arrows from right to left), so applying compose to each dart will go back (green arrows from left to right). The black vertical arrows are force. The solid blue arrows from right to left are (force . decompose) being applied to the successive forced Tgraphs. The green arrows in the reverse direction are compose again and the intermediate (partCompose) figures are shown in the top row with the remainder faces in pale green.

Figure 15: Arrows: black = force, green = composeG, solid blue = (force . decomposeG)
Figure 15: Arrows: black = force, green = compose, solid blue = (force . decompose)

Figure 16 shows the forced graphs of the seven vertex types (with the starting Tgraphs in red) along with a kite (top right).

Figure 16: Relating the forced seven vertex types and the kite
Figure 16: Relating the forced seven vertex types and the kite

These are related to each other as shown in the columns. Each Tgraph composes to the one above (an empty Tgraph for the ones in the top row) and the Tgraph below is its forced decomposition. [The rows have been scaled differently to make the vertex types easier to see.]

Adding Faces to a Tgraph

This is technically tricky because we need to discover what vertices (and implicitly edges) need to be newly created and which ones already exist in the Tgraph. This goes beyond a simple graph operation and requires use of the geometry of the faces. We have chosen not to do a full conversion to vectors to work out all the geometry, but instead we introduce a local representation of relative directions of edges at a vertex allowing a simple equality test.

Edge directions

All directions are integer multiples of 1/10th turn (mod 10) so we use these integers for face internal angles and boundary external angles. The face adding process always adds to the right of a given directed edge (a,b) which must be a boundary directed edge. [Adding to the left of an edge (a,b) would mean that (b,a) will be the boundary direction and so we are really adding to the right of (b,a)]. Face adding looks to see if either of the two other edges already exist in the Tgraph by considering the end points a and b to which the new face is to be added, and checking angles.

This allows an edge in a particular sought direction to be discovered. If it is not found it is assumed not to exist. However, the search will be undermined if there are crossing boundaries. In such a case there will be more than two boundary directed edges at the vertex and there is no unique external angle.

Establishing the no crossing boundaries property ensures these failures cannot occur. We can easily check this property for newly created Tgraphs (with checkedTgraph) and the face adding operations cannot create crossing boundaries.

Touching Vertices and Crossing Boundaries

When a new face to be added on (a,b) has neither of the other two edges already in the Tgraph, the third vertex needs to be created. However it could already exist in the Tgraph – it is not on an edge coming from a or b but from another non-local part of the Tgraph. We call this a touching vertex. If we simply added a new vertex without checking for a clash this would create a non-sensible Tgraph. However, if we do check and find an existing vertex, we still cannot add the face using this because it would create a crossing boundary.

Our version of forcing prevents face additions that would create a touching vertex/crossing boundary by calculating the positions of boundary vertices.

No conflicting edges

There is a final (simple) check when adding a new face, to prevent a long edge (phiEdge) sharing with a short edge. This can arise if we force an incorrect Tgraph (as we will see later).

Implementing Forcing

Our order of forcing prioritises updates (face additions) which do not introduce a new vertex. Such safe updates are easy to recognise and they do not require a touching vertex check. Surprisingly, this pretty much removes the problem of touching vertices altogether.

As an illustration, consider foolDMinus again on the left of figure 11. Adding the left dart onto edge (12,14) is not a safe addition (and would create a crossing boundary at 6). However, adding the right dart RD(6,14,11) is safe and creates the new edge (6,14) which then makes the left dart addition safe. In fact it takes some contrivance to come up with a Tgraph with an update that could fail the check during forcing when safe cases are always done first. Figure 17 shows such a contrived Tgraph formed by removing the faces shown in green from a twice decomposed sun on the left. The forced result is shown on the right. When there are no safe cases, we need to try an unsafe one. The four green faces at the bottom are blocked by the touching vertex check. This leaves any one of 9 half-kites at the centre which would pass the check. But after just one of these is added, the check is not needed again. There is always a safe addition to be done at each step until all the green faces are added.

Figure 17: A contrived example requiring a touching vertex check
Figure 17: A contrived example requiring a touching vertex check

Boundary information

The implementation of forcing has been made more efficient by calculating some boundary information in advance. This boundary information uses a type BoundaryState

data BoundaryState
  = BoundaryState
    { boundary    :: [Dedge]
    , bvFacesMap  :: Mapping Vertex [TileFace]
    , bvLocMap    :: Mapping Vertex (Point V2 Double)
    , allFaces    :: [TileFace]
    , nextVertex  :: Vertex
    } deriving (Show)

This records the boundary directed edges (boundary) plus a mapping of the boundary vertices to their incident faces (bvFacesMap) plus a mapping of the boundary vertices to their positions (bvLocMap). It also keeps track of all the faces and the vertex number to use when adding a vertex. The boundary information is easily incremented for each face addition without being recalculated from scratch, and a final Tgraph with all the new faces is easily recovered from the boundary information when there are no more updates.

makeBoundaryState  :: Tgraph -> BoundaryState
recoverGraph  :: BoundaryState -> Tgraph

The saving that comes from using boundary information lies in efficient incremental changes to the boundary information and, of course, in avoiding the need to consider internal faces. As a further optimisation we keep track of updates in a mapping from boundary directed edges to updates, and supply a list of affected edges after an update so the update calculator (update generator) need only revise these. The boundary and mapping are combined in a ForceState.

type UpdateMap = Mapping Dedge Update
type UpdateGenerator = BoundaryState -> [Dedge] -> UpdateMap
data ForceState = ForceState 
       { boundaryState:: BoundaryState
       , updateMap:: UpdateMap 
       }

Forcing then involves using a specific update generator (allUGenerator) and initialising the state, then using the recursive forceAll which keeps doing updates until there are no more, before recovering the final Tgraph.

force:: Tgraph -> Tgraph
force = forceWith allUGenerator

forceWith:: UpdateGenerator -> Tgraph -> Tgraph
forceWith uGen = recoverGraph . boundaryState . 
                 forceAll uGen . initForceState uGen

forceAll :: UpdateGenerator -> ForceState -> ForceState
initForceState :: UpdateGenerator -> Tgraph -> ForceState

In addition to force we can easily define

wholeTiles:: Tgraph -> Tgraph
wholeTiles = forceWith wholeTileUpdates 

which just uses the first forcing rule to make sure every half-tile has a matching other half.

We also have a version of force which counts to a specific number of face additions.

stepForce :: Int -> ForceState -> ForceState

This proved essential in uncovering problems of accumulated inaccuracy in calculating boundary positions (now fixed).

Some Other Experiments

Below we describe results of some experiments using the tools introduced above. Specifically: emplacements, sub-Tgraphs, incorrect tilings, and composition choices.

Emplacements

The finite number of rules used in forcing are based on local boundary vertex and edge information only. We thought we may be able to improve on this by considering a composition and forcing at the next level up before decomposing and forcing again. This thus considers slightly broader local information. In fact we can iterate this process to all the higher levels of composition. Some Tgraphs produce an empty Tgraph when composed so we can regard those as maximal compositions. For example compose fool produces an empty Tgraph.

The idea was to take an arbitrary Tgraph and apply (compose . force) repeatedly to find its maximally composed (non-empty) Tgraph, before applying (force . decompose) repeatedly back down to the starting level (so the same number of decompositions as compositions).

We called the function emplace, and called the result the emplacement of the starting Tgraph as it shows a region of influence around the starting Tgraph.

With earlier versions of forcing when we had fewer rules, emplace g often extended force g for a Tgraph g. This allowed the identification of some new rules. However, since adding the new rules we have not found Tgraphs where the result of force had fewer faces than the result of emplace.

[As an important update, we have now found examples where the result of force strictly includes the result of emplace (modulo vertex renumbering).

Sub-Tgraphs

In figure 18 on the left we have a four times decomposed dart dartD4 followed by two sub-Tgraphs brokenDart and badlyBrokenDart which are constructed by removing faces from dartD4 (but retaining the connectedness condition and the no crossing boundaries condition). These all produce the same forced result (depicted middle row left in figure 15).

Figure 18: dartD4, brokenDart, badlyBrokenDart
Figure 18: dartD4, brokenDart, badlyBrokenDart

However, if we do compositions without forcing first we find badlyBrokenDart fails because it produces a graph with crossing boundaries after 3 compositions. So compose on its own is not always safe, where safe means guaranteed to produce a valid Tgraph from a valid correct Tgraph.

In other experiments we tried force on Tgraphs with holes and on incomplete boundaries around a potential hole. For example, we have taken the boundary faces of a forced, 5 times decomposed dart, then removed a few more faces to make a gap (which is still a valid Tgraph). This is shown at the top in figure 19. The result of forcing reconstructs the complete original forced graph. The bottom figure shows an intermediate stage after 2200 face additions. The gap cannot be closed off to make a hole as this would create a crossing boundary, but the channel does get filled and eventually closes the gap without creating a hole.

Figure 19: Forcing boundary faces with a gap (after 2200 steps)
Figure 19: Forcing boundary faces with a gap (after 2200 steps)

Incorrect Tilings

When we say a Tgraph g is correct (respectively: incorrect), we mean g represents a correct tiling (respectively: incorrect tiling). A simple example of an incorrect Tgraph is a kite with a dart on each side (referred to as a mistake by Penrose) shown on the left of figure 20.

*Main> mistake
Tgraph [RK (1,2,4),LK (1,3,2),RD (3,1,5)
       ,LD (4,6,1),LD (3,5,7),RD (4,8,6)
       ]

If we try to force (or emplace) this Tgraph it produces an error in construction which is detected by the test for conflicting edge types (a phiEdge sharing with a non-phiEdge).

*Main> force mistake
... *** Exception: doUpdate:(incorrect tiling)
Conflicting new face RK (11,1,6)
with neighbouring faces
[RK (9,1,11),LK (9,5,1),RK (1,2,4),LK (1,3,2),RD (3,1,5),LD (4,6,1),RD (4,8,6)]
in boundary
BoundaryState ...

In figure 20 on the right, we see that after successfully constructing the two whole kites on the top dart short edges, there is an attempt to add an RK on edge (1,6). The process finds an existing edge (1,11) in the correct direction for one of the new edges so tries to add the erroneous RK (11,1,6) which fails a noConflicts test.

Figure 20: An incorrect Tgraph (mistake), and the point at which force mistake fails
Figure 20: An incorrect Tgraph (mistake), and the point at which force mistake fails

So it is certainly true that incorrect Tgraphs may fail on forcing, but forcing cannot create an incorrect Tgraph from a correct Tgraph.

If we apply decompose to mistake it produces another incorrect Tgraph (which is similarly detected if we apply force), but will nevertheless still compose back to mistake if we do not try to force.

Interestingly, though, the incorrectness of a Tgraph is not always preserved by decompose. If we start with mistake1 which is mistake with just two of the half darts (and also incorrect) we still get a similar failure on forcing, but decompose mistake1 is no longer incorrect. If we apply compose to the result or force then compose the mistake is thrown away to leave just a kite (see figure 21). This is an example where compose is not a left inverse to either decompose or (force . decompose).

Figure 21: mistake1 with its decomposition, forced decomposition, and recomposed.
Figure 21: mistake1 with its decomposition, forced decomposition, and recomposed.

Composing with Choices

We know that unknowns indicate possible choices (although some choices may lead to incorrect Tgraphs). As an experiment we introduce

makeChoices :: Tgraph -> [Tgraph]

which produces 2^n alternatives for the 2 choices of each of n unknowns (prior to composing). This uses forceLDB which forces an unknown to be a largeDartBase by adding an appropriate joined half dart at the node, and forceLKC which forces an unknown to be a largeKiteCentre by adding a half dart and a whole kite at the node (making up the 3 pieces for a larger half kite).

Figure 22 illustrates the four choices for composing fool this way. The top row has the four choices of makeChoices fool (with the fool shown embeded in red in each case). The bottom row shows the result of applying compose to each choice.

Figure 22: makeChoices fool (top row) and compose of each choice (bottom row)
Figure 22: makeChoices fool (top row) and compose of each choice (bottom row)

In this case, all four compositions are correct tilings. The problem is that, in general, some of the choices may lead to incorrect tilings. More specifically, a choice of one unknown can determine what other unknowns have to become with constraints such as

  • a and b have to be opposite choices
  • a and b have to be the same choice
  • a and b cannot both be largeKiteCentres
  • a and b cannot both be largeDartBases

This analysis of constraints on unknowns is not trivial. The potential exponential results from choices suggests we should compose and force as much as possible and only consider unknowns of a maximal Tgraph.

For calculating the emplacement of a Tgraph, we first find the forced maximal Tgraph before decomposing. We could also consider using makeChoices at this top step when there are unknowns, i.e a version of emplace which produces these alternative results (emplaceChoices)

The result of emplaceChoices is illustrated for foolD in figure 23. The first force and composition is unique producing the fool level at which point we get 4 alternatives each of which compose further as previously illustrated in figure 22. Each of these are forced, then decomposed and forced, decomposed and forced again back down to the starting level. In figure 23 foolD is overlaid on the 4 alternative results. What they have in common is (as you might expect) emplace foolD which equals force foolD and is the graph shown on the right of figure 11.

Figure 23: emplaceChoices foolD
Figure 23: emplaceChoices foolD

Future Work

I am collaborating with Stephen Huggett who suggested the use of graphs for exploring properties of the tilings. We now have some tools to experiment with but we would also like to complete some formalisation and proofs.

It would also be good to establish whether it is true that g is incorrect iff force g fails.

We have other conjectures relating to subgraph ordering of Tgraphs and Galois connections to explore.

by readerunner at April 01, 2024 12:53 PM

Graphs, Kites and Darts – Empires and SuperForce

We have been exploring properties of Penrose’s aperiodic tilings with kites and darts using Haskell.

Previously in Diagrams for Penrose tiles we implemented tools to draw finite tilings using Haskell diagrams. There we also noted that legal tilings are only correct tilings if they can be continued infinitely and are incorrect otherwise. In Graphs, Kites and Darts we introduced a graph representation for finite tilings (Tgraphs) which enabled us to implement operations that use neighbouring tile information. In particular we implemented a force operation to extend a Tgraph on any boundary edge where there is a unique choice for adding a tile.

In this note we find a limitation of force, show a way to improve on it (superForce), and introduce boundary coverings which are used to implement superForce and calculate empires.

Properties of Tgraphs

A Tgraph is a collection of half-tile faces representing a legal tiling and a half-tile face is either an LD (left dart) , RD (right dart), LK (left kite), or RK (right kite) each with 3 vertices to form a triangle. Faces of the Tgraph which are not half-tile faces are considered external regions and those edges round the external regions are the boundary edges of the Tgraph. The half-tile faces in a Tgraph are required to be connected and locally tile-connected which means that there are exactly two boundary edges at any boundary vertex (no crossing boundaries).

As an example Tgraph we show kingGraph (the three darts and two kites round a king vertex), where

  kingGraph = makeTgraph 
    [LD (1,2,3),RD (1,11,2),LD (1,4,5),RD (1,3,4),LD (1,10,11)
    ,RD (1,9,10),LK (9,1,7),RK (9,7,8),RK (5,7,1),LK (5,6,7)
    ]

This is drawn in figure 1 using

  hsep 1 [labelled drawj kingGraph, draw kingGraph]

which shows vertex labels and dashed join edges (left) and without labels and join edges (right). (hsep 1 provides a horizontal seperator of unit length.)

Figure 1: kingGraph with labels and dashed join edges (left) and without (right).
Figure 1: kingGraph with labels and dashed join edges (left) and without (right).

Properties of forcing

We know there are at most two legal possibilities for adding a half-tile on a boundary edge of a Tgraph. If there are zero legal possibilities for adding a half-tile to some boundary edge, we have a stuck tiling/incorrect Tgraph.

Forcing deals with all cases where there is exactly one possibility for extending on a boundary edge according to the legal tiling rules and consistent with the seven possible vertex types. That means forcing either fails at some stage with a stuck Tgraph (indicating the starting Tgraph was incorrect) or it enlarges the starting Tgraph until every boundary edge has exactly two legal possibilities (consistent with the seven vertex types) for adding a half-tile so a choice would need to be made to grow the Tgraph any further.

Figure 2 shows force kingGraph with kingGraph shown red.

Figure 2: force kingGraph with kingGraph shown red.
Figure 2: force kingGraph with kingGraph shown red.

If g is a correct Tgraph, then force g succeeds and the resulting Tgraph will be common to all infinite tilings that extend the finite tiling represented by g. However, we will see that force g is not a greatest lower bound of (infinite) tilings that extend g. Firstly, what is common to all extensions of g may not be a connected collection of tiles. This leads to the concept of empires which we discuss later. Secondly, even if we only consider the connected common region containing g, we will see that we need to go beyond force g to find this, leading to an operation we call superForce.

Our empire and superForce operations are implemented using boundary coverings which we introduce next.

Boundary edge covering

Given a successfully forced Tgraph fg, a boundary edge covering of fg is a list of successfully forced extensions of fg such that

  1. no boundary edge of fg remains on the boundary in each extension, and
  2. the list takes into account all legal choices for extending on each boundary edge of fg.

[Technically this is a covering of the choices round the boundary, but each extension is also a cover of the boundary edges.] Figure 3 shows a boundary edge covering for a forced kingGraph (force kingGraph is shown red in each extension).

Figure 3: A boundary edge covering of force kingGraph.
Figure 3: A boundary edge covering of force kingGraph.

In practice, we do not need to explore both choices for every boundary edge of fg. When one choice is made, it may force choices for other boundary edges, reducing the number of boundary edges we need to consider further.

The main function is boundaryECovering working on a BoundaryState (which is a Tgraph with extra boundary information). It uses covers which works on a list of extensions each paired with the remaining set of the original boundary edges not yet covered. (Initially covers is given a singleton list with the starting boundary state and the full set of boundary edges to be covered.) For each extension in the list, if its uncovered set is empty, that extension is a completed cover. Otherwise covers replaces the extension with further extensions. It picks the (lowest numbered) boundary edge in the uncovered set, tries extending with a half-dart and with a half-kite on that edge, forcing in each case, then pairs each result with its set of remaining uncovered boundary edges before adding the resulting extensions back at the front of the list to be processed again. If one of the choices for a dart/kite leads to an incorrect tiling (a stuck tiling) when forced, that choice is dropped (provided the other choice succeeds). The final list returned consists of all the completed covers.

  boundaryECovering:: BoundaryState -> [BoundaryState]
  boundaryECovering bs = covers [(bs, Set.fromList (boundary bs))]

  covers:: [(BoundaryState, Set.Set Dedge)] -> [BoundaryState]
  covers [] = []
  covers ((bs,es):opens) 
    | Set.null es = bs:covers opens -- bs is complete
    | otherwise   = covers (newcases ++ opens)
       where (de,des) = Set.deleteFindMin es
             newcases = fmap (\b -> (b, commonBdry des b))
                             (atLeastOne $ tryDartAndKite bs de)

Here we have used

  type Try a = Either String a
  tryDartAndKite:: BoundaryState -> Dedge -> [Try BoundaryState]
  atLeastOne    :: [Try a] -> [a]

We frequently use Try as a type for results of partial functions where we need to continue computation if there is a failure. For example we have a version of force (called tryForce) that returns a Try Tgraph so it does not fail by raising an error, but returns a result indicating either an explicit failure situation or a successful result with a final forced Tgraph. The function tryDartAndKite tries adding an appropriate half-dart and half-kite on a given boundary edge, then uses tryForceBoundary (a variant of tryForce which works with boundary states) on each result and returns a list of Try results. The list of Try results is converted with atLeastOne which collects the successful results but will raise an error when there are no successful results.

Boundary vertex covering

You may notice in figure 3 that the top right cover still has boundary vertices of kingGraph on the final boundary. We use a boundary vertex covering rather than a boundary edge covering if we want to exclude these cases. This involves picking a boundary edge that includes such a vertex and continuing the process of growing possible extensions until no boundary vertices of the original remain on the boundary.

Empires

A partial example of an empire was shown in a 1977 article by Martin Gardner 1. The full empire of a finite tiling would consist of the common faces of all the infinite extensions of the tiling. This will include at least the force of the tiling but it is not obviously finite. Here we confine ourselves to the empire in finite local regions.

For example, we can calculate a local empire for a given Tgraph g by finding the common faces of all the extensions in a boundary vertex covering of force g (which we call empire1 g).

This requires an efficient way to compare Tgraphs. We have implemented guided intersection and guided union operations which, when given a common edge starting point for two Tgraphs, proceed to compare the Tgraphs face by face and produce an appropriate relabelling of the second Tgraph to match the first Tgraph only in the overlap where they agree. These operations may also use geometric positioning information to deal with cases where the overlap is not just a single connected region. From these we can return a union as a single Tgraph when it exists, and an intersection as a list of common faces. Since the (guided) intersection of Tgraphs (the common faces) may not be connected, we do not have a resulting Tgraph. However we can arbitrarily pick one of the argument Tgraphs and emphasise which are the common faces in this example Tgraph.

Figure 4 (left) shows empire1 kingGraph where the starting kingGraph is shown in red. The grey-filled faces are the common faces from a boundary vertex covering. We can see that these are not all connected and that the force kingGraph from figure 2 corresponds to the connected set of grey-filled faces around and including the kingGraph in figure 4.

Figure 4: King's empire (level 1 and level 2).
Figure 4: King’s empire (level 1 and level 2).

We call this a level 1 empire because we only explored out as far as the first boundary covering. We could instead, find further boundary coverings for each of the extensions in a boundary covering. This grows larger extensions in which to find common faces. On the right of figure 4 is a level 2 empire (empire2 kingGraph) which finds the intersection of the combined boundary edge coverings of each extension in a boundary edge covering of force kingGraph. Obviously this process could be continued further but, in practice, it is too inefficient to go much further.

SuperForce

We might hope that (when not discovering an incorrect tiling), force g produces the maximal connected component containing g of the common faces of all infinite extensions of g. This is true for the kingGraph as noted in figure 4. However, this is not the case in general.

The problem is that forcing will not discover if one of the two legal choices for extending a resulting boundary edge always leads to an incorrect Tgraph. In such a situation, the other choice would be common to all infinite extensions.

We can use a boundary edge covering to reveal such cases, leading us to a superForce operation. For example, figure 5 shows a boundary edge covering for the forced Tgraph shown in red.

Figure 5: One choice cover.
Figure 5: One choice cover.

This example is particularly interesting because in every case, the leftmost end of the red forced Tgraph has a dart immediately extending it. Why is there no case extending one of the leftmost two red edges with a half-kite? The fact that such cases are missing from the boundary edge covering suggests they are not possible. Indeed we can check this by adding a half-kite to one of the edges and trying to force. This leads to a failure showing that we have an incorrect tiling. Figure 6 illustrates the Tgraph at the point that it is discovered to be stuck (at the bottom left) by forcing.

Figure 6: An incorrect extension.
Figure 6: An incorrect extension.

Our superForce operation starts by forcing a Tgraph. After a successful force, it creates a boundary edge covering for the forced Tgraph and checks to see if there is any boundary edge of the forced Tgraph for which each cover has the same choice. If so, that choice is made to extend the forced Tgraph and the process is repeated by applying superForce to the result. Otherwise, just the result of forcing is returned.

Figure 7 shows a chain of examples (rockets) where superForce has been used. In each case, the starting Tgraph is shown red, the additional faces added by forcing are shown black, and any further extension produced by superForce is shown in blue.

Figure 7: SuperForce rockets.
Figure 7: SuperForce rockets.

Coda

We still do not know if forcing decides that a Tgraph is correct/incorrect. Can we conclude that if force g succeeds then g (and force g) are correct? We found examples (rockets in figure 7) where force succeeds but one of the 2 legal choices for extending on a boundary edge leads to an incorrect Tgraph. If we find an example g where force g succeeds but both legal choices on a boundary edge lead to incorrect Tgraphs we will have a counter-example. If such a g exists then superForce g will raise an error. [The calculation of a boundary edge covering will call atLeastOne where both branches have led to failure for extending on an edge.]

This means that when superForce succeeds every resulting boundary edge has two legal extensions, neither of which will get stuck when forced.

I would like to thank Stephen Huggett who suggested the idea of using graphs to represent tilings and who is working with me on proof problems relating to the kite and dart tilings.

Reference [1] Martin Gardner (1977) MATHEMATICAL GAMES. Scientific American, 236(1), (pages 110 to 121). http://www.jstor.org/stable/24953856

by readerunner at April 01, 2024 12:48 PM

Graphs, Kites and Darts – and Theorems

We continue our exploration of properties of Penrose’s aperiodic tilings with kites and darts using Haskell and Haskell Diagrams.

In this blog we discuss some interesting properties we have discovered concerning the \small\texttt{decompose}, \small\texttt{compose}, and \small\texttt{force} operations along with some proofs.

Index

  1. Quick Recap (including operations \small\texttt{compose}, \small\texttt{force}, \small\texttt{decompose} on Tgraphs)
  2. Composition Problems and a Compose Force Theorem (composition is not a simple inverse to decomposition)
  3. Perfect Composition Theorem (establishing relationships between \small\texttt{compose}, \small\texttt{force}, \small\texttt{decompose})
  4. Multiple Compositions (extending the Compose Force theorem for multiple compositions)
  5. Proof of the Compose Force Theorem (showing \small\texttt{compose} is total on forced Tgraphs)

1. Quick Recap

Haskell diagrams allowed us to render finite patches of tiles easily as discussed in Diagrams for Penrose tiles. Following a suggestion of Stephen Huggett, we found that the description and manipulation of such tilings is greatly enhanced by using planar graphs. In Graphs, Kites and Darts we introduced a specialised planar graph representation for finite tilings of kites and darts which we called Tgraphs (tile graphs). These enabled us to implement operations that use neighbouring tile information and in particular operations \small\texttt{decompose}, \small\texttt{compose}, and \small\texttt{force}.

For ease of reference, we reproduce the half-tiles we are working with here.

Figure 1: Half-tile faces
Figure 1: Half-tile faces

Figure 1 shows the right-dart (RD), left-dart (LD), left-kite (LK) and right-kite (RK) half-tiles. Each has a join edge (shown dotted) and a short edge and a long edge. The origin vertex is shown red in each case. The vertex at the opposite end of the join edge from the origin we call the opp vertex, and the remaining vertex we call the wing vertex.

If the short edges have unit length then the long edges have length \phi (the golden ratio) and all angles are multiples of 36^{\circ} (a tenth turn) with kite halves having  two 2s and a 1, and dart halves having a 3 and two 1s. This geometry of the tiles is abstracted away from at the graph representation level but used when checking validity of tile additions and by the drawing functions.

There are rules for how the tiles can be put together to make a legal tiling (see e.g. Diagrams for Penrose tiles). We defined a Tgraph (in Graphs, Kites and Darts) as a list of such half-tiles which are constrained to form a legal tiling but must also be connected with no crossing boundaries (see below).

As a simple example consider kingGraph (2 kites and 3 darts round a king vertex). We represent each half-tile as a TileFace with three vertex numbers, then apply makeTgraph to the list of ten Tilefaces. The function makeTgraph :: [TileFace] -> Tgraph performs the necessary checks to ensure the result is a valid Tgraph.

kingGraph :: Tgraph
kingGraph = makeTgraph 
  [LD (1,2,3),RD (1,11,2),LD (1,4,5),RD (1,3,4),LD (1,10,11)
  ,RD (1,9,10),LK (9,1,7),RK (9,7,8),RK (5,7,1),LK (5,6,7)
  ]

To view the Tgraph we simply form a diagram (in this case 2 diagrams horizontally separated by 1 unit)

  hsep 1 [labelled drawj kingGraph, draw kingGraph]

and the result is shown in figure 2 with labels and dashed join edges (left) and without labels and join edges (right).

Figure 2: kingGraph with labels and dashed join edges (left) and without (right).
Figure 2: kingGraph with labels and dashed join edges (left) and without (right).

The boundary of the Tgraph consists of the edges of half-tiles which are not shared with another half-tile, so they go round untiled/external regions. The no crossing boundary constraint (equivalently, locally tile-connected) means that a boundary vertex has exactly two incident boundary edges and therefore has a single external angle in the tiling. This ensures we can always locally determine the relative angles of tiles at a vertex. We say a collection of half-tiles is a valid Tgraph if it constitutes a legal tiling but also satisfies the connectedness and no crossing boundaries constraints.

Our key operations on Tgraphs are \small\texttt{decompose}, \small\texttt{force}, and \small\texttt{compose} which are illustrated in figure 3.

Figure 3: decompose, force, and compose
Figure 3: decompose, force, and compose

Figure 3 shows the kingGraph with its decomposition above it (left), the result of forcing the kingGraph (right) and the composition of the forced kingGraph (bottom right).

Decompose

An important property of Penrose dart and kite tilings is that it is possible to divide the half-tile faces of a tiling into smaller half-tile faces, to form a new (smaller scale) tiling.

Figure 4: Decomposition of (left) half-tiles
Figure 4: Decomposition of (left) half-tiles

Figure 4 illustrates the decomposition of a left-dart (top row) and a left-kite (bottom row). With our Tgraph representation we simply introduce new vertices for dart and kite long edges and kite join edges and then form the new faces using these. This does not involve any geometry, because that is taken care of by drawing operations.

Force

Figure 5 illustrates the rules used by our \small\texttt{force} operation (we omit a mirror-reflected version of each rule).

Figure 5: Force rules
Figure 5: Force rules

In each case the yellow half-tile is added in the presence of the other half-tiles shown. The yellow half-tile is forced because, by the legal tiling rules and the seven possible vertex types, there is no choice for adding a different half-tile on the edge where the yellow tile is added.

We call a Tgraph correct if it represents a tiling which can be continued infinitely to cover the whole plane without getting stuck, and incorrect otherwise. Forcing involves adding half-tiles by the illustrated rules round the boundary until either no more rules apply (in which case the result is a forced Tgraph) or a stuck tiling is encountered (in which case an incorrect Tgraph error is raised). Hence \small\texttt{force} is a partial function but total on correct Tgraphs.

Compose: This is discussed in the next section.

2. Composition Problems and a Theorem

Compose Choices

For an infinite tiling, composition is a simple inverse to decomposition. However, for a finite tiling with boundary, composition is not so straight forward. Firstly, we may need to leave half-tiles out of a composition because the necessary parts of a composed half-tile are missing. For example, a half-dart with a boundary short edge or a whole kite with both short edges on the boundary must necessarily be excluded from a composition. Secondly, on the boundary, there can sometimes be a problem of choosing whether a half-dart should compose to become a half-dart or a half-kite. This choice in composing only arises when there is a half-dart with its wing on the boundary but insufficient local information to determine whether it should be part of a larger half-dart or a larger half-kite.

In the literature (see for example 1 and 2) there is an often repeated method for composing (also called inflating). This method always make the kite choice when there is a choice. Whilst this is a sound method for an unbounded tiling (where there will be no choice), we show that this is an unsound method for finite tilings as follows.

Clearly composing should preserve correctness. However, figure 6 (left) shows a correct Tgraph which is a forced queen, but the kite-favouring composition of the forced queen produces the incorrect Tgraph shown in figure 6 (centre). Applying our \small\texttt{force} function to this reveals a stuck tiling and reports an incorrect Tgraph.

Figure 6: An erroneous and a safe composition
Figure 6: An erroneous and a safe composition

Our algorithm (discussed in Graphs, Kites and Darts) detects dart wings on the boundary where there is a choice and classifies them as unknowns. Our composition refrains from making a choice by not composing a half dart with an unknown wing vertex. The rightmost Tgraph in figure 6 shows the result of our composition of the forced queen with the half-tile faces left out of the composition (the remainder faces) shown in green. This avoidance of making a choice (when there is a choice) guarantees our composition preserves correctness.

Compose is a Partial Function

A different composition problem can arise when we consider Tgraphs that are not decompositions of Tgraphs. In general, \small\texttt{compose} is a partial function on Tgraphs.

Figure 7: Composition may fail to produce a Tgraph
Figure 7: Composition may fail to produce a Tgraph

Figure 7 shows a Tgraph (left) with its sucessful composition (centre) and the half-tile faces that would result from a second composition (right) which do not form a valid Tgraph because of a crossing boundary (at vertex 6). Thus composition of a Tgraph may fail to produce a Tgraph when the resulting faces are disconnected or have a crossing boundary.

However, we claim that \small\texttt{compose} is a total function on forced Tgraphs.

Compose Force Theorem

Theorem: Composition of a forced Tgraph produces a valid Tgraph.

We postpone the proof (outline) for this theorem to section 5. Meanwhile we use the result to establish relationships between \small\texttt{compose}, \small\texttt{force}, and \small\texttt{decompose} in the next section.

3. Perfect Composition Theorem

In Graphs, Kites and Darts we produced a diagram showing relationships between multiple decompositions of a dart and the forced versions of these Tgraphs. We reproduce this here along with a similar diagram for multiple decompositions of a kite.

Figure 8: Commuting Diagrams
Figure 8: Commuting Diagrams

In figure 8 we show separate (apparently) commuting diagrams for the dart and for the kite. The bottom rows show the decompositions, the middle rows show the result of forcing the decompositions, and the top rows illustrate how the compositions of the forced Tgraphs work by showing both the composed faces (black edges) and the remainder faces (green edges) which are removed in the composition. The diagrams are examples of some commutativity relationships concerning \small\texttt{force}, \small\texttt{compose} and \small\texttt{decompose} which we will prove.

It should be noted that these diagrams break down if we consider only half-tiles as the starting points (bottom right of each diagram). The decomposition of a half-tile does not recompose to its original, but produces an empty composition. So we do not even have g = (\small\texttt{compose} \cdot \small\texttt{decompose}) \ g in these cases. Forcing the decomposition also results in an empty composition. Clearly there is something special about the depicted cases and it is not merely that they are wholetile complete because the decompositions are not wholetile complete. [Wholetile complete means there are no join edges on the boundary, so every half-tile has its other half.]

Below we have captured the properties that are sufficient for the diagrams to commute as in figure 8. In the proofs we use a partial ordering on Tgraphs (modulo vertex relabelling) which we define next.

Partial ordering of Tgraphs

If g_0 and g_1 are both valid Tgraphs and g_0 consists of a subset of the (half-tile) faces of g_1 we have

\displaystyle g_0 \subseteq g_1

which gives us a partial order on Tgraphs. Often, though, g_0 is only isomorphic to a subset of the faces of g_1, requiring a vertex relabelling to become a subset. In that case we write

\displaystyle g_0 \sqsubseteq g_1

which is also a partial ordering and induces an equivalence of Tgraphs defined by

\displaystyle g_0 \equiv g_1 \text{ if and only if } g_0 \sqsubseteq g_1 \text{ and } g_1 \sqsubseteq g_0

in which case g_0 and g_1 are isomorphic as Tgraphs.

Both \small\texttt{compose} and \small\texttt{decompose} are monotonic with respect to \sqsubseteq meaning:

\displaystyle g_0 \sqsubseteq g_1 \text{ implies } \small\texttt{compose} \ g_0 \sqsubseteq \small\texttt{compose} \ g_1 \text{ and } \small\texttt{decompose} \ g_0 \sqsubseteq \small\texttt{decompose} \ g_1

We also have \small\texttt{force} is monotonic, but only when restricted to correct Tgraphs. Also, when restricted to correct Tgraphs, we have \small\texttt{force} is non decreasing because it only adds faces:

\displaystyle g \sqsubseteq \small\texttt{force} \ g

and \small\texttt{force} is idempotent (forcing a forced correct Tgraph leaves it the same):

\displaystyle (\small\texttt{force} \cdot \small\texttt{force}) \ g \equiv \small\texttt{force} \ g

Composing perfectly and perfect compositions

Definition: A Tgraph g composes perfectly if all faces of g are composable (i.e there are no remainder faces of g when composing).

We note that the composed faces must be a valid Tgraph (connected with no crossing boundaries) if all faces are included in the composition because g has those properties. Clearly, if g composes perfectly then

\displaystyle (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g \equiv g

In general, for arbitrary g where the composition is defined, we only have

\displaystyle (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g \sqsubseteq g

Definition: A Tgraph g' is a perfect composition if \small\texttt{decompose} \ g' composes perfectly.

Clearly if g' is a perfect composition then

\displaystyle (\small\texttt{compose} \cdot \small\texttt{decompose}) \ g' \equiv g'

(We could use equality here because any new vertex labels introduced by \small\texttt{decompose} will be removed by \small\texttt{compose}). In general, for arbitrary g',

\displaystyle (\small\texttt{compose} \cdot \small\texttt{decompose}) \ g' \sqsubseteq g'

Lemma 1: g' is a perfect composition if and only if g' has the following 2 properties:

  1. every half-kite with a boundary join has either a half-dart or a whole kite on the short edge, and
  2. every half-dart with a boundary join has a half-kite on the short edge,

(Proof outline:) Firstly note that unknowns in g (= \small\texttt{decompose} \ g') can only come from boundary joins in g'. The properties 1 and 2 guarantee that g has no unknowns. Since every face of g has come from a decomposed face in g', there can be no faces in g that will not recompose, so g will compose perfectly to g'. Conversely, if g' is a perfect composition, its decomposition g can have no unknowns. This implies boundary joins in g' must satisfy properties 1 and 2. \square

(Note: a perfect composition g' may have unknowns even though its decomposition g has none.)

It is easy to see two special cases:

  1. If g' is wholetile complete then g' is a perfect composition.Proof: Wholetile complete implies no boundary joins which implies properties 1 and 2 in lemma 1 which implies g' is a perfect composition. \square
  2. If g' is a decomposition then g' is a perfect composition.Proof: If g' is a decomposition, then every half-dart has a half-kite on the short edge which implies property 2 of lemma 1. Also, any half-kite with a boundary join in g' must have come from a decomposed half-dart since a decomposed half-kite produces a whole kite with no boundary kite join. So the half-kite must have a half-dart on the short edge which implies property 1 of lemma 1. The two properties imply g' is a perfect composition. \square

We note that these two special cases cover all the Tgraphs in the bottom rows of the diagrams in figure 8. So the Tgraphs in each bottom row are perfect compositions, and furthermore, they all compose perfectly except for the rightmost Tgraphs which have empty compositions.

In the following results we make the assumption that a Tgraph is correct, which guarantees that when \small\texttt{force} is applied, it terminates with a correct Tgraph. We also note that \small\texttt{decompose} preserves correctness as does \small\texttt{compose} (provided the composition is defined).

Lemma 2: If g_f is a forced, correct Tgraph then

\displaystyle (\small\texttt{compose} \cdot \small\texttt{force} \cdot \small\texttt{decompose}) \ g_f \equiv g_f

(Proof outline:) The proof uses a case analysis of boundary and internal vertices of g_f. For internal vertices we just check there is no change at the vertex after (\small\texttt{compose} \cdot \small\texttt{force} \cdot \small\texttt{decompose}) using figure 11 (plus an extra case for the forced star). For boundary vertices we check local contexts similar to those depicted in figure 10 (but including empty composition cases). This reveals there is no local change of the boundary at any boundary vertex, and since this is true for all boundary vertices, there can be no global change. (We omit the full details). \square

Lemma 3: If g' is a perfect composition and a correct Tgraph, then

\displaystyle \small\texttt{force} \ g' \sqsubseteq (\small\texttt{compose} \cdot \small\texttt{force} \cdot \small\texttt{decompose}) \ g'

(Proof outline:) The proof is by analysis of each possible force rule applicable on a boundary edge of g' and checking local contexts to establish that (i) the result of applying (\small\texttt{compose} \cdot \small\texttt{force} \cdot \small\texttt{decompose}) to the local context must include the added half-tile, and (ii) if the added half tile has a new boundary join, then the result must include both halves of the new half-tile. The two properties of perfect compositions mentioned in lemma 1 are critical for the proof. However, since the result of adding a single half-tile may break the condition of the Tgraph being a pefect composition, we need to arrange that half-tiles are completed first then each subsequent half-tile addition is paired with its wholetile completion. This ensures the perfect composition condition holds at each step for a proof by induction. [A separate proof is needed to show that the ordering of applying force rules makes no difference to a final correct Tgraph (apart from vertex relabelling)]. \square

Lemma 4 If g composes perfectly and is a correct Tgraph then

\displaystyle \small\texttt{force} \ g \equiv (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose})\ g

Proof: Assume g composes perfectly and is a correct Tgraph. Since \small\texttt{force} is non-decreasing (with respect to \sqsubseteq on correct Tgraphs)

\displaystyle \small\texttt{compose} \ g \sqsubseteq (\small\texttt{force} \cdot \small\texttt{compose}) \ g

and since \small\texttt{decompose} is monotonic

\displaystyle (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g \sqsubseteq (\small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose}) \ g

Since g composes perfectly, the left hand side is just g, so

\displaystyle g \sqsubseteq (\small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose}) \ g

and since \small\texttt{force} is monotonic (with respect to \sqsubseteq on correct Tgraphs)

\displaystyle (*) \ \ \ \ \ \small\texttt{force} \ g \sqsubseteq (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose}) \ g

For the opposite direction, we substitute \small\texttt{compose} \ g for g' in lemma 3 to get

\displaystyle (\small\texttt{force} \cdot \small\texttt{compose}) \ g \sqsubseteq (\small\texttt{compose} \cdot \small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{compose}) \ g

Then, since (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g \equiv g, we have

\displaystyle (\small\texttt{force} \cdot \small\texttt{compose}) \ g \sqsubseteq (\small\texttt{compose} \cdot \small\texttt{force}) \ g

Apply \small\texttt{decompose} to both sides (using monotonicity)

\displaystyle (\small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose}) \ g \sqsubseteq (\small\texttt{decompose} \cdot \small\texttt{compose} \cdot \small\texttt{force}) \ g

For any g'' for which the composition is defined we have (\small\texttt{decompose} \cdot \small\texttt{compose})\ g'' \sqsubseteq g'' so we get

\displaystyle (\small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose}) \ g \sqsubseteq \small\texttt{force} \ g

Now apply \small\texttt{force} to both sides and note (\small\texttt{force} \cdot \small\texttt{force})\ g \equiv \small\texttt{force} \ g to get

\displaystyle (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose}) \ g \sqsubseteq \small\texttt{force} \ g

Combining this with (*) above proves the required equivalence. \square

Theorem (Perfect Composition): If g composes perfectly and is a correct Tgraph then

\displaystyle (\small\texttt{compose} \cdot \small\texttt{force}) \ g \equiv (\small\texttt{force} \cdot \small\texttt{compose}) \ g

Proof: Assume g composes perfectly and is a correct Tgraph. By lemma 4 we have

\displaystyle \small\texttt{force} \ g \equiv (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose})\ g

Applying \small\texttt{compose} to both sides, gives

\displaystyle (\small\texttt{compose} \cdot \small\texttt{force}) \ g \equiv (\small\texttt{compose} \cdot \small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force} \cdot \small\texttt{compose})\ g

Now by lemma 2, with g_f = (\small\texttt{force} \cdot \small\texttt{compose}) \ g, the right hand side is equivalent to

\displaystyle (\small\texttt{force} \cdot \small\texttt{compose}) \ g

which establishes the result. \square

Corollaries (of the perfect composition theorem):

  1. If g' is a perfect composition and a correct Tgraph then
    \displaystyle \small\texttt{force} \ g' \equiv (\small\texttt{compose} \cdot \small\texttt{force} \cdot \small\texttt{decompose}) \ g'

    Proof: Let g' = \small\texttt{compose} \ g (so g \equiv \small\texttt{decompose} \ g') in the theorem. \square

    [This result generalises lemma 2 because any correct forced Tgraph g_f is necessarily wholetile complete and therefore a perfect composition, and \small\texttt{force} \ g_f \equiv g_f.]

  2. If g' is a perfect composition and a correct Tgraph then
    \displaystyle (\small\texttt{decompose} \cdot \small\texttt{force}) \ g' \sqsubseteq (\small\texttt{force} \cdot \small\texttt{decompose}) \ g'

    Proof: Apply \small\texttt{decompose} to both sides of the previous corollary and note that

    \displaystyle (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g'' \sqsubseteq g'' \textit{ for any } g''

    provided the composition is defined, which it must be for a forced Tgraph by the Compose Force theorem. \square

  3. If g' is a perfect composition and a correct Tgraph then
    \displaystyle (\small\texttt{force} \cdot \small\texttt{decompose}) \ g' \equiv (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force}) \ g'

    Proof: Apply \small\texttt{force} to both sides of the previous corollary noting \small\texttt{force} is monotonic and idempotent for correct Tgraphs

    \displaystyle (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force}) \ g' \sqsubseteq (\small\texttt{force} \cdot \small\texttt{decompose}) \ g'

    From the fact that \small\texttt{force} is non decreasing and \small\texttt{decompose} and \small\texttt{force} are monotonic, we also have

    \displaystyle (\small\texttt{force} \cdot \small\texttt{decompose}) \ g' \sqsubseteq (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force}) \ g'

    Hence combining these two sub-Tgraph results we have

    \displaystyle (\small\texttt{force} \cdot \small\texttt{decompose}) \ g' \equiv (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{force}) \ g'

    \square

It is important to point out that if g is a correct Tgraph and \small\texttt{compose} \ g is a perfect composition then this is not the same as g composes perfectly. It could be the case that g has more faces than (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g and so g could have unknowns. In this case we can only prove that

\displaystyle (\small\texttt{force} \cdot \small\texttt{compose}) \ g \sqsubseteq (\small\texttt{compose} \cdot \small\texttt{force}) \ g

As an example where this is not an equivalence, choose g to be a star. Then its composition is the empty Tgraph (which is still a pefect composition) and so the left hand side is the empty Tgraph, but the right hand side is a sun.

Perfectly composing generators

The perfect composition theorem and lemmas and the three corollaries justify all the commuting implied by the diagrams in figure 8. However, one might ask more general questions like: Under what circumstances do we have (for a correct forced Tgraph g_f)

\displaystyle (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{compose}) \ g_f \equiv g_f

Definition A generator of a correct forced Tgraph g_f is any Tgraph g such that g \sqsubseteq g_f and \small\texttt{force} \ g \equiv g_f.

We can now state that

Corollary If a correct forced Tgraph g_f has a generator which composes perfectly, then

\displaystyle (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{compose}) \ g_f \equiv g_f

Proof: This follows directly from lemma 4 and the perfect composition theorem. \square

As an example where the required generator does not exist, consider the rightmost Tgraph of the middle row in figure 9. It is generated by the Tgraph directly below it, but it has no generator with a perfect composition. The Tgraph directly above it in the top row is the result of applying (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{compose}) which has lost the leftmost dart of the Tgraph.

Figure 9: A Tgraph without a perfectly composing generator
Figure 9: A Tgraph without a perfectly composing generator

We could summarise this section by saying that \small\texttt{compose} can lose information which cannot be recovered by a subsequent \small\texttt{force} and, similarly, \small\texttt{decompose} can lose information which cannot be recovered by a subsequent \small\texttt{force}. We have defined perfect compositions which are the Tgraphs that do not lose information when decomposed and Tgraphs which compose perfectly which are those that do not lose information when composed. Forcing does the same thing at each level of composition (that is it commutes with composition) provided information is not lost when composing.

4. Multiple Compositions

We know from the Compose Force theorem that the composition of a Tgraph that is forced is always a valid Tgraph. In this section we use this and the results from the last section to show that composing a forced, correct Tgraph produces a forced Tgraph.

First we note that:

Lemma 5: The composition of a forced, correct Tgraph is wholetile complete.

Proof: Let g' = \small\texttt{compose} \ g_f where g_f is a forced, correct Tgraph. A boundary join in g' implies there must be a boundary dart wing of the composable faces of g_f. (See for example figure 4 where this would be vertex 2 for the half dart case, and vertex 5 for the half-kite face). This dart wing cannot be an unknown as the half-dart is in the composable faces. However, a known dart wing must be either a large kite centre or a large dart base and therefore internal in the composable faces of g_f (because of the force rules) and therefore not on the boundary in g'. This is a contradiction showing that g' can have no boundary joins and is therefore wholetile complete. \square

Theorem: The composition of a forced, correct Tgraph is a forced Tgraph.

Proof: Let g' = \small\texttt{compose} \ g_f for some forced, correct Tgraph g_f, then g' is wholetile complete (by lemma 5) and therefore a perfect composition. Let g = \small\texttt{decompose} \ g', so g composes perfectly (g' \equiv \small\texttt{compose} \ g). By the perfect composition theorem we have

\displaystyle (**) \ \ \ \ \ (\small\texttt{compose} \cdot \small\texttt{force}) \ g \equiv (\small\texttt{force} \cdot \small\texttt{compose}) \ g \equiv \small\texttt{force} \ g'

We also have

\displaystyle g = \small\texttt{decompose} \ g' = (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g_f \sqsubseteq g_f

Applying \small\texttt{force} to both sides, noting that \small\texttt{force} is monotonic and the identity on forced Tgraphs, we have

\displaystyle \small\texttt{force} \ g \sqsubseteq \small\texttt{force} \ g_f \equiv g_f

Applying \small\texttt{compose} to both sides, noting that \small\texttt{compose} is monotonic, we have

\displaystyle (\small\texttt{compose} \cdot \small\texttt{force}) \ g \sqsubseteq \small\texttt{compose} \ g_f \equiv g'

By (**) above, the left hand side is equivalent to \small\texttt{force} \ g' so we have

\displaystyle \small\texttt{force} \ g' \sqsubseteq g'

but since we also have (\small\texttt{force} being non-decreasing)

\displaystyle g' \sqsubseteq \small\texttt{force} \ g'

we have established that

\displaystyle g' \equiv \small\texttt{force} \ g'

which means g' is a forced Tgraph. \square

This result means that after forcing once we can repeatedly compose creating valid Tgraphs until we reach the empty Tgraph.

We can also use lemma 5 to establish the converse to a previous corollary:

Corollary If a correct forced Tgraph g_f satisfies:

\displaystyle (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{compose}) \ g_f \equiv g_f

then g_f has a generator which composes perfectly.

Proof: By lemma 5, \small\texttt{compose} \ g_f is wholetile complete and hence a perfect composition. This means that (\small\texttt{decompose} \cdot \small\texttt{compose}) \ g_f composes perfectly and it is also a generator for g_f because

\displaystyle (\small\texttt{force} \cdot \small\texttt{decompose} \cdot \small\texttt{compose}) \ g_f \equiv g_f

\square

5. Proof of the Compose Force theorem

Theorem (Compose Force): Composition of a forced Tgraph produces a valid Tgraph.

Proof: For any forced Tgraph we can construct the composed faces. For the result to be a valid Tgraph we need to show no crossing boundaries and connectedness for the composed faces. These are proved separately by case analysis below.

Proof of no crossing boundaries

Assume g_f is a forced Tgraph and that it has a non-empty set of composed faces (we can ignore cases where the composition is empty as the empty Tgraph is valid). Consider a vertex v in the composed faces of g_f and first take the case that v is on the boundary of g_f . We consider the possible local contexts for a vertex v on a forced Tgraph boundary and the nature of the composed faces at v in each case.

Figure 10: Forced Boundary Vertex Contexts
Figure 10: Forced Boundary Vertex Contexts

Figure 10 shows local contexts for a boundary vertex v in a forced Tgraph where the composition is non-empty. In each case v is shown as a red dot, and the composition is shown filled yellow. The cases for v are shown in rows: the first row is for dart origins, the second row is for kite origins, the next two rows are for kite wings, and the last two rows are for kite opps. The dart wing cases are a subset of the kite opp cases, so not repeated, and dart opp vertices are excluded because they cannot be on the boundary of a forced Tgraph. We only show left-hand versions, so there is a mirror symmetric set for right-hand versions.

It is easy to see that there are no crossing boundaries of the composed faces at v in each case. Since any boundary vertex of any forced Tgraph (with a non-empty composition) must match one of these local context cases around the vertex, we can conclude that a boundary vertex of g_f cannot become a crossing boundary in compose \ g_f.

Next take the case where v is an internal vertex of g_f .

Figure 11: Vertex types and their relationships
Figure 11: Vertex types and their relationships

Figure 11 shows relationships between the forced Tgraphs of the 7 (internal) vertex types (plus a kite at the top right). The red faces are those around the vertex type and the black faces are those produced by forcing (if any). Each forced Tgraph has its composition directly above with empty compositions for the top row. We note that a (forced) star, jack, king, and queen vertex remains an internal vertex in the respective composition so cannot become a crossing boundary vertex. A deuce vertex becomes the centre of a larger kite and is no longer present in the composition (top right). That leaves cases for the sun vertex and ace vertex (=fool vertex). The sun Tgraph (sunGraph) and fool Tgraph (fool) consist of just the red faces at the respective vertex (shown top left and top centre). These both have empty compositions when there is no surrounding context. We thus need to check possible forced local contexts for sunGraph and fool.

The fool case is simple and similar to a duece vertex in that it is never part of a composition. [To see this consider inverting the decomposition arrows shown in figure 4. In both cases we see the half-dart opp vertex (labelled 4 in figure 4) is removed].

For the sunGraph there are only 7 local forced context cases to consider where the sun vertex is on the boundary of the composition.

Figure 12: Forced Contexts for a sun vertex v where v is on the composition boundary
Figure 12: Forced Contexts for a sun vertex v where v is on the composition boundary

Six of these are shown in figure 12 (the missing one is just a mirror reflection of the fourth case). Again, the relevant vertex v is shown as a red dot and the composed faces are shown filled yellow, so it is easy to check that there is no crossing boundary of the composed faces at v in each case. Every forced Tgraph containing an internal sun vertex where the vertex is on the boundary of the composition must match one of the 7 cases locally round the vertex.

Thus no vertex from g_f can become a crossing boundary vertex in the composed faces and since the vertices of the composed faces are a subset of those of g_f, we can have no crossing boundary vertex in the composed faces.

Proof of Connectedness

Assume g_f is a forced Tgraph as before. We refer to the half-tile faces of g_f that get included in the composed faces as the composable faces and the rest as the remainder faces. We want to prove that the composable faces are connected as this will imply the composed faces are connected.

As before we can ignore cases where the set of composable faces is empty, and assume this is not the case. We study the nature of the remainder faces of g_f. Firstly, we note:

Lemma (remainder faces)

The remainder faces of g_f are made up entirely of groups of half-tiles which are either:

  1. Half-fools (= a half dart and both halves of the kite attached to its short edge) where the other half-fool is entirely composable faces, or
  2. Both halves of a kite with both short edges on the (g_f) boundary (so they are not part of a half-fool) where only the origin is in common with composable faces, or
  3. Whole fools with just the shared kite origin in common with composable faces.
Figure 13: Remainder face groups (cases 1,2, and 3)
Figure 13: Remainder face groups (cases 1,2, and 3)

These 3 cases of remainder face groups are shown in figure 13. In each case the border in common with composable faces is shown yellow and the red edges are necessarily on the boundary of g_f (the black boundary could be on the boundary of g_f or shared with another reamainder face group). [A mirror symmetric version for the first group is not shown.] Examples can be seen in e.g. figure 12 where the first Tgraph has four examples of case 1, and two of case 2, the second has six examples of case 1 and two of case 2, and the fifth Tgraph has an example of case 3 as well as four of case 1. [We omit the detailed proof of this lemma which reasons about what gets excluded in a composition after forcing. However, all the local context cases are included in figure 14 (left-hand versions), where we only show those contexts where there is a non-empty composition.]

We note from the (remainder faces) lemma that the common boundary of the group of remainder faces with the composable faces (shown yellow in figure 13) is just a single vertex in cases 2 and 3. In case 1, the common boundary is just a single edge of the composed faces which is made up of 2 adjacent edges of the composable faces that constitute the join of two half-fools.

This means each (remainder face) group shares boundary with exactly one connected component of the composable faces.

Next we establish that if two (remainder face) groups are connected they must share boundary with the same connected component of the composable faces. We need to consider how each (remainder face) group can be connected with a neighbouring such group. It is enough to consider forced contexts of boundary dart long edges (for cases 1 and 3) and boundary kite short edges (for case 2). The cases where the composition is non-empty all appear in figure 14 (left-hand versions) along with boundary kite long edges (middle two rows) which are not relevant here.

Figure 14: Forced contexts for boundary edges
Figure 14: Forced contexts for boundary edges

We note that, whenever one group of the remainder faces (half-fool, whole-kite, whole-fool) is connected to a neighbouring group of the remainder faces, the common boundary (shared edges and vertices) with the compososable faces is also connected, forming either 2 adjacent composed face boundary edges (= 4 adjacent edges of the composable faces), or a composed face boundary edge and one of its end vertices, or a single composed face boundary vertex.

It follows that any connected collection of the remainder face groups shares boundary with a unique connected component of the composable faces. Since the collection of composable and remainder faces together is connected (g_f is connected) the removal of the remainder faces cannot disconnect the composable faces. For this to happen, at least one connected collection of remainder face groups would have to be connected to more than one connected component of composable faces.

This establishes connectedness of any composition of a forced Tgraph, and this completes the proof of the Compose Force theorem. \square

References

[1] Martin Gardner (1977) MATHEMATICAL GAMES. Scientific American, 236(1), (pages 110 to 121). http://www.jstor.org/stable/24953856

[2] Grünbaum B., Shephard G.C. (1987) Tilings and Patterns. W. H. Freeman and Company, New York. ISBN 0-7167-1193-1 (Hardback) (pages 540 to 542).

by readerunner at April 01, 2024 12:24 PM

Oleg Grenrus

Implicit arguments

Posted on 2024-04-01 by Oleg Grenrus

In programming languages with sophisticated type systems we easily run into inconvenience of providing many (often type) arguments explicitly. Let's take a simple map function as an example:

map :: forall a b. (a -> b) -> List a -> List b

If we had to always explicitly provide map's arguments, write something like

ys = map @Char @Char toLower xs

we would immediately give up on types, and switch to use some dynamically typed programming language. It wouldn't be fun to state "the obvious" all the time.

Fortunately we know a way (unification) which can be used to infer many such argument. Therefore we can write

ys = map toLower xs

and the type arguments will be inferred by compiler. However we usually are able to be explicit if we want or need to be, e.g. with TypeApplications in GHC Haskell.

Beyond Hindley-Milner

Conor McBride calls a following phenomenon "Milner's Coincidence":

The Hindley-Milner type system achieves the truly awesome coincidence of four distinct distinctions

  • terms vs types
  • explicitly written things vs implicitly written things
  • presence at run-time vs erasure before run-time
  • non-dependent abstraction vs dependent quantification

We’re used to writing terms and leaving types to be inferred. . . and then erased. We’re used to quantifying over type variables with the corresponding type abstraction and application happening silently and statically.

GHC Haskell type-system has been long far more expressive than vanilla Hindley-Milner, and the four distrinctions are already misaligned.

GHC developers are filling the cracks: For example we'll soon 1 get a forall a -> (with an arrow, not a dot) quantifier, which is erased (irrelevant), explicit (visible) dependent quantification. Later we'll get foreach a. and foreach a -> which are retained (i.e. not-erased, relevant) implicit/explicit dependent quantification.

(Agda also has "different" quantifiers: explicit (x : A) -> ... and implicit {y : B} -> ... dependent quantifiers, and erased variants look like (@0 x : A) -> ... and {@0 y : B} -> ....)

In Haskell, if we have a term with implicit quantifier (foo :: forall a. ...), we can use TypeApplications syntax to apply the argument explicitly:

bar = foo @Int

If the quantifier is explicit, we'll (eventually) write just

bar = foo Int

or

bar = foo (type Int)

for now.

Inferred type variables

That all is great, but consider we define a kind-polymorphic2 type like

type ProxyE :: forall k. k -> Type
data ProxyE a = MkProxyE

then when used at type level, forall behaves as previously, constructors

ghci> :kind ProxyE Int
ProxyE Int :: Type

ghci> :kind ProxyE @Type Int
ProxyE @Type Int :: Type

The type of constructor MkProxyE is

ghci> :type ProxyE
ProxyE :: forall k (a :: k). ProxyE @k a

So if we want to create a term of type Proxy Int, we need to provide both k and a arguments:

ghci> :type ProxyE @Type @Int
ProxyE @Type @Int :: ProxyE @(Type) Int

we could also jump over k:

ghci> :type MkProxyE @_ @Int
MkProxyE @_ @Int :: ProxyE @(*) Int

The above skipping over arguments is not convenient, luckily GHC has a feature, created for other needs, which we can (ab)use here. There are inferred variables (though the better name would be "very hidden"), these are arguments for which TypeApplication doesn't apply:

type Proxy :: forall {k}. k -> Type
data Proxy a = MkProxy

This is the way Proxy is defined in base (but I renamed the constructor to avoid name ambiguity)

And while GHCi prints

ghci> :type MkProxy @Int
MkProxy @Int :: Proxy @{Type} Int

the @{A} syntax is not valid Haskell, so we cannot explicitly apply inferred variables. Neither we can in types:

ghci> :kind! Proxy @{Type}

<interactive>:1:10: error: parse error on input ‘Type

I think this is plainly wrong, we should be able to apply these "inferred" arguments too.

The counterargument is that, inferred variables weren't meant to be "more implicit" variables. As GHC manual explains, inferred variables are a solution to TypeApplications with inferred types. We need to know the order of variables to be able to apply them; but especially in presence of type-class constraints the order is arbitrary.

I'm not convinced, I think that ability to be fully explicit is way more important than a chance to write brittle code.

One solution, which I think would work, is simply to not generalise. This is controversial proposal, but as GHC Haskell is moving towards having fancier type system, something needs to be sacrificed. (MonoLocalBinds is for local bindings, but I'd argue that should be for all bindings, not only local).

The challenge has been that library writes may not been aware of TypeApplications, but today they have no choice. Changing from foo :: forall a b. ... to foo :: forall b a. ... may break some code (even though PVP doesn't explicitly write that down, that should be common sense).

So in the GHC manual example

f :: (Eq b, Eq a) => a -> b -> Bool
f x y = (x == x) && (y == y)

g x y = (x == x) && (y == y)

the g would fail to type-check because there are unsolved type-variables. One way to think about this is that GHC would refuse to pick an order of variables. GHC could still generalise if there are no dictionary arguments, but on the other hand I don't think it would help much. It might help more if GHC wouldn't specialise as much, then

h = f

would type-check.

This might sound like we would need to write much many type signatures. I don't think that is true: it's already a best practice to write type signatures for type level bindings, and for local bindings we would mostly need to give signatures to function bindings.

This proposal subsumes monomorphism restriction, recall that without type defaulting:

-- turn off defaulting
default ()
fooLen = genericLength "foo"

will fail to compile with

Ambiguous type variable ‘i0’ arising from a use of ‘genericLength’
prevents the constraint ‘(Num i0)’ from being solved.

error. With NoMonomophismRestriction we have

ghci> :t fooLen
fooLen :: Num i => i

Another, a lot simpler option, is to simply remember whether the symbols' type was inferred, and issue a warning if TypeApplications is used with such symbol in application head. So if user writes

... (g @Int @Char ...)

GHC would warn that g has inferred type, and the TypeApplications with g are brittle. The solution is to give g a type signature. This warning could be issued early in a pipeline (maybe already in renamer), so it would explain further (possibly cryptic) type errors.

Let me summarise the above: If we could apply inferred variables, i.e. use curly brace application syntax, we would have complete explicit forall a ->, implicit forall a. and more implicit forall {a}. dependent quantifiers. Currently the forall {a}. quantifier is incomplete: we can abstract, but we cannot apply. We'll also need some alternative solution to TypeApplicaitons and inferred types. We should be able to bind these variables explicitly in lambda abstractions as well: \ a ->, \ @a -> and \ @{a} -> respectively (see TypeAbstractions).

Alternatives

The three level explicit/implicit/impliciter arguments may feel complicated. Doesn't other languages have similar problems, how they solve them?

As far as I'm aware Agda and Coq resolve this problem by supporting applying implicit arguments by name:

-- using indices instead of parameters,
-- to make constructor behave as in Haskell
data Proxy : {k : Set} (a : k) -> Set1 where
  MkProxy : {k : Set} {a : k} -> Proxy a

t = MkProxy {a = true}

Just adding named arguments to Haskell would be a bad move. It would add another way where a subtle and well-meaning change in the library could break downstream. For example unifying the naming scheme of type-variables in the libraries, so they are always Map k v and not Map k a sometimes, as it is in containers which uses both variable namings.

We could require library authors to explicitly declare that bindings in a module can be applied by name (i.e. that they have thought about the names, and recognise that changing them will be breaking change). You would still be able to always explicitly apply implicit arguments, but sometimes you won't be able to use more convenient named syntax.

It is fair to require library authors to make adjustments so that (numerous) library users would be able to use a new language feature with that library. In a healthy ecosystem that shouldn't be a problem. Specifically it is extra fair, if the alternative is to make feature less great, as then people might not use it at all.

Infinite level of implicitness

Another idea is to embrace implicit, more implicit and even more implicit arguments. Agda has two levels: explicit and implicit, GHC Haskell has two and a half, why stop there?

If we could start fresh, we could pick Agda's function application syntax and have

funE arg    -- explicit application
funI {arg}  -- explicit application of implicit argument

but additionally we could add

funJ {{arg}}    -- explicit application of implicit² argument
funK {{{arg}}}  -- explicit application of implicit³ argument
...             -- and so on

With unlimited levels of implicitness we could define Proxy as

type Proxy :: forall {k} -> k -> Type
data Proxy a where
    MkProxy :: forall {{k}} -> {a :: k} -> Proxy a

and use it as MkProxy, MkProxy {Int} or MkProxy {{Type}} {Int} :: Proxy Int. Unlimited possibilities.

For what it is worth, the implementation should be even simpler than of named arguments.

But I'd be quite happy already if GHC Haskell had a way to explicitly apply any function arguments, be it three levels (ordinary, @arg and @{arg}) of explicitness, many or just two; and figured another way to tackle TypeApplications with inferred types.


  1. GHC-9.10.1 release notes (for alpha1) mention "Partial implementation of the GHC Proposal #281, allowing visible quantification to be used in the types of terms."↩︎

  2. kind is type of types.↩︎

April 01, 2024 12:00 AM

March 18, 2024

Haskell Interlude

45: András Kovács

In this episode, András Kovács is being interviewed by Andres Löh and Matthias Pall Gissurarson. We learn how to go from economics to functional programming, how GHC's runtime system is superior to Rust's, the importance of looking at GHC's Core for spotting stray closures, and why staging might be the answer to all your optimisation problems.

March 18, 2024 08:00 AM

Michael Snoyman

How I Stay Organized

When I describe the Yesod web framework, one of the terms I use is the boundary issue. Internally, I view Yesod as an organized, structured, strongly typed ecosystem. But externally, it's dealing with all the chaos of network traffic. For example, within Yesod, we have clear typing delineations between normal strings, HTML, and raw binary data. But the network layer simply throws around bytes for all three. The boundary issue in Yesod is the idea that, before chaotic, untyped, unorganized data enters the system, it has to be cleaned, sanitized, typed, and then ingested.

This represents my overall organizational system too. I've taken a lot of inspiration from existing approaches, notably Getting Things Done and Inbox Zero. But I don't follow any such philosophy dogmatically. If your goal in reading this blog post is to get organized, I'd recommend reading this, searching for articles on organization, and then determining how you'd like to organize your life.

The process

I like to think of chaotic versus ordered systems. Chaotic systems are sources of stuff: ideas, work items, etc. There are some obvious chaotic sources:

  • Mobile app notifications

  • Incoming emails

  • Phone calls

  • Signal/WhatsApp messages

I think most of us consider these kinds of external interruptions to be chaotic. It doesn't matter what you're in the middle of, the interruption happens and you have to choose how to deal with it. (Note: that may include ignoring it, or putting notifications on silent.)

However, there's another source of chaos, arguably more important than the above: yourself. When I'm sitting working on some code and a thought comes up, it's an internally-driven interruption, and often harder to shake than something external.

Taking heavy inspiration from Getting Things Done, my process is simple for this: record the idea and move on. There are of course caveats to that. If I think of something that demands urgent attention (e.g., "oh shoot I left the food on the stove") chaos will reign. But most of the time, I'm either working on something else, taking a shower, or kicking back reading a book when one of these ideas comes up. The goal is to get the idea into one of the ordered systems so I can let go of it and get back to what I was doing.

For me, my ordered systems are basically my calendar, my todo list, and various reminders from the tools that I use. I'll get into the details of that below.

Other people

How do you treat other people in a system like this? While I think in reality there's a spectrum, we can talk about the extremes:

  • Chaotic people: these are people who don't follow your rules for organization, and will end up randomizing you. This could be a demanding boss, a petulant child, or a telemarketer trying to sell you chaos insurance (I'm sure that's a thing). In these cases, I treat the incoming messages with chaos mode: jot down all work items/ideas, or simply handle them immediately.

  • Ordered people: these are people you can rely on to participate in your system. In an ideal world, this would include your coworkers, close friends and family, etc. With these people, you can trust that "they have the ball" is equivalent to writing down the reminders in your ordered systems.

That's a bit abstract, so let's get concrete. Imagine I'm on a call with a few other developers and we're dividing up the work on the next feature we're implementing. Alice takes work item A, Bob takes work item B, etc. Alice is highly organized, so I rely on her to record the work somewhere (personal todo list, team tracker, Jira... somewhere). But suppose Bob is... less organized. I'd probably either create the Jira issue for Bob and assign it to him, or put a reminder in my own personal systems to follow up and confirm that Bob actually recorded this.

You may think that this kind of redundancy is going overboard. However, I've had to use this technique often to keep projects moving forward. I try as much as possible to encourage others to follow these kinds of organized systems. Project management is, to a large extent, trying to achieve the same goal. But it's important to be honest about other people's capabilities and not rely on them being more organized than they're capable of.

As mentioned, no one is 100% on either the order or chaos side. Even the most chaotic person will often remember to follow up on the most important actions, and even the most ordered will lose track of things from time to time.

Tooling

Once you have the basic system in mind for organizing things, you need to choose appropriate tooling to make it happen. "Tooling" here could be as simple as a paper-and-pen you carry around and write everything down. However, given how bad my handwriting is and the fact that I'm perpetually connected to an electronic device of some kind, I prefer the digital approach.

My tooling choices for organization come down to the following:

Todoist

I use Todoist as my primary todo list application. I've been very happy with it, and the ability to have shared projects has been invaluable. My wife (Miriam, aka LambdaMom) and I use a shared Todoist project for managing topics like purchases for the house, picking up medicines at the pharmacy, filing taxes, etc. And yes, having my spouse be part of the "ordered world" is a wonderful thing. We've given the advice of shared todo lists to many of our friends.

One recommendation if you have a large number of tasks scheduled each day: leverage your todo app's mechanisms for setting priorities and times of day for performing a task. When you have 30 items to cover in a day, including things like "take allergy medicine in the afternoon" and similar, it's easy to miss urgent items. In Todoist, I regularly use the priority feature to push work items to the top.

Calendars

While todo lists track work items and deliverables, calendars track specific times when actions need to be taken: show up to a meeting, go to the doctor, etc. I don't think anyone's too surprised by the idea of using a calendar to stay organized.

Email

Email is another classic organization method. Email is actually a much better ordered system than many other forms of communication, since it has:

  • Unread: things that need to be processed and organized

  • Read in inbox: things that have gone through initial processing but require more work

  • Snooze: for me a killer feature. Plenty of emails do not require immediate attention. In the past I used to create Todoist items for following up on emails that needed more work. But snoozing email is now a common feature in almost every mail system I use, and I rely on it heavily.

Other chat apps

But most communication these days is not happening in email. We have work-oriented chat (like Slack) and personal chat applications (Signal, WhatsApp, etc). My approach to these is:

  • If the app provides a "remind me later" feature, I use it to follow up on things later.

  • If the app doesn't provide such a feature, I add a reminder to Todoist.

Technically I could use "mark as unread" in many cases too. However, I prefer not doing that. You may have noticed that, with the approaches above, you'll very quickly get to 0 active notifications in your apps: no emails waiting to be processed, no messages waiting for a response. You'll have snoozed emails pop up in the future, "remind me later" messages that pop up, and an organized todo list with all the things you need to follow up on.

Notifications and interruptions

This is an area I personally struggle in. Notifications from apps are interruptions, and with the methods above I'm generally able to minimize the impact of an interruption. However, minimizing isn't eliminating: there's still a context switch. Overall, there are two main approaches you can take:

  • Receive all notifications and interruptions and always process them. This makes sure you aren't missing something important and aren't blocking others.

  • Disable notifications while you're in "deep work" and check in occasionally. This allows better work time, but may end up dropping the ball on something important.

For myself, which mode I operate in depends largely on my role. When I'm working as an individual contributor on a codebase, it's less vital to respond immediately, and I may temporarily disable notifications. When I'm leading a project, I try to stay available to answer things immediately to avoid blocking people.

My recommendation here is:

  • Establish some guidelines with the rest of your team about different signaling mechanisms to distinguish between "please answer at some point when you have a chance" and "urgent top priority please answer right now." This can be separate groups/channels with different notification settings, a rule that urgent topics require a phone call, or anything else.

  • Try to use tools that are optimized for avoiding distractions. I've been particularly enamored with Twist recently, which I think nails a sweet spot for this. I'm hoping to follow up with a blog post on team communication tools. (That's actually what originally inspired me to write this post.)

Work organization

I've focused here on personal organization, and the tools I use for that. Organizing things at work falls into similar paradigms. Instead of an individual todo list, at work we'll use project management systems. Instead of tracking messages in WhatsApp, at work it might be Teams. For the most part, the same techniques transfer over directly to the work tools.

One small recommendation: don't overthink the combining/separating of items between work and personal. I went through a period trying to keep the two completely separate, and I've gone through periods of trying to combine it all together. At this point, I simply use whatever tool seems best at the time. That could be a Jira issue, or a Todoist item, or even "remind me later" on a Slack message.

As long as the item is saved and will come up later in a reasonable timeframe, consider the item handled for now, and rely on the fact that it will pop back up (in sprint planning, your daily todo list review, or a notification from Slack) when you need to work on it.

Emotions

A bit of a word of warning for people who really get into organization. It's possible to take things too far, and relate to all impediments to your beautifully organized life as interruptions/distractions/bad things. Sometimes it's completely legitimate to respond with frustration: getting an email from your boss telling you that requirements on a project changed is difficult to deal with, regardless of your organizational system. Having a telemarketer call in the middle of dinner is always unwanted.

But taken too far, a system like this can lead you to interpreting all external interruptions as negative. And it can allow you to get overly upset by people who are disrupting your system by introducing more chaos. Try to avoid letting defense of the system become a new source of stress.

Also, remember that ultimately you are the arbiter of what you will do. Just because someone has sent you an email asking for something doesn't mean you're obligated to create a todo item and follow up. You're free to say no, or (to whatever extent it's appropriate, polite, and professional) simply ignore such requests. You control your life, not your todo program, your inbox, or anyone who knows how to ask for something.

My recommendation: try to remember that this system isn't a goal unto itself. You're trying to make your life better by organizing things. You expect that you won't hit 100%, and that others will not be following the same model. Avoiding the fixation on perfection can make all the difference.

Further reading

For now, I'm just including one "further reading" link. Overall, I really like Todoist as an app, but appreciate even more the thought they put into how the app would tie into a real organizational system. This guide is a good example:

Beyond that, I'd recommend looking up getting things done and inbox zero as search terms. And as I find other articles (or people put them in the comments), I'll consider expanding the list.

March 18, 2024 12:00 AM

March 17, 2024

Oleg Grenrus

ST with an early exit

Posted on 2024-03-17 by Oleg Grenrus

Implementation

I wish there were an early exit functionality in the ST monad. This need comes time to time when writing imperative algorithms in Haskell.

It's very likely there is a functional version of an algorithm, but it might be that ST-version is just simply faster, e.g. by avoiding allocations (as allocating even short lived garbage is not free).

But there are no early exit in the ST monad.

Recent GHC added delimited continuations. The TL;DR is that delimited continuations is somewhat like goto:

  • newPromptTag# creates a label (tag)
  • prompt# brackets the computation
  • control# kind of jumps (goes to) the end of enclosing prompt bracket, and continues from there.

So let's use this functionality to implement a version of ST which has an early exit. It turns out to be quite simple.

The ST monad is define like:

newtype ST s a = ST (State# s -> (# State# s, a #)

and we change it by adding an additional prompt tag argument:

newtype EST e s a = EST
    { unEST :: forall r. PromptTag# (Either e r)
            -> State# s -> (# State# s, a #) 
    }

(Why forall r.? We'll see soon).

It's easy to lift normal ST computations into EST ones:

liftST :: ST s a -> EST e s a
liftST (ST f) = EST (\_ -> f)

so EST is a generalisation of ST, good.

Now we need a way to run EST computations, and also a way to early exit in them.

The early exit is the simpler one. Given that tag prompt brackets the whole computation, we simply jump to the end with Left e. We ignore the captured continuation, we have no use for it.

earlyExitEST :: e -> EST e s any
earlyExitEST e = EST (\tag -> control0## tag (\_k s -> (# s, Left e #)))

Now, the job for runEST is to create the tag and prompt the computation:

runEST :: forall e a. (forall s. EST e s a) -> Either e a
runEST (EST f) = runRW#
    -- create tag
    (\s0 -> case newPromptTag# s0 of {
    -- prompt
    (# s1, tag #) -> case prompt# tag
         -- run the `f` inside prompt,
         -- and once we get to the end return `Right` value
         (\s2 -> case f tag s2 of (# s3, a #) -> (# s3, Right a #)) s1 of {
    (# _, a #) -> a }})

runRW# and forgetting the state at the end is the same as in runST, for comparison:

runST :: (forall s. ST s a) -> a
runST (ST st_rep) = case runRW# st_rep of (# _, a #) -> a
-- See Note [runRW magic] in GHC.CoreToStg.Prep

With all the pieces in place, we can run few simple examples:

-- | >>> ex1
-- Left 'x'
ex1 :: Either Char Bool
ex1 = runEST $ earlyExitEST 'x'

-- | >>> ex2
-- Right True
ex2 :: Either Char Bool
ex2 = runEST (return True)

Comments & wrinkles

Early exit is one of the simplest "effect" you can implement with delimited continuations. This is the throwing part of the exceptions, with only top-level exception handler. It's a nice exercise (and a brain twister) to implement catch blocks.

One wrinkle in this implementation is the control0## (not control0#) function I used. The delimited continuations primops are made to work only with RealWorld, not arbitrary State# tokens.

I think this is unnecessary specialization GHC issue #24165, I was advice to simply use unsafeIOToST, so I did:

control0##
    :: PromptTag# a
    -> (((State# s -> (# State# s, b #)) -> State# s -> (# State# s, a #))
                                         -> State# s -> (# State# s, a #))
    -> State# s -> (# State# s, b #)
control0## = unsafeCoerce# control0#

This still feels silly, especially realizing that the (only) example in the delimited continuations proposal goes like

type role CC nominal representational
newtype CC ans a = CC (State# RealWorld -> (# State# RealWorld, a #))
  deriving (Functor, Applicative, Monad) via IO

runCC :: (forall ans. CC ans a) -> a
runCC (CC m) = case runRW# m of (# _, a #) -> a

but if you look at that, it's just a ST monad done weirdly:

newtype ST s a = ST (State# RealWorld -> (# State# RealWorld, a #))
-- not using `s` argument !?

There might be a good reason why CC should be done like that (other than than primops are RealWorld specific), but the proposal doesn't explain that difference. To me having phantom ans instead of using nominally it as in ST is suspicious.

Conclusion

Delimited continutations are fun and could be very useful.

But surprisingly, at the moment of writing I cannot find any package on Hackage using them for anything! Search for newPromptTag returns only false positives (ghc-lib etc) right now. I wonder why they are unused?

Please try them out!

March 17, 2024 12:00 AM

March 14, 2024

Magnus Therning

Hackage revisions in Nix

Today I got very confused when using callHackageDirect to add the openapi3 package gave me errors like this

> Using Parsec parser
> Configuring openapi3-3.2.3...
> CallStack (from HasCallStack):
>   withMetadata, called at libraries/Cabal/Cabal/src/Distribution/Simple/Ut...
> Error: Setup: Encountered missing or private dependencies:
> base >=4.11.1.0 && <4.18,
> base-compat-batteries >=0.11.1 && <0.13,
> template-haskell >=2.13.0.0 && <2.20

When looking at its entry on Hackage those weren't the version ranges for the dependencies. Also, running ghc-pkg list told me that I already had all required packages at versions matching what Hackage said. So, what's actually happening here?

It took me a while before remembering about revisions but once I did it was clear that callHackageDirect always fetches the initial revision of a package (i.e. it fetches the original tar-ball uploaded by the author). After realising this it makes perfect sense – it's the only revision that's guaranteed to be there and won't change. However, it would be very useful to be able to pick a revision that actually builds.

I'm not the first one to find this, of course. It's been noted and written about on the discource several years ago. What I didn't find though was a way to influence what revision that's picked. It took a bit of rummaging around in the nixpkgs code but finally I found two variables that's used in the Hackage derivation to control this

  • revision - a string with the number of the revision, and
  • editedCabalFile - the SHA256 of the modified Cabal file.

Setting them is done using the overrideCabal function. This is a piece of my setup for a modified set of Haskell packages:

hl = nixpkgs.haskell.lib.compose;

hsPkgs = nixpkgs.haskell.packages.ghc963.override {
  overrides = newpkgs: oldpkgs: {
    openapi3 = hl.overrideCabal (drv: {
      revision = "4";
      editedCabalFile =
        "sha256-a5C58iYrL7eAEHCzinICiJpbNTGwiOFFAYik28et7fI=";
    }) (oldpkgs.callHackageDirect {
      pkg = "openapi3";
      ver = "3.2.3";
      sha256 = "sha256-0F16o3oqOB5ri6KBdPFEFHB4dv1z+Pw6E5f1rwkqwi8=";
    } { });

It's not very ergonomic, and I think an extended version of callHackageDirect would make sense.

March 14, 2024 09:31 PM

March 11, 2024

Joachim Breitner

Convenient sandboxed development environment

I like using one machine and setup for everything, from serious development work to hobby projects to managing my finances. This is very convenient, as often the lines between these are blurred. But it is also scary if I think of the large number of people who I have to trust to not want to extract all my personal data. Whenever I run a cabal install, or a fun VSCode extension gets updated, or anything like that, I am running code that could be malicious or buggy.

In a way it is surprising and reassuring that, as far as I can tell, this commonly does not happen. Most open source developers out there seem to be nice and well-meaning, after all.

Convenient or it won’t happen

Nevertheless I thought I should do something about this. The safest option would probably to use dedicated virtual machines for the development work, with very little interaction with my main system. But knowing me, that did not seem likely to happen, as it sounded like a fair amount of hassle. So I aimed for a viable compromise between security and convenient, and one that does not get too much in the way of my current habits.

For instance, it seems desirable to have the project files accessible from my unconstrained environment. This way, I could perform certain actions that need access to secret keys or tokens, but are (unlikely) to run code (e.g. git push, git pull from private repositories, gh pr create) from “the outside”, and the actual build environment can do without access to these secrets.

The user experience I thus want is a quick way to enter a “development environment” where I can do most of the things I need to do while programming (network access, running command line and GUI programs), with access to the current project, but without access to my actual /home directory.

I initially followed the blog post “Application Isolation using NixOS Containers” by Marcin Sucharski and got something working that mostly did what I wanted, but then a colleague pointed out that tools like firejail can achieve roughly the same with a less “global” setup. I tried to use firejail, but found it to be a bit too inflexible for my particular whims, so I ended up writing a small wrapper around the lower level sandboxing tool https://github.com/containers/bubblewrap.

Selective bubblewrapping

This script, called dev and included below, builds a new filesystem namespace with minimal /proc and /dev directories, it’s own /tmp directories. It then binds-mound some directories to make the host’s NixOS system available inside the container (/bin, /usr, the nix store including domain socket, stuff for OpenGL applications). My user’s home directory is taken from ~/.dev-home and some configuration files are bind-mounted for convenient sharing. I intentionally don’t share most of the configuration – for example, a direnv enable in the dev environment should not affect the main environment. The X11 socket for graphical applications and the corresponding .Xauthority file is made available. And finally, if I run dev in a project directory, this project directory is bind mounted writable, and the current working directory is preserved.

The effect is that I can type dev on the command line to enter “dev mode” rather conveniently. I can run development tools, including graphical ones like VSCode, and especially the latter with its extensions is part of the sandbox. To do a git push I either exit the development environment (Ctrl-D) or open a separate terminal. Overall, the inconvenience of switching back and forth seems worth the extra protection.

Clearly, isn’t going to hold against a determined and maybe targeted attacker (e.g. access to the X11 and the nix daemon socket can probably be used to escape easily). But I hope it will help against a compromised dev dependency that just deletes or exfiltrates data, like keys or passwords, from the usual places in $HOME.

Rough corners

There is more polishing that could be done.

  • In particular, clicking on a link inside VSCode in the container will currently open Firefox inside the container, without access to my settings and cookies etc. Ideally, links would be opened in the Firefox running outside. This is a problem that has a solution in the world of applications that are sandboxed with Flatpak, and involves a bunch of moving parts (a xdg-desktop-portal user service, a filtering dbus proxy, exposing access to that proxy in the container). I experimented with that for a bit longer than I should have, but could not get it to work to satisfaction (even without a container involved, I could not get xdg-desktop-portal to heed my default browser settings…). For now I will live with manually copying and pasting URLs, we’ll see how long this lasts.

  • With this setup (and unlike the NixOS container setup I tried first), the same applications are installed inside and outside. It might be useful to separate the set of installed programs: There is simply no point in running evolution or firefox inside the container, and if I do not even have VSCode or cabal available outside, so that it’s less likely that I forget to enter dev before using these tools.

    It shouldn’t be too hard to cargo-cult some of the NixOS Containers infrastructure to be able to have a separate system configuration that I can manage as part of my normal system configuration and make available to bubblewrap here.

So likely I will refine this some more over time. Or get tired of typing dev and going back to what I did before…

The script

The dev script (at the time of writing)

by Joachim Breitner (mail@joachim-breitner.de) at March 11, 2024 08:39 PM

February 29, 2024

Gabriella Gonzalez

The siren song of domain-specific languages

The siren song of domain-specific languages

I’ve seen a lot of engineering teams mistakenly believe that they can author domain-specific languages for less technical users on a budget. In particular they seem to believe that if they create this domain-specific language then the less technical users will be able to thoughtlessly churn out a bunch of code in that language and there won’t be any problem and they can then move onto the next project. This rarely works out in the way that people hope it will.

In the best case scenario, your less technical users will churn out a large amount of code using your domain-specific language (which is exactly the outcome you hoped for!) and that corpus of code will push the boundaries of what your language is capable of (like performance, compilation speed, features, or supporting integrations). The larger your userbase the greater the demand will be to improve your language in a myriad of ways.

In the worst case scenario your users will find increasingly inane ways to do things wrong with your language despite your best efforts and you will be expected to clean up their mess because you sold the project on the premise of “our users are not going to have to think”.

… and in either case this process will never end; the project will never be in a “done state” and require permanent staffing. Hell, even if you staff an entire team to support this language it’s still often a struggle to keep up with the needs of less technical users.

This tradeoff can still tempt businesses because it’s appealing to replace skilled labor with unskilled labor. The reasoning goes that a small investment of more skilled labor (the authors of the domain-specific language) can enable a larger pool of less skilled labor (the less technical users) to do most of the work. However, what you will often find in practice is that this larger group of less technical users is frequently blocked without continuous assistance from the engineers who created the language.

So in practice you’re not actually replacing skilled labor with unskilled labor. Rather, you’re merely “laundering” skilled labor as unskilled labor and creating more work for your engineers to make them seem more replaceable than they actually are.

I do think there are situations where domain-specific languages make sense, but typically not on the scale of a software engineering organization or even a small product. I personally think this sort of division of labor tends to only work on the scale of an open source ecosystem where you get a large enough economy of scale.

by Gabriella Gonzalez (noreply@blogger.com) at February 29, 2024 12:47 PM

February 22, 2024

Gabriella Gonzalez

Unification-free ("keyword") type checking

Unification-free ("keyword") type checking

From my perspective, one of the biggest open problems in implementing programming languages is how to add a type system to the language without significantly complicating the implementation.

For example, in my tutorial Fall-from-Grace implementation the type checker logic accounts for over half of the code. In the following lines of code report I’ve highlighted the modules responsible for type-checking with a :

$ cloc --by-file src/Grace/*.hs       

--------------------------------------------------------------------------------
File                                    blank        comment           code
--------------------------------------------------------------------------------
src/Grace/Infer.hs        ‡               499            334           1696
src/Grace/Type.hs         ‡                96             91            633
src/Grace/Syntax.hs                        61            163            543
src/Grace/Parser.hs                       166             15            477
src/Grace/Lexer.hs                         69             25            412
src/Grace/Normalize.hs                     47             48            409
src/Grace/Context.hs      ‡                72            165            249
src/Grace/Import.hs                        38              5            161
src/Grace/REPL.hs                          56              4            148
src/Grace/Interpret.hs                     30             28            114
src/Grace/Pretty.hs                        25             25            108
src/Grace/Monotype.hs     ‡                11             48             61
src/Grace/Location.hs                      16             15             60
src/Grace/TH.hs                            23             32             53
src/Grace/Value.hs                         12             53             53
src/Grace/Input.hs                         10              8             43
src/Grace/Compat.hs                         9              2             32
src/Grace/Existential.hs  ‡                12             23             25
src/Grace/Domain.hs       ‡                 4              7             20
--------------------------------------------------------------------------------
SUM:                                     1256           1091           5297
--------------------------------------------------------------------------------

That’s 2684 lines of code (≈51%) just for type-checking (and believe me: I tried very hard to simplify the type-checking code).

This is the reason why programming language implementers will be pretty keen to just not implement a type-checker for their language, and that’s how we end up with a proliferation of untyped programming languages (e.g. Godot or Nix), or ones that end up with a type system bolted on long after the fact (e.g. TypeScript or Python). You can see why someone would be pretty tempted to skip implementing a type system for their language (especially given that it’s an optional language feature) if it’s going to balloon the size of their codebase.

So I’m extremely keen on implementing a “lean” type checker that has a high power-to-weight ratio. I also believe that a compact type checker is an important foundational step for functional programming to “go viral” and displace imperative programming. This post outlines one approach to this problem that I’ve been experimenting with1.

Unification

The thing that bloats the size of most type-checking implementations is the need to track unification variables. These variables are placeholders for storing as-yet-unknown information about something’s type.

For example, when a functional programming language infers the type of something like this Grace expression:

(λx → x) true

… the way it typically works is that it will infer the type of the function (λx → x) which will be:

λx → x : α → α

… where α is a unification variable (an unsolved type). So you can read the above type annotation as saying “the type of λx → x is a function from some unknown input type (α) to the same output type (α).

Then the type checker will infer the type of the function’s input argument (true) which will be:

true : Bool

… and finally the type checker will combine those two pieces of information and reason about the final type like this:

  • the input to the function (true) is a Bool
  • therefore the function’s input type (α) must also be Bool
  • therefore the function’s output type (α) must also be Bool
  • therefore the entire expression’s type is Bool

… which gives the following conclusion of type inference:

(λx → x) true : Bool

However, managing unification variables like α is a lot trickier than it sounds. There are multiple unification algorithms/frameworks in the wild but the problem with all of them is that you have to essentially implement a bespoke logic programming language (with all of the complexity that entails). Like, geez, I’m already implementing a programming language and I don’t want to have to implement a logic programming language on top of that just to power my type-checker.

So there are a couple of ways I’ve been brainstorming how to address this problem and one idea I had was: what if we could get rid of unification variables altogether?

Deleting unification

Alright, so this is the part of the post that requires some familiarity/experience with implementing a type-checker. If you’re somebody new to programming language theory then you can still keep reading but this is where I have to assume some prior knowledge otherwise this post will get way too long.

The basic idea is that you start from the “Complete and Easy” bidirectional type checking algorithm which is a type checking algorithm that does use unification variables2 but is simpler than most type checking algorithms. The type checking rules look like this (you can just gloss over them):

Now, delete all the rules involving unification variables. Yes, all of them. That means that all of the type-checking judgments from Figures 9 and 10 are gone and also quite a few rules from Figure 11 disappear, too.

Surprisingly, you can still type check a lot of code with what’s left, but you lose two important type inference features if you do this:

  • you can no longer infer the types of lambda arguments

  • you can no longer automatically instantiate polymorphic code

… and I’ll dig into those two issues in more detail.

Inferring lambda argument types

You lose the ability to infer the type of a function like this one when you drop support for unification variables:

λx → x == False

Normally, a type checker that supports unification can infer that the above function has type Bool → Bool, but (in general) a type checker can no longer infer that when you drop unification variables from the implementation.

This loss is not too bad (in fact, it’s a pretty common trade-off proposed in the bidirectional type checking literature) because you can make up for it in a few ways (all of which are easy and efficient to implement in a type checker):

  • You can allow the input type to be inferred if the lambda is given an explicit type annotation, like this:

    λx → x == False : BoolBool

    More generally, you can allow the input type to be inferred if the lambda is checked against an expected type (and a type annotation is one case, but not the only case, where a lambda is checked against an expected type).

    We’re going to lean on this pretty heavily because it’s pretty reasonable to ask users to provide type annotations for function definitions and also because there are many situations where we can infer the expected type of a lambda expression from its immediate context.

  • You can allow the user to explicitly supply the type of the argument

    … like this:

    λ(x : Bool) → x == False

    This is how Dhall works, although it’s not as ergonomic.

  • You can allow the input type to be inferred if the lambda is applied to an argument

    This is not that interesting, but I’m mentioning it for completeness. The reason it’s not interesting is because you won’t often see expressions of the form (λx → e) y in the wild, because they can more idiomatically be rewritten as let x = y in e.

Instantiating polymorphic code

The bigger issue with dropping support for unification variables is: all user-defined polymorphic functions now require explicit type abstraction and explicit type application, which is a major regression in the type system’s user experience.

For example, in a language with unification variables you can write the polymorphic identity function as:

λx → x

… and use it like this3:

let id = λx → x
in  (id true, id 1)

… but when you drop support for unification variables then you have to do something like this:

let id = λ(a : Type) → λ(x : a) → x
in  (id Bool true, id Natural 1)

Most programmers do NOT want to program in a language where they have to explicitly manipulate type variables in this way. In particular, they really hate explicit type application. For example, nobody wants to write:

map { x : Bool, … large record … } Bool (λr → r.x) rs

So we need to figure out some way to work around this limitation.

The trick

However, there is a solution that I believe gives a high power-to-weight ratio, which I will refer to as “keyword” type checking:

  • add a bunch of built-in functions

    Specifically, add enough built-in functions to cover most use cases where users would need a polymorphic function.

  • add special type-checking rules for those built-in functions when they’re fully saturated with all of their arguments

    These special-cased type-checking rules would not require unification variables.

  • still require explicit type abstraction when these built-in functions are not fully saturated

    Alternatively, you can require that built-in polymorphic functions are fully saturated with their arguments and make it a parsing error if they’re not.

  • still require explicit type abstraction and explicit type application for all user-defined (i.e. non-builtin) polymorphic functions

  • optionally, turn these built-in functions into keywords or language constructs

I’ll give a concrete example: the map function for lists. In many functional programming languages this map function is not a built-in function; rather it’s defined within the host language as a function of the following type:

map : ∀(a b : Type) → (a → b) → List a → List b

What I’m proposing is that the map function would now become a built-in function within the language and you would now apply a special type-checking rule when the map function is fully saturated:

Γ ⊢ xs ⇒ List a   Γ ⊢ f ⇐ a → b
───────────────────────────────
Γ ⊢ map f xs ⇐ List b

In other words, we’re essentially treating the map built-in function like a “keyword” in our language (when it’s fully saturated). Just like a keyword, it’s a built-in language feature that has special type-checking rules. Hell, you could even make it an actual keyword or language construct (e.g. a list comprehension) instead of a function call.

I would even argue that you should make each of these special-cased builtin-functions a keyword or a language construct instead of a function call (which is why I call this “keyword type checking” in the first place). When viewed through this lens the restrictions that these polymorphic built-in functions (A) are saturated with their arguments and (B) have a special type checking judgment are no different than the restrictions for ordinary keywords or language constructs (which also must be saturated with their arguments and also require special type checking judgments).

To make an analogy, in many functional programming languages the if/then/else construct has this same “keyword” status. You typically don’t implement it as a user-space function of this type:

ifThenElse : ∀(a : Type) → Bool → a → a → a

Rather, you define if as a language construct and you also add a special type-checking rule for if:

Γ ⊢ b ⇐ Bool   Γ ⊢ x ⇒ a   Γ ⊢ y ⇐ a
────────────────────────────────────
Γ ⊢ if b then x else y ⇒ a

… and what I’m proposing is essentially greatly exploding the number of “keywords” in the implementation of the language by turning a whole bunch of commonly-used polymorphic functions into built-in functions (or keywords, or language constructs) that are given special type-checking treatment.

For example, suppose the user were to create a polymorphic function like this one:

let twice = λ(a : Type) → λ(x : a) → [ x, x ]

in  twice (List Bool) (twice Bool true)

That’s not very ergonomic to define and use, but we also can’t reasonably expect our programming language to provide a twice built-in function. However, our language could provide a generally useful replicate builtin function (like Haskell’s replicate function):

replicate : ∀(a : Type) → Natural → a → List a

… with the following type-checking judgment:

Γ ⊢ n ⇐ Natural   Γ ⊢ x ⇒ a
───────────────────────────
Γ ⊢ replicate n x ⇒ List a

… and then you would tell the user to use replicate directly instead of defining their own twice function:

replicate 2 (replicate 2 true)

… and if the user were to ask you “How do I define a twice synonym for replicate 2” you would just tell them “Don’t do that. Use replicate 2 directly.”

Conclusion

This approach has the major upside that it’s much easier to implement a large number of keywords than it is to implement a unification algorithm, but there are other benefits to doing this, too!

  • It discourages complexity and fragmentation in user-space code

    Built-in polymorphic functions have an ergonomic advantage over user-defined polymorphic functions because under this framework type inference works better for built-in functions. This creates an ergonomic incentive to stick to the “standard library” of built-in polymorphic functions, which in turn promotes an opinionated coding style across all code written in that language.

    You might notice that this approach is somewhat similar in spirit to how Go handles polymorphism which is to say: it doesn’t handle user-defined polymorphic code well. For example, Go provides a few built-in language features that support polymorphism (e.g. the map data structure and for loops) but if users ask for any sort of user-defined polymorphism then the maintainers tell them they’re wrong for wanting that. The main difference here is that (unlike Go) we do actually support user-defined polymorphism; it’s not forbidden, but it is less ergonomic than sticking to the built-in utilities that support polymorphism..

  • It improves error messages

    When you special-case the type-checking logic you can also special-case the error messages, too! With general-purpose unification the error message can often be a bit divorced from the user’s intent, but with “keyword type checking” the error message is not only more local to the problem but it can also suggest highly-specific tips or fixes appropriate for that built-in function (or keyword or language construct).

  • It can in some cases more closely match the expectations of imperative programmers

    What I mean is: most programmers coming from an imperative and typed background are used to languages where (most of the time) polymorphism is “supported” via built-in language constructs and keywords and user-defined polymorphism might be supported but considered “fancy”. Leaning on polymorphism via keywords and language constructs would actually make them more comfortable using polymorphism instead of trying to teach them how to produce and consume user-defined polymorphic functions.

    For example, in a lot of imperative languages the idiomatic solution for how to do anything with a list is “use a for loop” where you can think of a for loop as a built-in keyword that supports polymorphic code. The functional programming equivalent of “just use a for loop” would be something like “just use a list comprehension” (where a list comprehension is a “keyword” that supports polymorphic code that we can give special type checking treatment).

That said, this approach is still more brittle than unification and will require more type annotations in general. The goal here isn’t to completely recover the full power of unification but rather to get something that’s not too bad but significantly easier to implement.

I think this “keyword type checking” can potentially occupy a “low tech” point in the type checking design space for functional programming languages that need to have efficient and compact implementations (e.g. for ease of embedding). Also, this can potentially provide a stop-gap solution for novice language implementers that want some sort of a type system but they’re not willing to commit to implementing a unification-based type system.

There’s also variation on this idea which Verity Scheel has been exploring, which is to provide userland support for defining new functions with special type-checking rules and there’s a post from her outlining how to do that:

User Operators with Implicits & Overloads


  1. The other approach is to create essentially an “ABNF for type checkers” that would let you write type-checking judgments in a standard format that could generate the corresponding type-checking code in multiple languages. That’s still a work-in-progress, though.↩︎

  2. I believe some people might take issue with calling these unification variables because they consider bidirectional type checking as a distinct framework from unification. Moreover, in the original bidirectional type checking paper they’re called “unsolved” variables rather than unification variables. However, I feel that for the purpose of this post it’s still morally correct to refer to these unsolved variables as unification variables since their usage and complexity tradeoffs are essentially identical to unification variables in traditional unification algorithms.↩︎

  3. … assuming let expressions are generalized.↩︎

by Gabriella Gonzalez (noreply@blogger.com) at February 22, 2024 04:04 PM

February 03, 2024

Magnus Therning

Bending Warp

In the past I've noticed that Warp both writes to stdout at times and produces some default HTTP responses, but I've never bothered taking the time to look up what possibilities it offers to changes this behaviour. I've also always thought that I ought to find out how Warp handles signals.

If you wonder why this would be interesting to know there are three main points:

  1. The environments where the services run are set up to handle structured logging. In our case it should be JSONL written to stdout, i.e. one JSON object per line.
  2. We've decided that the error responses we produce in our code should be JSON, so it's irritating to have to document some special cases where this isn't true just because Warp has a few default error responses.
  3. Signal handling is, IMHO, a very important part of writing a service that runs well in k8s as it uses signals to handle the lifetime of pods.

Looking through the Warp API

Browsing through the API documentation for Warp it wasn't too difficult to find the interesting pieces, and that Warp follows a fairly common pattern in Haskell libraries

  • There's a function called runSettings that takes an argument of type Settings.
  • The default settings are available in a variable called defaultSettings (not very surprising).
  • There are several functions for modifying the settings and they all have the same shape

    setX :: X -> Settings -> Settings.
    

    which makes it easy to chain them together.

  • The functions I'm interested in now are
    setOnException
    the default handler, defaultOnException, prints the exception to stdout using its Show instance
    setOnExceptionResponse
    the default responses are produced by defaultOnExceptionResponse and contain plain text response bodies
    setInstallShutdownHandler
    the default behaviour is to wait for all ongoing requests and then shut done
    setGracefulShutdownTimeout
    sets the number of seconds to wait for ongoing requests to finnish, the default is to wait indefinitely

Some experimenting

In order to experiment with these I put together a small API using servant, app, with a main function using runSettings and stringing together a bunch of modifications to defaultSettings.

main :: IO ()
main = Log.withLogger $ \logger -> do
    Log.infoIO logger "starting the server"
    runSettings (mySettings logger defaultSettings) (app logger)
    Log.infoIO logger "stopped the server"
  where
    mySettings logger = myShutdownHandler logger . myOnException logger . myOnExceptionResponse

myOnException logs JSON objects (using the logging I've written about before, here and here). It decides wether to log or not using defaultShouldDisplayException, something I copied from defaultOnException.

myOnException :: Log.Logger -> Settings -> Settings
myOnException logger = setOnException handler
  where
    handler mr e = when (defaultShouldDisplayException e) $ case mr of
        Nothing -> Log.warnIO logger $ lm $ "exception: " <> T.pack (show e)
        Just _ -> do
            Log.warnIO logger $ lm $ "exception with request: " <> T.pack (show e)

myExceptionResponse responds with JSON objects. It's simpler than defaultOnExceptionResponse, but it suffices for my learning.

myOnExceptionResponse :: Settings -> Settings
myOnExceptionResponse = setOnExceptionResponse handler
  where
    handler _ =
        responseLBS
            H.internalServerError500
            [(H.hContentType, "application/json; charset=utf-8")]
            (encode $ object ["error" .= ("Something went wrong" :: String)])

Finally, myShutdownHandler installs a handler for SIGTERM that logs and then shuts down.

myShutdownHandler :: Log.Logger -> Settings -> Settings
myShutdownHandler logger = setInstallShutdownHandler shutdownHandler
  where
    shutdownAction = Log.infoIO logger "closing down"
    shutdownHandler closeSocket = void $ installHandler sigTERM (Catch $ shutdownAction >> closeSocket) Nothing

Conclusion

I really ought to have looked into this sooner, especially as it turns out that Warp offers all the knobs and dials I could wish for to control these aspects of its behaviour. The next step is to take this and put it to use in one of the services at $DAYJOB

February 03, 2024 09:16 PM

January 25, 2024

Joachim Breitner

GHC Steering Committee Retrospective

After seven years of service as member and secretary on the GHC Steering Committee, I have resigned from that role. So this is a good time to look back and retrace the formation of the GHC proposal process and committee.

In my memory, I helped define and shape the proposal process, optimizing it for effectiveness and throughput, but memory can be misleading, and judging from the paper trail in my email archives, this was indeed mostly Ben Gamari’s and Richard Eisenberg’s achievement: Already in Summer of 2016, Ben Gamari set up the ghc-proposals Github repository with a sketch of a process and sent out a call for nominations on the GHC user’s mailing list, which I replied to. The Simons picked the first set of members, and in the fall of 2016 we discussed the committee’s by-laws and procedures. As so often, Richard was an influential shaping force here.

Three ingredients

For example, it was him that suggested that for each proposal we have one committee member be the “Shepherd�, overseeing the discussion. I believe this was one ingredient for the process effectiveness: There is always one person in charge, and thus we avoid the delays incurred when any one of a non-singleton set of volunteers have to do the next step (and everyone hopes someone else does it).

The next ingredient was that we do not usually require a vote among all members (again, not easy with volunteers with limited bandwidth and occasional phases of absence). Instead, the shepherd makes a recommendation (accept/reject), and if the other committee members do not complain, this silence is taken as consent, and we come to a decision. It seems this idea can also be traced back on Richard, who suggested that “once a decision is requested, the shepherd [generates] consensus. If consensus is elusive, then we vote.�

At the end of the year we agreed and wrote down these rules, created the mailing list for our internal, but publicly archived committee discussions, and began accepting proposals, starting with Adam Gundry’s OverloadedRecordFields.

At that point, there was no “secretary� role yet, so how I did become one? It seems that in February 2017 I started to clean-up and refine the process documentation, fixing “bugs in the process� (like requiring authors to set Github labels when they don’t even have permissions to do that). This in particular meant that someone from the committee had to manually handle submissions and so on, and by the aforementioned principle that at every step there ought to be exactly one person in change, the role of a secretary followed naturally. In the email in which I described that role I wrote:

Simon already shoved me towards picking up the “secretary� hat, to reduce load on Ben.

So when I merged the updated process documentation, I already listed myself “secretary�.

It wasn’t just Simon’s shoving that put my into the role, though. I dug out my original self-nomination email to Ben, and among other things I wrote:

I also hope that there is going to be clear responsibilities and a clear workflow among the committee. E.g. someone (possibly rotating), maybe called the secretary, who is in charge of having an initial look at proposals and then assigning it to a member who shepherds the proposal.

So it is hardly a surprise that I became secretary, when it was dear to my heart to have a smooth continuous process here.

I am rather content with the result: These three ingredients – single secretary, per-proposal shepherds, silence-is-consent – helped the committee to be effective throughout its existence, even as every once in a while individual members dropped out.

Ulterior motivation

I must admit, however, there was an ulterior motivation behind me grabbing the secretary role: Yes, I did want the committee to succeed, and I did want that authors receive timely, good and decisive feedback on their proposals – but I did not really want to have to do that part.

I am, in fact, a lousy proposal reviewer. I am too generous when reading proposals, and more likely mentally fill gaps in a specification rather than spotting them. Always optimistically assuming that the authors surely know what they are doing, rather than critically assessing the impact, the implementation cost and the interaction with other language features.

And, maybe more importantly: why should I know which changes are good and which are not so good in the long run? Clearly, the authors cared enough about a proposal to put it forward, so there is some need… and I do believe that Haskell should stay an evolving and innovating language… but how does this help me decide about this or that particular feature.

I even, during the formation of the committee, explicitly asked that we write down some guidance on “Vision and Guideline�; do we want to foster change or innovation, or be selective gatekeepers? Should we accept features that are proven to be useful, or should we accept features so that they can prove to be useful? This discussion, however, did not lead to a concrete result, and the assessment of proposals relied on the sum of each member’s personal preference, expertise and gut feeling. I am not saying that this was a mistake: It is hard to come up with a general guideline here, and even harder to find one that does justice to each individual proposal.

So the secret motivation for me to grab the secretary post was that I could contribute without having to judge proposals. Being secretary allowed me to assign most proposals to others to shepherd, and only once in a while myself took care of a proposal, when it seemed to be very straight-forward. Sneaky, ain’t it?

7 Years later

For years to come I happily played secretary: When an author finished their proposal and public discussion ebbed down they would ping me on GitHub, I would pick a suitable shepherd among the committee and ask them to judge the proposal. Eventually, the committee would come to a conclusion, usually by implicit consent, sometimes by voting, and I’d merge the pull request and update the metadata thereon. Every few months I’d summarize the current state of affairs to the committee (what happened since the last update, which proposals are currently on our plate), and once per year gathered the data for Simon Peyton Jones’ annually GHC Status Report. Sometimes some members needed a nudge or two to act. Some would eventually step down, and I’d sent around a call for nominations and when the nominations came in, distributed them off-list among the committee and tallied the votes.

Initially, that was exciting. For a long while it was a pleasant and rewarding routine. Eventually, it became a mere chore. I noticed that I didn’t quite care so much anymore about some of the discussion, and there was a decent amount of naval-gazing, meta-discussions and some wrangling about claims of authority that was probably useful and necessary, but wasn’t particularly fun.

I also began to notice weaknesses in the processes that I helped shape: We could really use some more automation for showing proposal statuses, notifying people when they have to act, and nudging them when they don’t. The whole silence-is-assent approach is good for throughput, but not necessary great for quality, and maybe the committee members need to be pushed more firmly to engage with each proposal. Like GHC itself, the committee processes deserve continuous refinement and refactoring, and since I could not muster the motivation to change my now well-trod secretarial ways, it was time for me to step down.

Luckily, Adam Gundry volunteered to take over, and that makes me feel much less bad for quitting. Thanks for that!

And although I am for my day job now enjoying a language that has many of the things out of the box that for Haskell are still only language extensions or even just future proposals (dependent types, BlockArguments, do notation with (� foo) expressions and 💜 Unicode), I’m still around, hosting the Haskell Interlude Podcast, writing on this blog and hanging out at ZuriHac etc.

by Joachim Breitner (mail@joachim-breitner.de) at January 25, 2024 12:21 AM

January 18, 2024

Michael Snoyman

My Best and Worst Deadlock in Rust

We're going to build up a deadlock together. If you're unfamiliar with Rust and/or its multithreaded concepts, you'll probably learn a lot from this. If you are familiar with Rust's multithreading capabilities, my guess is you'll be as surprised by this deadlock as I was. And if you spot the deadlock immediately, you get a figurative hat-tip from me.

As to the title, this deadlock was the worst I ever experienced because of how subtle it was. It was the best because of the tooling told me exactly where the problem was. You'll see both points come out below.

Access control

If you've read much of my writing, you'll know I almost always introduce a data structure that looks like this:

struct Person {
    name: String,
    age: u32,
}

So we'll do something very similar here! I'm going to simulate some kind of an access control program that allows multiple threads to use some shared, mutable state representing a person. And we'll make two sets of accesses to this state:

  • A read-only thread that checks if the user has access
  • A writer thread that will simulate a birthday and make the person 1 year older

Our access control is really simple: we grant access to people 18 years or older. One way to write this program looks like this:

use std::sync::Arc;

use parking_lot::RwLock;

#[derive(Clone)]
struct Person {
    inner: Arc<RwLock<PersonInner>>,
}

struct PersonInner {
    name: String,
    age: u32,
}

impl Person {
    fn can_access(&self) -> bool {
        const MIN_AGE: u32 = 18;

        self.inner.read().age >= MIN_AGE
    }

    /// Returns the new age
    fn birthday(&self) -> u32 {
        let mut guard = self.inner.write();
        guard.age += 1;
        guard.age
    }
}

fn main() {
    let alice = Person {
        inner: Arc::new(RwLock::new(PersonInner {
            name: "Alice".to_owned(),
            age: 15,
        })),
    };

    let alice_clone = alice.clone();
    std::thread::spawn(move || loop {
        println!("Does the person have access? {}", alice_clone.can_access());
        std::thread::sleep(std::time::Duration::from_secs(1));
    });

    for _ in 0..10 {
        std::thread::sleep(std::time::Duration::from_secs(1));
        let new_age = alice.birthday();

        println!("Happy birthday! Person is now {new_age} years old.");
    }
}

We're using the wonderful parking-lot crate for this example. Since we have one thread which will exclusively read, an RwLock seems like the right data structure to use. It will allow us to take multiple concurrent read locks or one exclusive write lock at a time. For those familiar with it, this is very similar to the general Rust borrow rules, which allow for multiple read-only (or shared) references or a single mutable (or exclusive) reference.

Anyway, we follow a common pattern with our Person data type. It has a single inner field, which contains an Arc and RwLock wrapping around our inner data structure, which contains the actual name and age. Now we can cheaply clone the Person, keep a single shared piece of data in memory for multiple threads, and either read or mutate the values inside.

Next up, to provide nicely encapsulated access, we provide a series of methods on Person that handle the logic of getting read or write locks. In particular, the can_access method takes a read lock, gets the current age, and compares it to the constant value 18. The birthday method takes a write lock and increments the age, returning the new value.

If you run this on your computer, you'll see something like the following output:

Does the person have access? false
Happy birthday! Person is now 16 years old.
Does the person have access? false
Happy birthday! Person is now 17 years old.
Does the person have access? false
Does the person have access? false
Happy birthday! Person is now 18 years old.
Does the person have access? true
Happy birthday! Person is now 19 years old.
Does the person have access? true
Happy birthday! Person is now 20 years old.
Does the person have access? true
Happy birthday! Person is now 21 years old.
Does the person have access? true
Happy birthday! Person is now 22 years old.
Does the person have access? true
Happy birthday! Person is now 23 years old.
Does the person have access? true
Happy birthday! Person is now 24 years old.
Does the person have access? true
Happy birthday! Person is now 25 years old.

The output may look slightly different due to timing differences, but you get the idea. The person, whoever that happens to be, suddenly has access starting at age 18.

NOTE TO READER I'm not going to keep asking this, but I encourage you to look at each code sample and ask: is this the one that introduces the deadlock? I'll give you the answers towards the end of the post.

What's in a name?

It's pretty annoying having now idea who has access. Alice has a name! We should use it. Let's implement a helper method for getting the person's name:

fn get_name(&self) -> &String {
    &self.inner.read().name
}

While this looks nice, it doesn't actually compile:

error[E0515]: cannot return value referencing temporary value
  --> src/main.rs:30:9
   |
30 |         &self.inner.read().name
   |         ^-----------------^^^^^
   |         ||
   |         |temporary value created here
   |         returns a value referencing data owned by the current function

You see, the way an RwLock's read method works is that it returns a RwLockReadGuard. This implements all the borrow rules we want to see at runtime via value creation and dropping. Said more directly: when you call read, it does something like the following:

  1. Waits until it's allowed to take a read guard. For example, if there's an existing write guard active, it will block until that write guard finishes.
  2. Increments a counter somewhere indicating that there's a new active read guard.
  3. Constructs the RwLockReadGuard value.
  4. When that value gets dropped, its Drop impl will decrement that counter.

And this is basically how many interior mutability primitives in Rust work, whether it's an RwLock, Mutex, or RefCell.

The problem with our implementation of get_name is that it tries to take a lock and then borrow a value through the lock. However, when we exit the get_name method it's still holding a reference to the RwLockReadGuard which we're trying to drop. So how do we implement this method? There are a few possibilities:

  • Return the RwLockReadGuard<PersonInner>. This is no longer a get_name method, but now a general purpose "get a read lock" method. It's also unsatisfying because it requires exposing the innards of our inner data structure.
  • Clone the inner String, which is unnecessary allocation.
  • Wrap the name field with an Arc and clone the Arc, which is probably cheaper than cloning the String.

There are really interesting API design points implied by all this, and it would be fun to explore them another time. However, right now, I've got a tight deadline from my boss on the really important feature of print out the person's name, so I better throw together something really quick and direct. And the easiest thing to do is to just lock the RwLock directly wherever we want a name.

We'll make a small tweak to our spawned thread's closure:

std::thread::spawn(move || loop {
    let guard = alice_clone.inner.read();
    println!(
        "Does the {} have access? {}",
        guard.name,
        alice_clone.can_access()
    );
    std::thread::sleep(std::time::Duration::from_secs(1));
});

Delays

The definition of insanity is doing the same thing over and over and expecting different results

- Somebody, but almost certainly not Albert Einstein

By the above definition of insanity, many have pointed out that multithreaded programming is asking the programmer to become insane. You need to expect different results for different runs of a program. That's because the interleaving of actions between two different threads is non-deterministic. Random delays, scheduling differences, and much more can cause a program to behave correctly on one run and completely incorrectly on another. Which is what makes deadlocks so infuriatingly difficult to diagnose and fix.

So let's simulate some of those random delays in our program by pretending that we need to download some super cute loading image while checking access. I've done so with a println call and an extra sleep to simulate the network request time:

    std::thread::spawn(move || loop {
        let guard = alice_clone.inner.read();
        println!("Downloading a cute loading image, please wait...");
        std::thread::sleep(std::time::Duration::from_secs(1));
        println!(
            "Does the {} have access? {}",
            guard.name,
            alice_clone.can_access()
        );
        std::thread::sleep(std::time::Duration::from_secs(1));
    });

And when I run my program, lo and behold, output stops after printing Downloading a cute loading image, please wait.... Maybe the output will be a bit different on your computer, maybe not. That's the nature of the non-deterministic beast. But this appears to be a deadlock.

The best deadlock experience ever

It turns out that the parking-lot crate provides an experimental feature: deadlock detection. When we were facing the real-life deadlock in our production systems, Sibi found this feature and added it to our executable. And boom! The next time our program deadlocked, we immediately got a backtrace pointing us to the exact function where the deadlock occurred. Since it was a release build, we didn't get line numbers, since those had been stripped out. But since I'm doing a debug build for this blog post, we're going to get something even better here.

Let's add in the following code to the top of our main function:

    std::thread::spawn(move || loop {
        std::thread::sleep(std::time::Duration::from_secs(2));
        for deadlock in parking_lot::deadlock::check_deadlock() {
            for deadlock in deadlock {
                println!(
                    "Found a deadlock! {}:\n{:?}",
                    deadlock.thread_id(),
                    deadlock.backtrace()
                );
            }
        }
    });

Every 2 seconds, this background thread will check if parking-lot has detected any deadlocks and print out the thread they occurred in and the full backtrace. (Why 2 seconds? Totally arbitrary. You could use any sleep amount you want.) When I add this to my program, I get some very helpful output. I'll slightly trim the output to not bother with a bunch of uninteresting backtraces outside of the main function:

Found a deadlock! 140559740036800:
   0: parking_lot_core::parking_lot::deadlock_impl::on_unpark
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.9/src/parking_lot.rs:1211:32
   1: parking_lot_core::parking_lot::deadlock::on_unpark
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.9/src/parking_lot.rs:1144:9
   2: parking_lot_core::parking_lot::park::{{closure}}
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.9/src/parking_lot.rs:637:17
   3: parking_lot_core::parking_lot::with_thread_data
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.9/src/parking_lot.rs:207:5
      parking_lot_core::parking_lot::park
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.9/src/parking_lot.rs:600:5
   4: parking_lot::raw_rwlock::RawRwLock::lock_common
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115:17
   5: parking_lot::raw_rwlock::RawRwLock::lock_shared_slow
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:719:9
   6: <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_shared
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:109:26
   7: lock_api::rwlock::RwLock<R,T>::read
             at /home/michael/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.11/src/rwlock.rs:459:9
   8: access_control::Person::can_access
             at src/main.rs:19:9
   9: access_control::main::{{closure}}
             at src/main.rs:59:13
  10: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:154:18

Wow, this gave us a direct pointer to where in our codebase the problem occurs. The deadlock happens in the can_access method, which is called from our println! macro call in main.

In a program of this size, getting a direct link to the relevant code isn't terribly helpful. There were only a few lines that could have possibly caused the deadlock. However, in our production codebase, we have thousands of lines of code in the program itself that could have possibly been related. And it turns out the program itself wasn't even the culprit, it was one of the support libraries we wrote!

Being able to get such direct information on a deadlock is a complete gamechanger for debugging problems of this variety. Absolutely huge props and thanks to the parking-lot team for providing this.

But what's the problem?

OK, now it's time for the worst. We still need to identify what's causing the deadlock. Let's start off with the actual deadlock location: the can_access method:

fn can_access(&self) -> bool {
    const MIN_AGE: u32 = 18;

    self.inner.read().age >= MIN_AGE
}

Is this code, on its own, buggy? Try as I might, I can't possibly find a bug in this code. And there isn't one. This is completely legitimate usage of a read lock. In fact, it's a great demonstration of best practices: we take the lock for as little time as needed, ensuring we free the lock and avoiding contention.

So let's go up the call stack and look at the body of our subthread infinite loop:

let guard = alice_clone.inner.read();
println!("Downloading a cute loading image, please wait...");
std::thread::sleep(std::time::Duration::from_secs(1));
println!(
    "Does the {} have access? {}",
    guard.name,
    alice_clone.can_access()
);
std::thread::sleep(std::time::Duration::from_secs(1));

This code is already pretty suspicious. The first thing that pops out to me when reading this code is the sleeps. We're doing something very inappropriate: holding onto a read lock while sleeping. This is a sure-fire way to cause contention for locks. It would be far superior to only take the locks for a limited period of time. Because lexical scoping leads to drops, and drops lead to freeing locks, one possible implementation would look like this:

println!("Downloading a cute loading image, please wait...");
std::thread::sleep(std::time::Duration::from_secs(1));
{
    let guard = alice_clone.inner.read();
    println!(
        "Does the {} have access? {}",
        guard.name,
        alice_clone.can_access()
    );
}
std::thread::sleep(std::time::Duration::from_secs(1));

This version of the code is an improvement. We've eliminated a legitimate performance issue of over-locking a value. And if you run it, you might see output like the following:

Downloading a cute loading image, please wait...
Happy birthday! Person is now 16 years old.
Does the Alice have access? false
Happy birthday! Person is now 17 years old.
Downloading a cute loading image, please wait...
Happy birthday! Person is now 18 years old.
Does the Alice have access? true
Happy birthday! Person is now 19 years old.
Downloading a cute loading image, please wait...
Happy birthday! Person is now 20 years old.
Does the Alice have access? true
Downloading a cute loading image, please wait...
Happy birthday! Person is now 21 years old.
Happy birthday! Person is now 22 years old.
Does the Alice have access? true
Downloading a cute loading image, please wait...
Happy birthday! Person is now 23 years old.
Does the Alice have access? true
Happy birthday! Person is now 24 years old.
Happy birthday! Person is now 25 years old.
Downloading a cute loading image, please wait...

However, you may also see another deadlock message! So our change is a performance improvement, and makes it more likely for our program to complete without hitting the deadlock. But the deadlock is still present. But where???

Why I thought this isn't a deadlock

It's worth pausing one quick moment before explaining where the deadlock is. (And figurative hat-tip if you already know.) Our program has three threads of execution:

  1. The deadlock detection thread. We know this isn't the cause of the deadlock, because we added that thread after we saw the deadlock. (Though "deadlock detection thread leads to deadlock" would be an appropriately mind-breaking statement to make.)
  2. The access check thread, which only does read locks.
  3. The main thread, where we do the birthday updates. We'll call it the birthday thread instead. This thread takes write locks.

And my assumption going into our debugging adventure is that this is perfectly fine. The birthday thread will keep blocking waiting for a write lock. It will block as long as the access check thread is holding a read lock. OK, that's part of a deadlock: thread B is waiting on thread A. And the check access thread will wait for the birthday thread to release its write lock before it can grab a read lock. That's another component of a deadlock. But it seems like each thread can always complete its locking without waiting on the other thread.

If you don't know what the deadlock is yet, and want to try to figure it out for yourself, go check out the RwLock docs from the standard library. But we'll continue the analysis here.

How many read locks?

At this point in our real-life debugging, Sibi observed something: our code was less efficient than it should be. Focus on this bit of code:

let guard = alice_clone.inner.read();
println!(
    "Does the {} have access? {}",
    guard.name,
    alice_clone.can_access()
);

If we inline the definition of can_access, the problem becomes more obvious:

let guard = alice_clone.inner.read();
println!("Does the {} have access? {}", guard.name, {
    const MIN_AGE: u32 = 18;

    alice_clone.inner.read().age >= MIN_AGE
});

The inefficiency is that we're taking two read locks instead of one! We already read-lock inner to get the name, and then we call alice_clone.can_access() which makes its own lock. This is good from a code reuse standpoint. But it's not good from a resource standpoint. During our debugging session, I agreed that this warranted further investigation, but we continued looking for the deadlock.

Turns out, I was completely wrong. This wasn't just an inefficiency. This is the deadlock. But how? It turns out, I'd missed a very important piece of the documentation for RwLock.

This lock uses a task-fair locking policy which avoids both reader and writer starvation. This means that readers trying to acquire the lock will block even if the lock is unlocked when there are writers waiting to acquire the lock. Because of this, attempts to recursively acquire a read lock within a single thread may result in a deadlock.

Or, to copy from std's docs, we have a demonstration of how to generate a potential deadlock with seemingly innocuous code:

// Thread 1             |  // Thread 2
let _rg = lock.read();  |
                        |  // will block
                        |  let _wg = lock.write();
// may deadlock         |
let _rg = lock.read();  |

This is exactly what our code above was doing: the access check thread took a first read lock to get the name, then took a second read lock inside the can_access method to check the age. By introducing a sleep in between these two actions, we increased the likelihood of the deadlock occurring by giving a wider timespan when the write lock from the birthday thread could come in between those two locks. But the sleep was not the bug. The bug was taking two read locks in the first place!

Let's first try to understand why RwLock behaves like this, and then put together some fixes.

Fairness and starvation

Imagine that, instead of a single access check thread, we had a million of them. Each of them is written so that it grabs a read lock, holds onto it for about 200 milliseconds, and then releases it. With a million such threads, there's a fairly high chance that the birthday thread will never be able to get a write lock. There will always be at least one read lock active.

This problem is starvation: one of the workers in a system is never able to get a lock, and therefore it's starved from doing any work. This can be more than just a performance issue, it can completely undermine the expected behavior of a system. In our case, Alice would remain 15 for the entire lifetime of the program and never be able to access the system.

The solution to starvation is fairness, where you make sure all workers get a chance to do some work. With a simpler data structure like a Mutex, this is relatively easy to think about: everyone who wants a lock stands in line and takes the lock one at a time.

However, RwLocks are more complex. They have both read and write locks, so there's not really just one line to stand in. A naive implementation--meaning what I would have implemented before reading the docs from std and parking-lot--would look like this:

  • read blocks until all write locks are released
  • write blocks until all read and write locks are released

However, the actual implementation with fairness accounted for looks something like this:

  • read blocks if there's an active write lock, or if another thread is waiting for a write lock
  • write blocks until all read and write locks are released

And now we can see the deadlock directly:

  1. Access check thread takes a read lock (for reading the name)
  2. Birthday thread tries to take a write lock, but it can't because there's already a read lock. It stands in line waiting its turn.
  3. Access check thread tries to take a read lock (for checking the age). It sees that there's a write lock waiting in line, and to avoid starving it, stands in line behind the birthday thread
  4. The access check thread is blocked until the birthday thread releases its lock. The birthday thread is blocked until the access check thread releases its first lock. Neither thread can make progress. Deadlock!

This, to me, is the worst deadlock I've encountered. Every single step of this process is logical. The standard library and parking-lot both made the correct decisions about implementation. And it still led to confusing behavior at runtime. Yes, the answer is "you should have read the docs," which I've now done. Consider this blog post an attempt to make sure that everyone else reads the docs too.

OK, so how do we resolve this problem? Let's check out two approaches.

Easiest: read_recursive

The parking-lot crate provides a read_recursive method. Unlike the normal read method, this method will not check if there's a waiting write lock. It will simply grab a read lock. By using read_recursive in our can_access method, we don't have a deadlock anymore. And in this program, we also don't have a risk of starvation, because the read_recursive call is always gated after our thread already got a read lock.

However, this isn't a good general purpose solution. It's essentially undermining all the fairness work that's gone into RwLock. Instead, even though it requires a bit more code change, there's a more idiomatic solution.

Just take one lock

This is the best approach we can take. We only need to take one read lock inside our access check thread. One way to make this work is to move the can_access method from Person to PersonInner, and then call can_access on the guard, like so:

impl PersonInner {
    fn can_access(&self) -> bool {
        const MIN_AGE: u32 = 18;

        self.age >= MIN_AGE
    }
}

// ...


let guard = alice_clone.inner.read();
println!("Downloading a cute loading image, please wait...");
std::thread::sleep(std::time::Duration::from_secs(1));
println!(
    "Does the {} have access? {}",
    guard.name,
    guard.can_access()
);
std::thread::sleep(std::time::Duration::from_secs(1));

This fully resolves the deadlock issue. There are still questions about exposing the innards of our data structure. We could come up with a more complex API that keeps some level of encapsulation, e.g.:

use std::sync::Arc;

use parking_lot::{RwLock, RwLockReadGuard};

#[derive(Clone)]
struct Person {
    inner: Arc<RwLock<PersonInner>>,
}

struct PersonInner {
    name: String,
    age: u32,
}

struct PersonReadGuard<'a> {
    guard: RwLockReadGuard<'a, PersonInner>,
}

impl Person {
    fn read(&self) -> PersonReadGuard {
        PersonReadGuard {
            guard: self.inner.read(),
        }
    }

    /// Returns the new age
    fn birthday(&self) -> u32 {
        let mut guard = self.inner.write();
        guard.age += 1;
        guard.age
    }
}

impl PersonReadGuard<'_> {
    fn can_access(&self) -> bool {
        const MIN_AGE: u32 = 18;

        self.guard.age >= MIN_AGE
    }

    fn get_name(&self) -> &String {
        &self.guard.name
    }
}

fn main() {
    std::thread::spawn(move || loop {
        std::thread::sleep(std::time::Duration::from_secs(2));
        for deadlock in parking_lot::deadlock::check_deadlock() {
            for deadlock in deadlock {
                println!(
                    "Found a deadlock! {}:\n{:?}",
                    deadlock.thread_id(),
                    deadlock.backtrace()
                );
            }
        }
    });

    let alice = Person {
        inner: Arc::new(RwLock::new(PersonInner {
            name: "Alice".to_owned(),
            age: 15,
        })),
    };

    let alice_clone = alice.clone();
    std::thread::spawn(move || loop {
        let guard = alice_clone.read();
        println!("Downloading a cute loading image, please wait...");
        std::thread::sleep(std::time::Duration::from_secs(1));
        println!(
            "Does the {} have access? {}",
            guard.get_name(),
            guard.can_access()
        );
        std::thread::sleep(std::time::Duration::from_secs(1));
    });

    for _ in 0..10 {
        std::thread::sleep(std::time::Duration::from_secs(1));
        let new_age = alice.birthday();

        println!("Happy birthday! Person is now {new_age} years old.");
    }
}

Is this kind of overhead warranted? Definitely not for this case. But such an approach might make sense for larger programs.

So when did we introduce the bug?

Just to fully answer the question I led with: we introduced the deadlock in the section title "What's in a name". In the real life production code, the bug came into existance in almost exactly the same way I described above. We had an existing helper method that took a read lock, then ended up introducing another method that took a read lock on its own and, while that lock was held, called into the existing helper method.

It's very easy to introduce a bug like that. (Or at least that's what I'm telling myself to feel like less of an idiot.) Besides the deadlock problem, it also introduces other race conditions. For example, if I had taken-and-released the read lock in the parent function before calling the helper function, I'd have a different kind of race condition: I'd be pulling data from the same RwLock in a non-atomic manner. Consider if, for example, Alice's name changes to "Alice the Adult" when she turns 18. In the program above, it's entirely possible to imagine a scenario where we say that "Alice the Adult" doesn't have access.

All of this to say: any time you're dealing with locking, you need to be careful to avoid potential data races. Rust makes it so much nicer than many other languages to avoid race conditions through things like RwLockReadGuard, the Send and Sync traits, mutable borrow checking, and other techniques. But it's still not a panacea.

January 18, 2024 12:00 AM

January 15, 2024

Monday Morning Haskell

Functional Programming vs. Object Oriented Programming

Functional Programming (FP) and Object Oriented Programming (OOP) are the two most important programming paradigms in use today. In this article, we'll discuss these two different programming paradigms and compare their key differences, strengths and weaknesses. We'll also highlight a few specific ways Haskell fits into this discussion. Here's a quick outline if you want to skip around a bit!

What is a Programming Paradigm?

A paradigm is a way of thinking about a subject. It's a model against which we can compare examples of something.

In programming, there are many ways to write code to solve a particular task. Our tasks normally involve taking some kind of input, whether data from a database or commands from a user. A program's job is then to produce outputs of some kind, like updates in that database or images on the user's screen.

Programming paradigms help us to organize our thinking so that we can rapidly select an implementation path that makes sense to us and other developers looking at the code. Paradigms also provide mechanisms for reusing code, so that we don't have to start from scratch every time we write a new program.

The two dominant paradigms in programming today are Object Oriented Programming (OOP) and Functional Programming (FP).

The Object Oriented Paradigm

In object oriented programming, our program's main job is to maintain objects. Objects almost always store data, and they have particular ways of acting on other objects and being acted on by other objects (these are the object's methods). Objects often have mutable data - many actions you take on your objects are capable of changing some of the object's underlying data.

Object oriented programming allows code reuse through a system called inheritance. Objects belong to classes which share the same kinds of data and actions. Classes can inherit from a parent class (or multiple classes, depending on the language), so that they also have access to the data from the base class and some of the same code that manipulates it.

The Functional Paradigm

In functional programming, we think about programming in terms of functions. This idea is rooted in the mathematical idea of a function. A function in math is a process which takes some input (or a series of different inputs) and produces some kind of output. A simple example would be a function that takes an input number and produces the square of that number. Many functional languages emphasize pure functions, which produce the exact same output every time when given the same input.

In programming, we may view our entire program as a function. It is a means by which some kind of input (file data or user commands), is transformed into some kind of output (new files, messages on our terminal). Individual functions within our program might take smaller portions of this input and produce some piece of our output, or some intermediate result that is needed to eventually produce this output.

In functional programming, we still need to organize our data in some way. So some of the ideas of objects/classes are still used to combine separate pieces of data in meaningful ways. However, we generally do not attach "actions" to data in the same way that classes do in OOP languages.

Since we don't perform actions directly on our data, functional languages are more likely to use immutable data as a default, rather than mutable data. (We should note though that both paradigms use both kinds of data in their own ways).

Functional Programming vs. OOP

The main point of separation between these paradigms is the question of "what is the fundamental building block of my program?" In object oriented programming, our programs are structured around objects. Functions are things we can do to an object or with an object.

In functional programming, functions are always first class citizens - the main building block of our code. In object oriented programming, functions can be first class citizens, but they do not need to be. Even in languages where they can be, they often are not used in this way, since this isn't as natural within the object oriented paradigm.

Object Oriented Programming Languages

Many of the most popular programming languages are OOP languages. Java, for a long time the most widely used language, is perhaps the most archetypal OO language. All code must exist within an object, even in a simple "Hello World" program:

class MyProgram {
  public static void main(String[] args) {
    System.out.println("Hello World!");
  }
}

In this example, we could not write our 'main' function on its own, without the use of 'class MyProgram'.

Java has a single basic 'Object' class, and all other classes (including any new classes you write) must inherit from it for basic behaviors like memory allocation. Java classes only allow single inheritance. This means that a class cannot inherit from multiple different types. Thus, all Java classes you would use can be mapped out on a tree structure with 'Object' as the root of the tree.

Other object oriented languages use the general ideas of classes, objects, and inheritance, but with some differences. C++ and Python both allow multiple inheritance, so that a class can inherit behavior from multiple existing classes. While these are both OOP languages, they are also more flexible in allowing functions to exist outside of classes. A basic script in either of these languages need not use any classes. In Python, we'd just write:

if __name__ == "__main__":
  print("Hello World!")

In C++, this looks like:

int main() {
  std::cout << "Hello World!" << std::endl;
}

These languages also don't have such a strictly defined inheritance structure. You can create classes that do not inherit from anything else, and they'll still work.

FP Languages

Haskell is perhaps the language that is most identifiable with the functional paradigm. Its type system and compiler really force you to adopt functional ideas, especially around immutable data, pure functions, and tail call optimization. It also embraces lazy evaluation, which is aligned with FP principles, but not a requirement for a functional language.

There are several other programming languages that generally get associated with the functional paradigm include Clojure, OCaml, Lisp, Scala and Rust. These languages aren't all functional in the same way as Haskell; there are many notable differences. Lisp bills itself specifically as a multi-paradigm language, and Scala is built to cross-compile with Java! Meanwhile Rust's syntax looks more object oriented, but its inheritance system (traits) feel much more like Haskell. However, on balance, these languages express functional programming ideas much more than their counterparts.

Amongst the languages mentioned in the object oriented section, Python has the most FP features. It is more natural to write functions outside of your class objects, and concepts like higher order functions and lambda expressions are more idiomatic than in C++ or Java. This is part of the reason Python is often recommended for beginners, with another reason being that its syntax makes it a relatively simple language to learn.

Advantages of Functional Programming

Fewer Bugs

FP code has a deserved reputation for having fewer bugs. Anecdotally, I certainly find I have a much easier time writing bug free code in Haskell than Python. Many bugs in object oriented code are caused by the proliferation of mutable state. You might pass an object to a method and expect your object to come back unchanged...only to find that the method does in fact change your object's state. With objects, it's also very easy for unstated pre-conditions to pop up in class methods. If your object is not in the state you expect when the method is called, you'll end up with behavior you didn't intend.

A lot of function-based code makes these errors impossible by imposing immutable objects as the default, if not making it a near requirement, as Haskell does. When the function is the building block of your code, you must specify precisely what the inputs of the function are. This gives you more opportunities to determine pre-conditions for this data. It also ensures that the return results of the function are the primary way you affect the rest of your program.

Functions also tend to be easier to test than objects. It is often tricky to create objects with the precise state you want to assess in a unit test, whereas to test a function you only need to reproduce the inputs.

More Expressive, Reasonable Design

The more you work with functions as your building blocks, and the more you try to fill your code with pure functions, the easier it will be to reason about your code. Imagine you have a couple dozen fields on an object in OO code. If someone calls a function on that object, any of those fields could impact the result of the method call.

Functions give you the opportunity to narrow things down to the precise values that you actually need to perform the computation. They let you separate the essential information from superfluous information, making it more obvious what the responsibilities are for each part of your code.

Multithreading

You can do parallel programming no matter what programming language you're using, but the functional programming paradigm aligns very well with parallel processing. To kick off a new thread in any language, you pretty much always have to pass a function as an argument, and this is more natural in FP. And with pure functions that don't modify shared mutable objects, FP is generally much easier to break into parallelizable pieces that don't require complex locking schemes.

Disadvantages of Functional Programming

Intuition of Complete Objects

Functional programming can feel less intuitive than object oriented programming. Perhaps one reason for this is that object oriented programming allows us to reason about "complete" objects, whose state at any given time is properly defined.

Functions are, in a sense, incomplete. A function is not a what that you can hold as a picture in your head. A function is a how. Given some inputs, how do you produce the outputs? In other words, it's a procedure. And a procedure can only really be imagined as a concrete object once you've filled in its inputs. This is best exemplified by the fact that functions have no native 'Show' instance in Haskell.

>> show (+)
No instance for Show (Integer -> Integer -> Integer) arising from a use of 'show'

If you apply the '+' function to arguments (and so create what could be called an "object"), then we can print it. But until then, it doesn't make much sense. If objects are the building block of your code though, you could, hypothetically, print the state of the objects in your code every step of the way.

Mutable State can be Useful!

As much as mutable state can cause a lot of bugs, it is nonetheless a useful tool for many problems, and decidedly more intuitive for certain data structures. If we just imagine something like the "Snake" game, it has a 2D grid that remains mostly the same from tick to tick, with just a couple things updating. This is easier to capture with mutable data.

Web development is another area where mutable objects are extremely useful. Anytime the user enters information on the page, some object has to change! Web development in FP almost requires its own paradigm (see "Functional Reactive Programming"). Haskell can represent mutable data, but the syntax is more cumbersome; you essentially need a separate data structure. Likewise, other functional languages might make mutability easier than Haskell, but mutability is still, again, more intuitive when objects are your fundamental building block, rather than functions on those objects.

We can see this even with something as simple as loops. Haskell doesn't perform "for-loops" in the same way as other languages, because most for loops essentially rely on the notion that there is some kind of state updating on each iteration of the loop, even if that state is only the integer counter. To write loops in Haskell, you have to learn concepts like maps and folds, which require you to get very used to writing new functions on the fly.

A Full Introduction to Haskell (and its Functional Aspects)

So functional programming languages are perhaps a bit more difficult to learn, but can offer a significant payoff if you put in the time to master the skills. Ultimately, you can use either paradigm for most kinds of projects and keep your development productive. It's down to your personal preference which you try while building software.

If you really want to dive into functional programming though, Haskell is a great language, since it will force you to learn FP principles more than other functional languages. For a complete introduction to Haskell, you should take a look at Haskell From Scratch, our beginner-level course for those new to the language. It will teach you everything you need to know about syntax and fundamental concepts, while providing you with a ton of hands-on practice through exercises and projects.

Haskell From Scratch also includes Making Sense of Monads, our course that shows the more functional side of Haskell by teaching you about the critical concept of monads. With these two courses under your belt, you'll be well on your way to mastery of functional programming! Head over here to learn more about these courses!

by James Bowen at January 15, 2024 04:00 PM

Derek Elkins

The Pullback Lemma in Gory Detail (Redux)

Introduction

Andrej Bauer has a paper titled The pullback lemma in gory detail that goes over the proof of the pullback lemma in full detail. This is a basic result of category theory and most introductions leave it as an exercise. It is a good exercise, and you should prove it yourself before reading this article or Andrej Bauer’s.

Andrej Bauer’s proof is what most introductions are expecting you to produce. I very much like the representability perspective on category theory and like to see what proofs look like using this perspective.

So this is a proof of the pullback lemma from the perspective of representability.

Preliminaries

The key thing we need here is a characterization of pullbacks in terms of representability. To just jump to the end, we have for |f : A \to C| and |g : B \to C|, |A \times_{f,g} B| is the pullback of |f| and |g| if and only if it represents the functor \[\{(h, k) \in \mathrm{Hom}({-}, A) \times \mathrm{Hom}({-}, B) \mid f \circ h = g \circ k \}\]

That is to say we have the natural isomorphism \[ \mathrm{Hom}({-}, A \times_{f,g} B) \cong \{(h, k) \in \mathrm{Hom}({-}, A) \times \mathrm{Hom}({-}, B) \mid f \circ h = g \circ k \} \]

We’ll write the left to right direction of the isomorphism as |\langle u,v\rangle : U \to A \times_{f,g} B| where |u : U \to A| and |v : U \to B| and they satisfy |f \circ u = g \circ v|. Applying the isomorphism right to left on the identity arrow gives us two arrows |p_1 : A \times_{f,g} B \to A| and |p_2 : A \times_{f,g} B \to B| satisfying |p_1 \circ \langle u, v\rangle = u| and |p_2 \circ \langle u,v \rangle = v|. (Exercise: Show that this follows from being a natural isomorphism.)

One nice thing about representability is that it reduces categorical reasoning to set-theoretic reasoning that you are probably already used to, as we’ll see. You can connect this definition to a typical universal property based definition used in Andrej Bauer’s article. Here we’re taking it as the definition of the pullback.

Proof

The claim to be proven is if the right square in the below diagram is a pullback square, then the left square is a pullback square if and only if the whole rectangle is a pullback square. \[ \xymatrix { A \ar[d]_{q_1} \ar[r]^{q_2} & B \ar[d]_{p_1} \ar[r]^{p_2} & C \ar[d]^{h} \\ X \ar[r]_{f} & Y \ar[r]_{g} & Z }\]

Rewriting the diagram as equations, we have:

Theorem: If |f \circ q_1 = p_1 \circ q_2|, |g \circ p_1 = h \circ p_2|, and |(B, p_1, p_2)| is a pullback of |g| and |h|, then |(A, q_1, q_2)| is a pullback of |f| and |p_1| if and only if |(A, q_1, p_2 \circ q_2)| is a pullback of |g \circ f| and |h|.

Proof: If |(A, q_1, q_2)| was a pullback of |f| and |p_1| then we’d have the following.

\[\begin{align} \mathrm{Hom}({-}, A) & \cong \{(u_1, u_2) \in \mathrm{Hom}({-}, X)\times\mathrm{Hom}({-}, B) \mid f \circ u_1 = p_1 \circ u_2 \} \\ & \cong \{(u_1, (v_1, v_2)) \in \mathrm{Hom}({-}, X)\times\mathrm{Hom}({-}, Y)\times\mathrm{Hom}({-}, C) \mid f \circ u_1 = p_1 \circ \langle v_1, v_2\rangle \land g \circ v_1 = h \circ v_2 \} \\ & = \{(u_1, (v_1, v_2)) \in \mathrm{Hom}({-}, X)\times\mathrm{Hom}({-}, Y)\times\mathrm{Hom}({-}, C) \mid f \circ u_1 = v_1 \land g \circ v_1 = h \circ v_2 \} \\ & = \{(u_1, v_2) \in \mathrm{Hom}({-}, X)\times\mathrm{Hom}({-}, C) \mid g \circ f \circ u_1 = h \circ v_2 \} \end{align}\]

The second isomorphism is |B| being a pullback and |u_2| is an arrow into |B| so it’s necessarily of the form |\langle v_1, v_2\rangle|. The first equality is just |p_1 \circ \langle v_1, v_2\rangle = v_1| mentioned earlier. The second equality merely eliminates the use of |v_1| using the equation |f \circ u_1 = v_1|.

This overall natural isomorphism, however, is exactly what it means for |A| to be a pullback of |g \circ f| and |h|. We verify the projections are what we expect by pushing |id_A| through the isomorphism. By assumption, |u_1| and |u_2| will be |q_1| and |q_2| respectively in the first isomorphism. We see that |v_2 = p_2 \circ \langle v_1, v_2\rangle = p_2 \circ q_2|.

We simply run the isomorphism backwards to get the other direction of the if and only if. |\square|

The simplicity and compactness of this proof demonstrates why I like representability.

January 15, 2024 01:33 AM

January 10, 2024

Chris Reade

Diagrams for Penrose Tiles

Penrose Kite and Dart Tilings with Haskell Diagrams

Revised version (no longer the full program in this literate Haskell)

Infinite non-periodic tessellations of Roger Penrose’s kite and dart tiles.

leftFilledSun6
leftFilledSun6

As part of a collaboration with Stephen Huggett, working on some mathematical properties of Penrose tilings, I recognised the need for quick renderings of tilings. I thought Haskell diagrams would be helpful here, and that turned out to be an excellent choice. Two dimensional vectors were well-suited to describing tiling operations and these are included as part of the diagrams package.

This literate Haskell uses the Haskell diagrams package to draw tilings with kites and darts. It also implements the main operations of compChoices and decompPatch which are used for constructing tilings (explained below).

Firstly, these 5 lines are needed in Haskell to use the diagrams package:

{-# LANGUAGE NoMonomorphismRestriction #-}
{-# LANGUAGE FlexibleContexts          #-}
{-# LANGUAGE TypeFamilies              #-}
import Diagrams.Prelude
import Diagrams.Backend.SVG.CmdLine

and we will also import a module for half tiles (explained later)

import HalfTile

These are the kite and dart tiles.

Kite and Dart
Kite and Dart

The red line marking here on the right hand copies, is purely to illustrate rules about how tiles can be put together for legal (non-periodic) tilings. Obviously edges can only be put together when they have the same length. If all the tiles are marked with red lines as illustrated on the right, the vertices where tiles meet must all have a red line or none must have a red line at that vertex. This prevents us from forming a simple rombus by placing a kite top at the base of a dart and thus enabling periodic tilings.

All edges are powers of the golden section \phi which we write as phi.

phi::Double
phi = (1.0 + sqrt 5.0) / 2.0

So if the shorter edges are unit length, then the longer edges have length phi. We also have the interesting property of the golden section that phi^2 = phi + 1 and so 1/phi = phi-1, phi^3 = 2phi +1 and 1/phi^2 = 2-phi.

All angles in the figures are multiples of tt which is 36 deg or 1/10 turn. We use ttangle to express such angles (e.g 180 degrees is ttangle 5).

ttangle:: Int -> Angle Double
ttangle n = (fromIntegral (n `mod` 10))*^tt
             where tt = 1/10 @@ turn

Pieces

In order to implement compChoices and decompPatch, we need to work with half tiles. We now define these in the separately imported module HalfTile with constructors for Left Dart, Right Dart, Left Kite, Right Kite

data HalfTile rep = LD rep -- defined in HalfTile module
                  | RD rep
                  | LK rep
                  | RK rep

where rep is a type variable allowing for different representations. However, here, we want to use a more specific type which we will call Piece:

type Piece = HalfTile (V2 Double)

where the half tiles have a simple 2D vector representation to provide orientation and scale. The vector represents the join edge of each half tile where halves come together. The origin for a dart is the tip, and the origin for a kite is the acute angle tip (marked in the figure with a red dot).

These are the only 4 pieces we use (oriented along the x axis)

ldart,rdart,lkite,rkite:: Piece
ldart = LD unitX
rdart = RD unitX
lkite = LK (phi*^unitX)
rkite = RK (phi*^unitX)
pieces
pieces

Perhaps confusingly, we regard left and right of a dart differently from left and right of a kite when viewed from the origin. The diagram shows the left dart before the right dart and the left kite before the right kite. Thus in a complete tile, going clockwise round the origin the right dart comes before the left dart, but the left kite comes before the right kite.

When it comes to drawing pieces, for the simplest case, we just want to show the two tile edges of each piece (and not the join edge). These edges are calculated as a list of 2 new vectors, using the join edge vector v. They are ordered clockwise from the origin of each piece

pieceEdges:: Piece -> [V2 Double]
pieceEdges (LD v) = [v',v ^-^ v'] where v' = phi*^rotate (ttangle 9) v
pieceEdges (RD v) = [v',v ^-^ v'] where v' = phi*^rotate (ttangle 1) v
pieceEdges (RK v) = [v',v ^-^ v'] where v' = rotate (ttangle 9) v
pieceEdges (LK v) = [v',v ^-^ v'] where v' = rotate (ttangle 1) v

Now drawing lines for the 2 outer edges of a piece is simply

drawPiece:: Piece -> Diagram B
drawPiece = strokeLine . fromOffsets . pieceEdges

and drawing all 3 edges round a piece is

drawRoundPiece:: Piece -> Diagram B
drawRoundPiece = strokeLoop . closeLine . fromOffsets . pieceEdges

To fill half tile pieces, we can use fillOnlyPiece which fills without showing edges of a half tile (by using line width none).

fillOnlyPiece:: Colour Double -> Piece -> Diagram B
fillOnlyPiece col piece = drawRoundPiece piece # fc col # lw none

We also use fillPieceDK which fills darts and kites with given colours and also draws edges using drawPiece.

fillPieceDK:: Colour Double -> Colour Double -> Piece -> Diagram B
fillPieceDK dcol kcol piece = drawPiece piece <> fillOnlyPiece col piece where
    col = case piece of (LD _) -> dcol
                        (RD _) -> dcol
                        (LK _) -> kcol
                        (RK _) -> kcol

For an alternative fill operation on whole tiles, it is useful to calculate a list of the 4 tile edges of a completed half-tile piece clockwise from the origin of the tile. (This will allow colour filling a whole tile)

wholeTileEdges:: Piece -> [V2 Double]
wholeTileEdges (LD v) = pieceEdges (RD v) ++ map negated (reverse (pieceEdges (LD v)))
wholeTileEdges (RD v) = wholeTileEdges (LD v)
wholeTileEdges (LK v) = pieceEdges (LK v) ++ map negated (reverse (pieceEdges (RK v)))
wholeTileEdges (RK v) = wholeTileEdges (LK v)

To fill whole tiles with colours, darts with dcol and kites with kcol we can now use leftFillPieceDK. This uses only the left pieces to identify the whole tile and ignores right pieces so that a tile is not filled twice.

leftFillPieceDK:: Colour Double -> Colour Double -> Piece -> Diagram B
leftFillPieceDK dcol kcol c = case c of 
  (LD _) -> (strokeLoop $ glueLine $ fromOffsets $ wholeTileEdges c)  # fc dcol
  (LK _) -> (strokeLoop $ glueLine $ fromOffsets $ wholeTileEdges c)  # fc kcol
  _      -> mempty

By making Pieces transformable we can reuse generic transform operations. These 4 lines of code are required to do this

type instance N (HalfTile a) = N a
type instance V (HalfTile a) = V a
instance Transformable a => Transformable (HalfTile a) where
    transform t ht = fmap (transform t) ht

So we can also scale and rotate a piece by an angle. (Positive rotations are in the anticlockwise direction.)

scale :: Double -> Piece -> Piece
rotate :: Angle Double -> Piece -> Piece

Patches

A patch is a list of located pieces (each with a 2D point)

type Patch = [Located Piece]

To turn a whole patch into a diagram using some function pd for drawing the pieces, we use

drawPatchWith:: (Piece -> Diagram B) -> Patch -> Diagram B 
drawPatchWith pd patch = position $ fmap (viewLoc . mapLoc pd) patch

Here mapLoc applies a function to the piece in a located piece – producing a located diagram in this case, and viewLoc returns the pair of point and diagram from a located diagram. Finally position forms a single diagram from the list of pairs of points and diagrams.

Update: We now use a class for drawable tilings, making Patch an instance

class Drawable a where
 drawWith :: (Piece -> Diagram B) -> a -> Diagram B
instance Drawable Patch where
 drawWith = drawPatchWith

We then introduce special cases:

draw :: Drawable a => a -> Diagram B
draw = drawWith drawPiece
fillDK:: Drawable a => Colour Double -> Colour Double -> a -> Diagram B
fillDK c1 c2 = drawWith (fillPieceDK c1 c2)

Patches are automatically inferred to be transformable now Pieces are transformable, so we can also scale a patch, translate a patch by a vector, and rotate a patch by an angle.

scale :: Double -> Patch -> Patch
rotate :: Angle Double -> Patch -> Patch
translate:: V2 Double -> Patch -> Patch

As an aid to creating patches with 5-fold rotational symmetry, we combine 5 copies of a basic patch (rotated by multiples of ttangle 2 successively).

penta:: Patch -> Patch
penta p = concatMap copy [0..4] 
            where copy n = rotate (ttangle (2*n)) p

This must be used with care to avoid nonsense patches. But two special cases are

sun,star::Patch         
sun =  penta [rkite `at` origin, lkite `at` origin]
star = penta [rdart `at` origin, ldart `at` origin]

This figure shows some example patches, drawn with draw The first is a star and the second is a sun.

tile patches
tile patches

The tools so far for creating patches may seem limited (and do not help with ensuring legal tilings), but there is an even bigger problem.

Correct Tilings

Unfortunately, correct tilings – that is, tilings which can be extended to infinity – are not as simple as just legal tilings. It is not enough to have a legal tiling, because an apparent (legal) choice of placing one tile can have non-local consequences, causing a conflict with a choice made far away in a patch of tiles, resulting in a patch which cannot be extended. This suggests that constructing correct patches is far from trivial.

The infinite number of possible infinite tilings do have some remarkable properties. Any finite patch from one of them, will occur in all the others (infinitely many times) and within a relatively small radius of any point in an infinite tiling. (For details of this see links at the end)

This is why we need a different approach to constructing larger patches. There are two significant processes used for creating patches, namely inflate (also called compose) and decompose.

To understand these processes, take a look at the following figure.

experiment
experiment

Here the small pieces have been drawn in an unusual way. The edges have been drawn with dashed lines, but long edges of kites have been emphasised with a solid line and the join edges of darts marked with a red line. From this you may be able to make out a patch of larger scale kites and darts. This is an inflated patch arising from the smaller scale patch. Conversely, the larger kites and darts decompose to the smaller scale ones.

Decomposition

Since the rule for decomposition is uniquely determined, we can express it as a simple function on patches.

decompPatch :: Patch -> Patch
decompPatch = concatMap decompPiece

where the function decompPiece acts on located pieces and produces a list of the smaller located pieces contained in the piece. For example, a larger right dart will produce both a smaller right dart and a smaller left kite. Decomposing a located piece also takes care of the location, scale and rotation of the new pieces.

decompPiece lp = case viewLoc lp of
  (p, RD vd)-> [ LK vd  `at` p
               , RD vd' `at` (p .+^ v')
               ] where v'  = phi*^rotate (ttangle 1) vd
                       vd' = (2-phi) *^ (negated v') -- (2-phi) = 1/phi^2
  (p, LD vd)-> [ RK vd `at` p
               , LD vd' `at` (p .+^ v')
               ]  where v'  = phi*^rotate (ttangle 9) vd
                        vd' = (2-phi) *^ (negated v')  -- (2-phi) = 1/phi^2
  (p, RK vk)-> [ RD vd' `at` p
               , LK vk' `at` (p .+^ v')
               , RK vk' `at` (p .+^ v')
               ] where v'  = rotate (ttangle 9) vk
                       vd' = (2-phi) *^ v' -- v'/phi^2
                       vk' = ((phi-1) *^ vk) ^-^ v' -- (phi-1) = 1/phi
  (p, LK vk)-> [ LD vd' `at` p
               , RK vk' `at` (p .+^ v')
               , LK vk' `at` (p .+^ v')
               ] where v'  = rotate (ttangle 1) vk
                       vd' = (2-phi) *^ v' -- v'/phi^2
                       vk' = ((phi-1) *^ vk) ^-^ v' -- (phi-1) = 1/phi

This is illustrated in the following figure for the cases of a right dart and a right kite.

explanation
explanation

The symmetric diagrams for left pieces are easy to work out from these, so they are not illustrated.

With the decompPatch operation we can start with a simple correct patch, and decompose repeatedly to get more and more detailed patches. (Each decomposition scales the tiles down by a factor of 1/phi but we can rescale at any time.)

This figure illustrates how each piece decomposes with 4 decomposition steps below each one.

four decompositions of pieces
four decompositions of pieces
thePieces =  [ldart, rdart, lkite, rkite]  
fourDecomps = hsep 1 $ fmap decomps