Planet Haskell

May 16, 2012

"A multi-user browser-based GHCi"

A multi-user browser-based interactive ghci

Dear Haskellers,

this is the blog for the Google Summer of Code 2012 project “A multi-user browser-based interactive ghci”.

Student: Shae Erisson (shapr)
Mentor: Heinrich Apfelmus

The primary goal of this project is to create an interactive Haskell interpreter that lives in the web browser and allows users to share code and have simulatenous interpreter sessions, thereby facilitating the teaching and learning of Haskell. The idea is that in a not too far future, a typical conversation on the Haskell IRC channel may look like this:

Newbie: “Hey, I got a problem with my code here.”
Expert: “Sure, fire up the magic interpreter and we’ll check it out together.”
Newbie *shares link*
Expert: “So, where is the problem?”
Newbie *evaluates expression*
Newbie: “See, it has a type error here.”
Expert: “Hm. Ah, I see. You have to use a prehistomorphic zygomorphism.”
Expert *changes code and reloads*
Newbie: “Oh, it works! But how did you do that?”
Expert: “Well, it’s probably easier to understand if I write it like this.”
Expert *simplifies code*
etc.

The secondary goal of this project is to explore some of the possibilities of HTML output for a Haskell interpreter.


by shapr at May 16, 2012 02:53 PM

Edward Z. Yang

What happens when you mix three research programming languages together

“...so that’s what we’re going to build!”

“Cool! What language are you going to write it in?”

“Well, we were thinking we were going to need three programming languages...”

“...three?”

“...and they’ll be research programming languages too...”

“Are you out of your mind?”


This was the conversation in streaming through my head when I decided that I would be writing my latest software project in Coq, Haskell and Ur/Web. I had reasonably good reasons for the choice: I wanted Coq because I didn’t actually want to implement a theorem prover from scratch, I wanted Ur/Web because I didn’t actually want to hand write JavaScript to get an AJAX interface, and I wanted Haskell because I didn’t want to write a bucket of C to get Ur/Web and Coq to talk to each other. But taken altogether the whole thing seemed a bit ludicrous, like an unholy fusion of a trinity of research programming languages.

In the end, it worked out quite well. Now, what this means depends on your expectations: it was not the case that “everything worked out of the box and had very nice instructions attached.” However, if it was the case that:

  • No single issue ended up requiring an unbounded amount of time and yak shaving,
  • Any patches written made it into upstream, improving the situation of the software for future developers, and
  • The time spent on engineering grease is less than the time it would have taken to build the system with inferior languages,
  • Everyone involved in the project is willing to learn all of the languages involved (easy if it’s only one person),

then yes, it worked “quite well”. In this post, I’d like to describe in a little more detail what happened when I put these three languages together and speculate wildly about general maxims that might apply when someone is doing something similar. (A description of what the project actually is will unfortunately have to wait a little; it’s not quite done yet.)

Coq

While Coq is a research language, it is also in very wide use among academics, and most of its instability lies in advanced features that I did not use in my project. So the primary issues I encountered with Coq were not bugs, but in integrating it with the system (namely, making it talk to Haskell).

Maxim 1. Interchange formats will be undocumented and just good enough to get the job done.

Coq is already designed to allow for communication between processes (this is how the Proof General/Emacs and Coq talk to each other), but the format between coqtop and Proof General was undocumented, ad hoc, and didn't transmit enough information for my application. In the face of such a situation, there are two ways to proceed: grit your teeth and implement the bad protocol or patch the compiler to make a better one. I chose the latter, and learned something very interesting:

Maxim 2. In ML-like languages, it’s very easy to make simple but far reaching changes to a codebase, due to the assistance of the typechecker.

Making the changes to the frontend was very simple; there was nothing deep about the change, and a combination of the typechecker and grep allowed me to pull off the patch with zero debugging. With a few XML tags at a few key spots, I got output reasonable enough to build the rest of the system with.

Aside. Later, I learned that coqide in recent versions of Coq (8.4 and later) has another interchange format. Moving forward, it is probably the correct mechanism to interact with Coq interactively, though this is made somewhat more difficult by the fact that the interchange format is undocumented; however, I've filed a bug. With any luck, it will hopefully do better than my patch. My patch was originally intended to be a partial implementation of PGIP, a generic interchange format for interacting with theorem provers, but the Coq developers and I later discovered that the PGIP project is inactive, and the other user, Isabelle, has discontinued using their PGIP backend. (Sometimes standards don’t help!)

Ur/Web

Ur/Web is comparatively less used, and accordingly we ran into a variety of bugs and other infelicities spanning all parts of the system, from the frontend to the compiler. Were they blockers? No!

Maxim 3. A deterministically reproducible bug in some core functionality will get fixed very quickly by an active original author of the code.

This maxim doesn’t apply to fundamental limitations in design (where the fix will take a lot of elbow grease, though the author will usually have a good idea when that’s the case), but other bugs of this type, I found I could get freakishly quick turnaround times for fixes. While I may attribute part of this to the fact that my advisor was the one who wrote the compiler, I don’t think that’s all there is to it. There is a certain pride that comes with an interesting, tricky bit of code you wrote, that makes it an irresistible little puzzle when someone shows you a bug. And we love little puzzles.

There’s also a corollary:

Maxim 4. The less interesting a problem is to the academic, the more likely it is you’ll be able to fix it yourself.

Academics are somewhat allergic to problems that they’re not interested in and which aren’t vital for their research. This means they don’t like working on these bits, but it also means that they’ve probably kept it simple, which means you’re more likely to be able to figure it out. (A good typechecker also really helps! See maxim 2.) There was a simple bug with serving 404s from FastCGI's compiled by Ur/Web, which had a very simple fix; I also made some modifications to Ur/Web made it runnable without having to make install first. Maintainers of active research software tend to be quite receptive to these "engineering" patches, which serve no direct research purpose. I consider these contributes to be a vital component of being a good citizen of the open source community.

Haskell

OK, Haskell is not really “just” a research language anymore; it is also a very flexible general purpose language which has seen quite a bit of real world use and can be treated as an “ordinary” language in that respect. This made it a good choice for gluing the two other languages together; it can do just about anything, and has very good FFI support for calling into and out of Haskell. This brings us to our next maxim:

Maxim 5. An FFI is a crucial feature for any DSL, and should be a top priority among tasks involved in preparing a language for general usage.

Having Haskell and Ur/Web talk to each other through their FFIs was key for making this all work. Ur/Web is a domain specific language for writing web applications, and among other things it does not include robust systems libraries (e.g. executing external processes and interfacing with them). Most languages will have this problem, since library support takes a bit of work to add, but Ur/Web has a second problem: all side-effectful transactions need to also be able to be rolled back, and this is rather hard to achieve for general input-output. However, with an FFI, we can implement any code which needs this library support in a more suitable language (Haskell), wrap it up in an API which gives the appropriate transactional guarantees, and let Ur/Web use it. Without it, we would not have been able to use Ur/Web: it’s an extremely flexible escape hatch.

Specifying an FFI also is a good way of demonstrating how your language is different from C: it forces you to think about what invariants you expect foreign functions to have (referential transparency? thread-safety?): these invariants are exactly the ones that get automatically fulfilled by code written in your language. That’s pretty cool!

However, because functions which manipulate C pointers are non-transactional, Ur/Web is limited to FFI functions which handle basic C types, e.g. integers and strings. Thus the question of parsing becomes one of utmost importance for Ur/Web, as strings are the preferred interchange format for complex structures. While different languages will have different situations, in general:

Maxim 6. Make sure you know how to do parsing in all of the languages involved.

Conclusion

I’ve presented six maxims of research polyglottism:

  1. Interchange formats will be undocumented and just good enough to get the job done.
  2. In ML-like languages, it’s very easy to make simple but far reaching changes to a codebase, due to the assistance of the typechecker.
  3. A deterministically reproducible bug in some core functionality will get fixed very quickly by an active original author of the code.
  4. The less interesting a problem is to the academic, the more likely it is you’ll be able to fix it yourself.
  5. An FFI is a crucial feature for any DSL, and should be a top priority among tasks involved in preparing a language for general usage.
  6. Make sure you know how to do parsing in all of the languages involved.

If you keep all of these maxims in mind, I believe that the tradeoff between some extra bugfixing and yak shaving for the benefits of the research programming language is a compelling one, and one that should be considered seriously. Yes, you have to be willing to muck around with the innards of all the tools you use, but for any sufficiently important tool, this is inevitably true. And what is a more important tool than your compiler?

by Edward Z. Yang at May 16, 2012 06:54 AM

May 15, 2012

Paul Johnson

Online lectures, early movies, and the Open University

There is a lot of excitement floating around the blogosphere right now about various experiments in remote learning. Some of the big name US universities are putting substantial amounts of material on line: lectures, course material, basically everything except the diploma. This post describes how an 11 year old boy took a Stanford University course in game theory (written by his proud Dad, but still...)

One thing the boy said to his father was "I think the concepts are interesting but the presentation is dull. Couldn’t they have done animations and things to make it better?". The presentation in question was Powerpoint slides with an occasional lecturer's head.

This reminds me of the early days of movie making. When the movie camera was first invented it seemed obvious how to film a drama: put some actors on a stage to perform a play, put a camera about where the best seat would be, and film the action. Of course this combined the worst features of film (monochrome, no sound, low resolution) with the worst features of theatre (action a long way away, small stage, static scenery). It didn't take movie makers long to realise that taking the actors out of the theatre and putting the camera in amongst them gave better results. The same goes for video lectures.

The sad thing is, none of this is new. When I was young there was (and still is) a UK institution called the Open University. It was created back in 1969 by the Labour government as part of its anti-elitist education-for-all vision. From the beginning it was designed as a mass-market concept: very few students would be at its physical campus; almost all its work would be done by distance learning, with lectures delivered by television, audio cassette (link for too young to know what they were) and any other form of high technology communications medium. So for many years you could get up at 6:00 am and watch a couple of hours of half hour lectures on anything from Mathematical Modelling to Sociology 101.  In my teenage years I finally got a TV in my bedroom and set up a timeswitch to turn it on at 6:00am. I still remember the theme tune to Sociology 101: "We socialise and we vandalise, We lock the sick awaaayy, Politicians promises, keep changing every daaayy...".

The presentation was similar to a normal documentary, albeit with a drastically reduced budget; a combination of talking heads, pictures of the thing being described (Sociology 101 showed lots of deprived multi-storey housing estates of the kind the Americans call "projects"), and various visual aids. Here are some examples from 1989 (Youtube).

One I remember in particular; it was explaining trigonometry. First it showed a circle with a radius line revolving around it, and the right angle triangle that resulted.   Then it turned the circle end on, so all you could see was a vertical line with the blob at the end of the radius moving up and down. Then it moved the line to the right, and the blob traced out a sine wave. And the lightbulb went on in my head. I knew about sine and cosine for right-angled triangles, and I'd seen that shape called a "sine wave", but until that point I hadn't understood the connection. Now I understood perfectly.

MIT, Stanford and the rest need to get back to the future (although they can leave out the flares and courderoys).

by Paul Johnson (noreply@blogger.com) at May 15, 2012 08:30 PM

apfelmus

FRP - Release of reactive-banana version 0.6

I have released version 0.6.0.0 of my reactive-banana library on hackage.

Compared to the previous release, the public API is essentially unchanged, except that I have renamed the module Reactive.Banana.Experimental.Model to Reactive.Banana.Model again, hence the major change in the version number.

Moreover, this release now works with the latest versions (0.13 and 0.90) of the wxHaskell library. Make sure that you install the latest patch release of wx, wxcore and wc if applicable, as they resolve a few problems with wxWidgets 2.9.3.

The major change introduced in this release is that the library can now be compiled with the Utrecht Haskell Compiler, in particular the JavaScript backend. It was Mathijs Kwik who suggested this feature to me. In other words, you can now use functional reactive programming with Haskell in the web browser! Well, if you manage to get UHC up and running, that is. The infrastructure still needs some work compared to GHC. But hey, if you want to program browser-based user interfaces with Haskell, my bananas are loaded and ready.

The main problem was that UHC doesn’t support all the language extensions that I needed for my efficient push-driven implementation. To solve this, I have introduced a new flag UseExtensions in the .cabal file. It is on by default, but will (hopefully) be disabled automatically if the compiler doesn’t support extensions like GADTs or TypeFamilies. (You can also disable it by hand.) The implementation will then fall back to the less efficient model implementation.

Many thanks to all users for suggesting and discussing changes with me, especially Mathijs Kwik and Henning Thielemann. Special thanks to Jeremy O’Donoghue for his work on wxHaskell.

Howdy, cowboy, it’s advertisment time! If you like the reactive-banana project, you can support me with a small donation: Flattr this!. Furthermore, at the end of every month, all ardent supporters will receive ¢2 worth of status information and fruity puns! (This magnificent offer can be respectfully declined in your Flattr preferences.)

PS: What happened to dynamic event switching? Well, I can tell you that the next version of reactive-banana, 0.7.0.0, will definitely include dynamic event switching and it will likely work with UHC as well!


Flattr this

May 15, 2012 03:33 PM

Luke Palmer

Life as a Musician?

So, it turns out I’m not dead. How about that?

I have dropped out of school, and am busking for a living. It is tiring (especially when I forget to drink enough water), sometimes discouraging (when I play things to no response whatsoever or make $5 in an hour), but mostly great. My job is making music! And more importantly, my job is making my music, or music I am in love with — although certain pieces tend to attract more tippers than others, so it’s not truly free (what is?).

My grandmother contacted me telling me about a startup mixer so I could find a job. I don’t think she really understands my decision. I can understand that — she wants me to get a stable, well-paying job, have kids and a family, and go to church. The usual narrative. The other day I was idly contemplating being a father. Not now, of course. But I can see the draw; I can see that being a pretty special thing. The question is whether it is worth it to me. Sacrifice is part of love. But do I sacrifice for my child, or do I sacrifice a child (umm! — sacrifice having a child) for my other loves? That is not a question I am remotely prepared to answer.

I used to think — perhaps I still do — that big questions like those aren’t really worth answering, at least not rationally. I suppose this “used to” is fairly recent, as I had spent a long time on them prior, and they led me nowhere but in circles of unfulfilled dustkicking. My self-image can be so limited at times, and the rational mind is a slave to its images. What I can really do, what I’m really made of, I perhaps thought, won’t be small enough to be so easily decided — it must be eased into, made part of myself through exploration and long, gradual growth. But the liberation I feel from this new occupation of mine has shown me that perhaps at points along this process such a life decision is valuable, that it can be a beacon that reminds me that I chose this because it was important to me — more important than anything else at one time — and so gives me something to hold onto in times of uncertainty or suffering. It sounds very compelling, doesn’t it? But I am still in the honeymoon phase of my relationship with my life as a musician, so the only thing I can be sure of is that my thoughts about it are distorted.

And am I really good enough to make this a living? Maybe Boulder is the only place people appreciate public performances of amateur classical music. Maybe when I migrate for the winter I will be met with indifference or contempt, and I will be stuck in a new city with no job. Maybe when I improvise or play my originals people only tip me because I have brought the piano out, not because the music speaks to them in any deep way — I know that is not true, my second piano sonata is almost always met by applause, but it has been 10 years since I wrote that; do I still have it? A teenager passes by and plays most of the pieces I do — not as well, but not badly — and he will surpass me by my age. Will I ever have the guts to sing out there?

A thousand fears and doubts dance their rite around my dream — all I can do is to go out there every day and hope it goes well. I think it’s proof that I’m alive. I pose this question to myself: would I rather be wildly successful in a software company, or wildly successful as a musician? The latter, by any metric. “Wildly” need not even appear. Standing on a plank and singing to the jury, my heart beating a thousand times a minute, with the conviction of a soldier — this outshines any vision of a successful software idea.

I’m not leaving software. But my most exciting software ideas aren’t the kinds of things one can easily make a living on. I’m working on a browser-based programming environment which explores a new way of designing and organizing code. I don’t want to say too much about it because as I code the idea continues to develop in my mind, and I don’t want to nail it down yet (maybe ever). But anyway, to make money with that would sacrifice its beauty; this tool is not for productivity, at least not at first: it is exploring a way of thinking. It is easier to make a living making the music I love than the software I love. If my life is to overflow with love and happiness, music is the breadwinner.

Again — only a month in. But I think this is the way to do it, for me. I’m not setting myself up for a comfortable life, but comfort is a trap anyway. It is the contrast that feels so good, and without that contrast comfort is just normal. Without discomfort to prepare the contrast, comfort is dull and boring. Anyway, that’s how I see it. Funny coming from a hedonist like me. I guess I’m having a stint of long-term hedonism at the expense of short-term.

Maybe someday I won’t even feel the need to justify my choice anymore. That’s when I’ll really be in it.


by Luke at May 15, 2012 12:44 PM

Ken T Takusagawa

[rwcxbwbv] Pierpont primes

Pierpont primes are of the form 2^m * 3^n + 1.  Because of their special form, they can be easily tested for primality using the Lucas primality test (not Lucas-Lehmer).  Trial division can also be greatly accelerated by precomputed tables of remainders.

Here is some Haskell code investigating Pierpont primes, including the classic lazy list exercise of generating in order all numbers of Pierpont form. Also there are some disorganized log files enumerating the largest Pierpont prime less than each power of two up to 2^9787, and a few larger.

The original motivation was, can all the known Pierpont primes be compiled into a table, much like the Mersenne and Fermat primes?  Unlike the other two, there are probably millions to billions of Pierpont primes with the reach of current limits of computing.  But disk space is cheap.  In the end, I concluded that computing is the prohibitive limiting factor due to a much larger search area than Mersenne prime searching.

by Ken (noreply@blogger.com) at May 15, 2012 05:32 AM

May 14, 2012

Brandon Simmons

A Simulation of a Biological MIS Algorithm

I've finished a simulation-visualization of the very cool algorithm for generating a Maximal Independant Set from "A Biological Solution to a Fundamental Distributed Computing Problem" by Afek et al. (sorry I don't have a link to the PDF, and respect you too much to link you to a paywall).

You can play with it or check out the code on GitHub; in particular, see "algorithm.js" for the didactic implementation of the algorithm.

It uses jQuery and Raphael.js for the visuals and events.

Goals for the project

  1. create a fun visual simulation that can give a nice intuition for the algorithm

  2. remain true to the algorithm as described in the paper, and to avoid anything novel or clever at this point

  3. attempt to organize my code in such a way as to be a readable explanation of the algorithm as described in the paper, hiding or isolating implementation details

  4. attempt to model the behavior of a real-world network of nodes using the OO abstraction, and (synchronous) events.

Trying to meet these demands was revealing. For one thing I was forced to break up the two message exchange steps into four discrete steps which must be performed synchronously across the network for things to work without race conditions.

Later

The pseudocode description is hiding some of the complexity and brittleness of the algorithm IMHO. I'd like to take it apart and try to rip out as many of the synchronous-network dependencies as I can, or explore how an identical but asynchronous system behaves: give all the nodes jittery clocks and without a coordinated start.

May 14, 2012 11:25 PM

Darcs

darcs weekly news #96

News and discussions

  1. Darcs 2.8.1 was released, the only change is a build dependency that will make it buildable with the next Haskell Platform:
  2. Meanwhile in HEAD, a new test strategy has been implemented:
  3. On the developers' mailing list, we are discussing how to make the darcs development process more friendly:

Issues resolved in the last week (3)

issue1921 Owen Stephens
issue2095 Ganesh Sittampalam
issue2161 Owen Stephens

Patches applied in the last week (77)

See darcs wiki entry for details.

by guillaume (noreply@blogger.com) at May 14, 2012 06:52 PM

Manuel M T Chakravarty

GPU-accelerated array computations in Haskell

I just published stable release 0.12 of the embedded array language Accelerate for high-performance GPU computing on Hackage. The release includes example applications, such as a real-time Canny edge detector, a fluid flow simulator, and a quasicrystal animation, as well as example algorithms, such as radixsort, matrix computations, and a Black-Scholes option pricing model.

In the new modularised architecture, the Accelerate release comprises four packages:

  • accelerate (the main package providing the Accelerate language),
  • accelerate-io (conversion operations with other Haskell array libraries, including the data-parallel companion library Repa),
  • accelerate-cuda (backend targetting NVIDIA GPUs via the CUDA SDK), and
  • accelerate-examples (example applications and regression tests).

Posted via email from Just Testing | Comment »

May 14, 2012 06:48 AM

May 13, 2012

Edward Z. Yang

Some thoughts about literature review

While working on my senior thesis, I had to write a prior work section, which ended up being a minisurvey for the particular subfield my topic was in. In the process, a little bird told me some things...

  • If you can, ask someone who might know a little bit about the subject to give you the rundown: there's a lot of knowledge in people's heads which never got written down. But also be aware that they will probably have their blind spots.
  • It is better to do the literature review later rather than earlier, after you have started digging into the topic. I have been told if you read the literature too early, you will get spoiled and stop thinking novel thoughts. But I also think there is also a little bit of "you'll understand the literature better" if you've already thought about the topic on your own. Plus, it's easy to think that everything has been done before: it's simply not true! (But if you think this, you will get needlessly discouraged.)
  • Don't indiscriminately add papers to your database. You should have something you want to do with it: is it an important paper that you have to cite because everyone knows about it? Is it directly addressing the issue you're dealing with? Does it happen to be particularly well written? Is it something that you could see yourself reading more carefully later? Don't be afraid to toss the paper out; if it actually was important, you'll run into it again later.
  • Every researcher is a historian. When you look at a paper, you're not just looking at what is written inside it, but its social context. There's a reason why "prior work" is so important to academics. If you don't understand a paper's context, it's unlikely you'll understand the paper.
  • Researchers don't necessarily talk to each other. Pay attention to who they cite; it says a little bit about what community they're in.
  • Researchers are happy to send you copies of papers they have written (so fear not the paywall that your university hasn't subscribed to). They may even volunteer extra information which may come in handy.
  • Be methodical. You're doing a search, and this means carefully noting down which papers you skimmed, and what you got out of them, and keeping track of other veins of research that you need to follow up on. It's like chasing a rabbit down a hole, but if you have some clearly defined search criteria, eventually you'll bottom out. You can prune the uninteresting papers later; the point here is to avoid duplicating work.
  • Read papers critically. Not everything that is published is good; that's the point of research!

What are your favorite maxims to keep in mind while you're surveying the literature?

by Edward Z. Yang at May 13, 2012 11:03 PM

Keegan McAllister

Automatic binary hardening with Autoconf

How do you protect a C program against memory corruption exploits? We should try to write code with no bugs, but we also need protection against any bugs which may lurk. Put another way, I try not to crash my bike but I still wear a helmet.

Operating systems now support a variety of tricks to make life difficult for would-be attackers. But most of these hardening features need to be enabled at compile time. When I started contributing to Mosh, I made it a goal to build with full hardening on every platform, not just proactive distributions like Ubuntu. This means detecting available hardening features at compile time.

Mosh uses Autotools, so this code is naturally part of the Autoconf script. I know that Autotools has a bad reputation in some circles, and I'm not going to defend it here. But a huge number of existing projects use Autotools. They can benefit today from a drop-in hardening recipe.

I've published an example project which uses Autotools to detect and enable some binary hardening features. To the extent possible under law, I waive all copyright and related or neighboring rights to the code I wrote for this project. (There are some third-party files in the m4/ subdirectory; those are governed by the respective licenses which appear in each file.) I want this code to be widely useful, and I welcome any refinements you have.

This article explains how my auto-detection code works, with some detail about the hardening measures themselves. If you just want to add hardening to your project, you don't necessarily need to read the whole thing. At the end I talk a bit about the performance implications.

How it works

The basic idea is simple. We use AX_CHECK_{COMPILE,LINK}_FLAG from the Autoconf Archive to detect support for each feature. The syntax is

AX_CHECK_COMPILE_FLAG(flag, action-if-supported, action-if-unsupported, extra-flags)

For extra-flags we generally pass -Werror so the compiler will fail on unrecognized flags. Since the project contains both C and C++ code, we check each flag once for the C compiler and once for the C++ compiler. Also, some flags depend on others, or have multiple alternative forms. This is reflected in the nesting structure of the action-if-supported and action-if-unsupported blocks. You can see the full story in configure.ac.

We accumulate all the supported flags into HARDEN_{C,LD}FLAGS and substitute these into each Makefile.am. The hardening flags take effect even if the user overrides CFLAGS on the command line. To explicitly disable hardening, pass

./configure --disable-hardening

A useful command when testing is

grep HARDEN config.log

Complications

Clang will not error out on unrecognized flags, even with -Werror. Instead it prints a message like

clang: warning: argument unused during compilation: '-foo'

and continues on blithely. I don't want these warnings to appear during the actual build, so I hacked around Clang's behavior. The script wrap-compiler-for-flag-check runs a command and errors out if the command prints a line containing "warning: argument unused". Then configure temporarily sets

CC="$srcdir/scripts/wrap-compiler-for-flag-check $CC"

while performing the flag checks.

When I integrated hardening into Mosh, I discovered that Ubuntu's default hardening flags conflict with ours. For example we set -Wstack-protector, meaning "warn about any unprotected functions", and they set --param=ssp-buffer-size=4, meaning "don't protect functions with fewer than 4 bytes of buffers". Our stack-protector flags are strictly more aggressive, so I disabled Ubuntu's by adding these lines to debian/rules:

export DEB_BUILD_MAINT_OPTIONS = hardening=-stackprotector
-include /usr/share/dpkg/buildflags.mk

We did something similar for Fedora.

Yet another problem is that Debian distributes skalibs (a Mosh dependency) as a static-only library, built without -fPIC, which in turn prevents Mosh from using -fPIC. Mosh can build the relevant parts of skalibs internally, but Debian and Ubuntu don't want us doing that. The unfortunate solution is simply to reimplement the small amount of skalibs we were using on Linux.

The flags

Here are the specific protections I enabled.

  • -D_FORTIFY_SOURCE=2 enables some compile-time and run-time checks on memory and string manipulation. This requires -O1 or higher. See also man 7 feature_test_macros.

  • -fno-strict-overflow prevents GCC from optimizing away arithmetic overflow tests.

  • -fstack-protector-all detects stack buffer overflows after they occur, using a stack canary. We also set -Wstack-protector (warn about unprotected functions) and --param ssp-buffer-size=1 (protect regardless of buffer size). (Actually, the "-all" part of -fstack-protector-all might imply ssp-buffer-size=1.)

  • Attackers can use fragments of legitimate code already in memory to stitch together exploits. This is much harder if they don't know where any of that code is located. Shared libraries get random addresses by default, but your program doesn't. Even an exploit against a shared library can take advantage of that. So we build a position independent executable (PIE), with the goal that every executable page has a randomized address.

  • Exploits can't overwrite read-only memory. Some areas could be marked as read-only except that the dynamic loader needs to perform relocations there. The GNU linker flag -z relro arranges to set them as read-only once the dynamic loader is done with them.

    In particular, this can protect the PLT and GOT, which are classic targets for memory corruption. But PLT entries normally get resolved on demand, which means they're writable as the program runs. We set -z now to resolve PLT entries at startup so they get RELRO protection.

In the example project I also enabled -Wall -Wextra -Werror. These aren't hardening flags and we don't need to detect support, but they're quite important for catching security problems. If you can't make your project -Wall-clean, you can at least add security-relevant checks such as -Wformat-security.

Demonstration

On x86 Linux, we can check the hardening features using Tobias Klein's checksec.sh. First, as a control, let's build with no hardening.

$ ./build.sh --disable-hardening
+ autoreconf -fi
+ ./configure --disable-hardening
...
+ make
...
$ ~/checksec.sh --file src/test
No RELRO No canary found NX enabled No PIE

The no-execute bit (NX) is mainly a kernel and CPU feature. It does not require much compiler support, and is enabled by default these days. Now we'll try full hardening:

$ ./build.sh
+ autoreconf -fi
+ ./configure
...
checking whether C compiler accepts -fno-strict-overflow... yes
checking whether C++ compiler accepts -fno-strict-overflow... yes
checking whether C compiler accepts -D_FORTIFY_SOURCE=2... yes
checking whether C++ compiler accepts -D_FORTIFY_SOURCE=2... yes
checking whether C compiler accepts -fstack-protector-all... yes
checking whether C++ compiler accepts -fstack-protector-all... yes
checking whether the linker accepts -fstack-protector-all... yes
checking whether C compiler accepts -Wstack-protector... yes
checking whether C++ compiler accepts -Wstack-protector... yes
checking whether C compiler accepts --param ssp-buffer-size=1... yes
checking whether C++ compiler accepts --param ssp-buffer-size=1... yes
checking whether C compiler accepts -fPIE... yes
checking whether C++ compiler accepts -fPIE... yes
checking whether the linker accepts -fPIE -pie... yes
checking whether the linker accepts -Wl,-z,relro... yes
checking whether the linker accepts -Wl,-z,now... yes
...
+ make
...
$ ~/checksec.sh --file src/test
Full RELRO Canary found NX enabled PIE enabled

We can dig deeper on some of these. objdump -d shows that the unhardened executable puts main at a fixed address, say 0x4006e0, while the position-independent executable specifies a small offset like 0x9e0. We can also see the stack-canary checks:

b80:  sub   $0x18,%rsp
b84: mov %fs:0x28,%rax
b8d: mov %rax,0x8(%rsp)

... function body ...

b94: mov 0x8(%rsp),%rax
b99: xor %fs:0x28,%rax
ba2: jne bb4 <c_fun+0x34>

... normal epilogue ...

bb4: callq 9c0 <__stack_chk_fail@plt>

The function starts by copying a "canary" value from %fs:0x28 to the stack. On return, that value had better still be there; otherwise, an attacker has clobbered our stack frame.

The canary is chosen randomly by glibc at program start. The %fs segment has a random offset in linear memory, which makes it hard for an attacker to discover the canary through an information leak. This also puts it within thread-local storage, so glibc could use a different canary value for each thread (but I'm not sure if it does).

The hardening flags adapt to any other compiler options we specify. For example, let's try a static build:

$ ./build.sh LDFLAGS=-static
+ autoreconf -fi
+ ./configure LDFLAGS=-static
...
checking whether C compiler accepts -fPIE... yes
checking whether C++ compiler accepts -fPIE... yes
checking whether the linker accepts -fPIE -pie... no
...
+ make
...
$ file src/test
src/test: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux),
statically linked, for GNU/Linux 2.6.26, not stripped
$ ~/checksec.sh --file src/test
Partial RELRO Canary found NX enabled No PIE

We can't have position independence with static linking. And checksec.sh thinks we aren't RELRO-protecting the PLT — but that's because we don't have one.

Performance

So what's the catch? These protections can slow down your program significantly. I ran a few benchmarks for Mosh, on three test machines:

  • A wimpy netbook: 1.6 GHz Atom N270, Ubuntu 12.04 i386

  • A reasonable laptop: 2.1 GHz Core 2 Duo T8100, Debian sid amd64

  • A beefy desktop: 3.0 GHz Phenom II X6 1075T, Debian sid amd64

In all three cases I built Mosh using GCC 4.6.3. Here's the relative slowdown, in percent.

Protections Netbook Laptop Desktop
Everything 16.0 4.4 2.1
All except PIE 4.7 3.3 2.2
All except stack protector 11.0 1.0 1.1

PIE really hurts on i386 because data references use an extra register, and registers are scarce to begin with. It's much cheaper on amd64 thanks to PC-relative addressing.

There are other variables, of course. One Debian stable system with GCC 4.4 saw a 30% slowdown, with most of it coming from the stack protector. So this deserves further scrutiny, if your project is performance-critical. Mosh doesn't use very much CPU anyway, so I decided security is the dominant priority.

by keegan (noreply@blogger.com) at May 13, 2012 04:59 PM

Kevin Reid (kpreid)

Mikael Vejdemo Johansson (Syzygy-)

Recursively counting numbers with fixed bit counts

I ran across this problem in a reddit side-bar job-ad, and was intrigued by the task (description paraphrased to decrease googleability):

Write a function

uint64_t bitsquares(uint64_t a, uint64_t b);

such that it return the number of integers in [a,b] that have a square number of bits set to 1. Your function should run in less than O(b-a).

I think I see how to do it in something like logarithmic time. Here’s how:

First off, we notice that we can list all the squares between 0 and 64: these are 0, 1, 9, 16, 25, 36, 49, and 64. The function I will propose will run through a binary tree of depth 64, shortcutting through branches whenever it can. In fact; changing implementation language completely, I wonder if I cannot even write it comprehensively in Haskell.

The key insight I had was that whenever you try to find the number of numbers with a bitcount matching some element of some list within the bounds of 0b0000…0000 and 0b000…01111…11, then it reduces to a simple binomial coefficient — n choose k gives the number of numbers with k bits set among the n last. Furthermore, we can reduce the total size of the problem by removing a matching prefix from the two numbers we test from.

Hence, we trace how many bits off the top agree between the two numbers. We count the set bits among these, subtract them from each representative in the list of squares, giving us the counts we need to hit in the remainder.

Write a’ for a with the agreeing prefix removed, and similarly for b’. Then the total count is the count for the reduced things from a’ to 0b000…01…111 plus the count for the reduced things from 0 to b’. The reduction count for b’ needs to be 1 larger than the one for a’ since in one case, we are working with the prefix before the varying bit increases, and in the other, we work with the prefix after the varying bit increases — the latter count is not really from 0 to b’, but this is a useful proxy for the count from 0b0000…010…000 to b’ with the additional high bit set.

In code, I managed to boil this down to:

import Data.Word
import Data.Bits
import Data.List (elemIndices)

bitsquare :: Word64 -> Word64 -> Word64bitsquare a b = bitcountin a b squares -- # integers in [a,b] with square # of 1
s

squares = [1,4,9,16,25,36,49,64] :: [Word64]
allones = [fromIntegral (2^k - 1) | k <- [1..64]]

choose n 0 = 1
choose 0 k = 0
choose n k = (choose (n-1) (k-1)) * n `div` k

popCount :: Word64 -> Word64
popCount w = sum [1 | x <- [0..63], testBit w x]

-- # integers in [a,b] with 1-counts in counts
bitcountin :: Word64 -> Word64 -> [Word64] -> Word64
bitcountin a b counts
| a > b = 0
| a == b = if popCount b `elem` counts then 1 else 0 | (a == 0) && (b `elem` allones) = sum [choose n k | n <- [popCount b], k <- c
ounts]
| otherwise = (bitcountin a' low [c-lobits | c <- counts, c>= lobits]) +
(bitcountin hi b' [c-hibits | c <- counts, c>= hibits])
where
agreements = [(testBit a n) == (testBit b n) | n <- [0..63]]
agreeI = elemIndices False agreements
prefixIndex = last agreeI
prefixCount = sum [1 | x <- [prefixIndex..63], testBit a x]
a' = a .&. (2^prefixIndex - 1)
b' = b .&. (2^prefixIndex - 1)
low = 2^prefixIndex - 1
hi = 0
lobits = prefixCount
hibits = prefixCount+1

by Michi at May 13, 2012 12:04 AM

May 12, 2012

Joachim Breitner

10 years of using Debian

Today, it must have been exactly 10 years that I started using Debian. The story of how I came to Debian shows some of its strengths, so I’ll use this occasion to share it.

I spent the first half of 2002 as an high-school exchange student in Wenatchee, USA. I was already a user of Linux at that time: I made my first contact roughly in 1996 and did my first installation at home two years later, but all that time I was dual-booting and my main system was a well-arranged Windows 98. The machine was a regular tower PC, but nevertheless I put the computer into my trunk when I flew to the US. It took away most of the space, and I had to put some of my cloths inside the case.

So I was there, happily using my Windows and my manually set up “Linux From Scratch” until one day the inevitable happened; inevitable at least until you start doing backups: On April 30th, my hard drive crashed, and took the two systems together with 4 years of personal data with it.

Two weeks later I had a new hard drive and was pondering my options. I did plan to install Windows again; at that time Windows XP was just released. But I wanted a German version of Windows, which would be hard to get there. Also, I did not want to use Linux from Scratch any more, and wanted to make a well-founded choice of a distribution. On the other hand, I really wanted to get my machine up and running quickly, to be able to read my mail more comfortably. I had heard that Debian had good support for network installations (downloading a full 700MB CD was something to avoid at that time), so I grabbed some netinst images, burned a CD, and quickly installed Debian.

I was planning to use the system for about two weeks. I did not pay any particular attention to the setup. Heck, I even picked from my Simpsons sidekick machine naming scheme one that I would not miss being used up (“barney”). Nevertheless, I was using this installation for many years (and many upgrades), until I eventually switched to using laptops. In fact, that very installation is still on the machine somewhere and works. I did install Windows XP a few weeks later as well, but hardly used it. So May 12th of 2002 was when I turned into a full-time Linux and Debian user.

I soon became interested in Debian the project and started to contribute. But that is another story for another ten year anniversary blog post, most likely on October 21, 2013...


Flattr this post

by nomeata (mail@joachim-breitner.de) at May 12, 2012 10:00 AM

May 11, 2012

Yesod Web Framework

Keter: Web App Deployment

tl;dr: There's an experimental deployment system available for testing.

I mentioned on the Yesod and web-devel mailing lists last week that I was unsatisfied with the existing Yesod deployment scripts. I worked on the problem a bit this week, and now have some code worth testing. For now, this system is called Keter, though I expect that to change once someone gives me a better idea.

If you're wondering, Keter is Hebrew for "crown." (You should all be very proud of me; it took a lot of self restraint not to call this post "Yesod's crowning achievement.") Those familiar with kabalah might notice a connection between Yesod and Keter. But I digress.

Advantages

This system already provides quite a bit of convenience for users:

  • Allows updating your app with zero downtime.
  • Checks that new versions of your app actually work before switching to them.
  • Automatically configures PostgreSQL databases as necessary.
  • Monitors processes and restarts them if they crash.

Note: As pointed out on Reddit, this tool by itself is not sufficient to guarantee you'll never have downtime with deployments. One issue is database migrations, which can still cause downtime. My point is that with this approach, as opposed to the previous deployment scripts, it is possibility to have zero downtime between versions.

Usage

Keter is a single executable. The program takes a single argument: a folder from which to run. This folder will have an incoming subfolder, where you'll copy in your keter bundles (more on this in a moment).

Keter relies on a preinstalled Nginx to act as a reverse proxy. It will write out an nginx config file (for now, hardcoded to /etc/nginx/sites-enabled/keter) and then tell Nginx to reload (via /etc/init.d/nginx reload) on any changes.

It also requires a running PostgreSQL server. It will create new accounts and databases as necessary via sudo -u postgres psql. Again, the exact approach used is currently hard-coded, but will be made more extensible going forward.

Keter Bundles

In order to have Keter manage an app, you must package it up in a keter bundle. This is simply a GZIPed tar file with a .keter extension. There is only one required file in this bundle: config/keter.yaml. Let's look at a sample keter.yaml file for a scaffolded Yesod site:

exec: ../dist/build/pgtest/pgtest
args:
    - production
host: pgtest
postgres: true

exec gives the location of the executable, relative to the config file itself. args is the list of command line arguments. host gives the hostname that should be bound to, and postgres indicates whether or not a PostgreSQL database should be created. If postgres is true, then a randomly named database will be created, and the information will be passed to your app via environment variables. Persistent is already configured to accept these variables, so no additional work is necessary on your part.

You might be concerned about that host value. Doesn't it violate DRY to have to specify the host in keter.yaml, and then specify the approot in settings.yml? The answer is yes. That's why updating approot is not required. Keter will set an APPROOT environment variable automatically, based on the host value, and Yesod will automatically use that to override the value in settings.yml.

In other words: that 5-line Yaml file is just about all you need to do to configure your app for deployment. Making a bundle is equally easy, e.g.:

rm -f pgtest.keter
tar zcfv pgtest.keter dist/build/pgtest/pgtest static config

Once Keter is more fully developed and ready to be used, we'll set up the scaffolding to include the keter.yaml file, and add a command like `yesod keter` to create these bundles automatically.

How it Works

The projects README file actually gives a pretty thorough breakdown of the components involved in the code. (Note: as mentioned below, logging has not yet been implemented.) One thing to note is that old versions of apps will be automatically terminated (via SIGTERM) and their folders deleted when a new version is available. This has two important ramifications:

  1. If you want to handle long-running connections, you need to catch SIGTERM and let your app continue to run.
  2. You can't put anything important in your local folder. If you need to persist data, either write it elsewhere in the filesystem, or put it in your database.

As for the code itself, it makes heavy use of message passing. Each component (Nginx manager, process watcher, Postgres configuration) is handled by a separate thread, and messages are passed over Chans. I think this actually makes the code fairly easy to work with, though input from others is greatly appreciated. (This is really my first time making such heavy use of this style.) There isn't much in the way of code comments yet, but that will be fixed going forward as well.

TODO

I still intend to add a comprehensive logging framework. However, I consider myself to be quite a n00b at proper log file maintenance. If anyone wants to brainstorm on the best approach here, I'd appreciate it. Once the log framework is in place, I intend to stick a web frontend on the whole thing to give you access to app logs, in addition to status of apps and possibly more statistics. Input here is welcome as well.

There are also likely plenty of ways to clean up the code. If anyone has ideas, please let me know and/or send a pull request.

May 11, 2012 04:01 PM

May 10, 2012

TypLAB

Today we open up Silk

Today is a good day: we just launched the Silk editor. Everyone can now create their own Silk site.

by Sander Koppelaar at May 10, 2012 01:00 PM

May 09, 2012

Tim Docker

Composable Value Editors

Graphical User Interfaces (GUIs) in haskell are frustrating. It’s not yet clear what is the cleanest model for fitting GUIs into functional programming. Currently there are two main approaches:

  • Various effort at applying Functional Reactive Programming (FRP) to GUIs. These are somewhat experimental, and tend to be proof of concepts implementing a small range of GUI features (several of these libraries are listed here).

  • The full blown toolkits which provide a comprehensive imperative binding to mainstream toolkits. The two key contenders here are gtk2hs and wxHaskell.

Whilst enticing, the FRP approach doesn’t currently look appropriate for building rich GUI applications. wxHaskell and gtk2hs at least provide the functionality required, but the low level imperative approach based in the IO monad is tedious to a fluent haskell developer. Here’s a code snippet:

b <- buttonNew
image <- imageNewFromStock stockAdd IconSizeSmallToolbar
containerAdd b image
set b [buttonRelief := ReliefNone]
on b buttonActivated {
     ... button activated action ...
}

It’s not hard to write this sort of code, but it is tedious, especially considering the amount that is required to build a whole application.

This post outlines my experiments to reduce the amount of imperative code required for GUIs, yet retaining compatibility with the imperative toolkits. Initially I’ve been focussed on "value editors" (VEs) aka "forms". These are GUI components to capture/edit values of ideally arbitrary complexity. I’ve two key goals, composability and abstraction.

Composability: I want to be able to compose my value editors effortlessly. Whilst the existing toolkits let you compose widgets using containers and glue code, it’s verbose indeed.

Abstraction: I’d like to define my VEs independently from the underlying toolkit. But I’m looking for something more than a thin layer over the existing toolkits. I want to define my VEs in terms of the structure of the values involved, and worry about the formatting and layout later, if at all.

If we take this abstraction far enough, it should be possible to reuse our structural VEs definitions beyond gtk2hs and wxWindows. For example, a JSON generator+parser pair can be considered a VE – in the sense that to edit a value, one can generate the json text, edit the text, and then parse to recover the new value. Of course, it’s likely to be a balancing act between abstraction and functionality – we’ll have to see how this pans out.

An Abstract UI

OK, enough preamble, here’s a GADT I’ve devised to capture VEs:

-- | A GADT describing abstracted, user interface components for manipulating
-- values of type a.
data VE a where
    -- | A String field
    Entry :: VE String

    -- | An enumeration. A list of label string are supplied,
    -- the VE value is the integer index of the selected label.
    EnumVE :: [String] -> VE Int

    -- | Annotate a VE with a text label
    Label :: String -> VE a -> VE a

    -- | A "product" VE that combines values from two other VEs
    AndVE :: (VE a) -> (VE b) -> VE (a,b)

    -- | A "sum" VE that captures the value from either of two other VEs
    OrVE  :: (VE a) -> (VE b) -> VE (Either a b)

    -- | A VE for manipulating  a list of values. The supplied function lets the
    -- the VE display the list items to the user (eg for selection).
    ListVE :: (a->String) -> VE a -> VE [a]

    -- | Convert a VE over a type a, to a VE over a type b, given
    -- the necessary mappings. Either String captures the potential
    -- failure in the mapping.
    MapVE :: (a -> Either String b) -> (b -> a) -> VE a -> VE b

    -- | Annotate a VE with a default value
    DefaultVE :: a -> VE a -> VE a

-- A typeclass to build VEs
class HasVE a where
  mkVE :: VE a

(.*.) = AndVE
(.+.) = OrVE
infixr 5 .*.
infixr 5 .+.

And here’s an example usage for a simple data type:

data Gender = Male | Female deriving (Show,Enum)

data Person = Person {
    st_name :: String,
    st_age :: Int,
    st_gender :: Gender
} deriving (Show)

instance HasVE Person
  where
    mkVE = MapVE toStruct fromStruct
        (   Label "Name" nonEmptyString
        .*. Label "Age"   mkVE
        .*. Label "Gender"   mkVE
        )
      where
        toStruct (a,(b,c)) = Right (Person a b c)
        fromStruct (Person a b c) = (a,(b,c))

nonEmptyString :: VE String
nonEmptyString = ...

instance HasVE Int ...
instance HasVE String ...
instance HasVE Gender ...

This captures in some sense the abstract semantics for an editor of Person values. We need to capture:

  • a non-empty string for the name,
  • an integer for the age
  • a gender enumeration

and know how to pack/unpack these into a person value.

A GTK UI

But what can we do with this? We need to turn this abstruct VE into a concrete UI. There’s a library function to do this for an arbitrary VE:

data GTKWidget a = GTKWidget {
    ui_widget :: Widget,
    ui_set :: a -> IO (),
    ui_get :: IO (ErrVal a),
    ui_reset :: IO ()
}

uiGTK  :: VE  a -> IO (GTKWidget a)

The uiGTK function turns our abstract VE a into GTK component for editing a value of type a. In addition to building the compound widget, it gives us functions to:

  • put a value into the widget
  • recover a value from the widget
  • restore the widget to a default value

A higher level function constructs a modal dialog to get a value of type a from the user.

data ModalDialog e a = ModalDialog {
    md_dialog :: Dialog,
    md_gw :: GTKWidget a,
    md_run :: IO (Maybe a)
}

modalDialogNew :: String -> VE a -> IO (ModalDialog a)

Hence running this:

dialog <- modalDialogNew "Example 2" (mkVE :: Person)
ma <- md_run dialog

Results in this:

Example 2

Example 2

The automatically generated dialog is simple, but quite functional:

  • invalid fields have a red background, dynamically updated with each keystroke
  • Fields have sensible defaults – often invalid to force entry from a user

More complex UIs are of course possible. As should be clear from the VE GADT above we support sum and product types, lists, etc, and can map these with arbitrary code. Hence we can construct GTK UIs for a very large range of haskell values. A slightly more complex example composes the previous VE:

data Team = Team {
    t_leader :: Person,
    t_followers :: [Person]
} deriving (Show)

instance HasVE Team ...

Resulting in:

Example 3

Example 3

Recursive types are supported, so its possible to build GTK VEs for expression trees, etc.

JSON Serialisation

As I alluded to previously, given VE a, we can automatically generate a JSON generator and parser for values of type a:

data VEJSON a = VEJSON {
        uj_tojson ::  a -> DA.Value,
        uj_fromjson :: DA.Value -> Maybe a
}

uiJSON :: VE ConstE a -> VEJSON a

Related Work

Well into working on these ideas, I was reminded of two somewhat similar haskell projects: Functional Forms and Tangible Values. Functional Forms aims to ease the creation of wxHaskell dialogs to edit values. The exact purpose Tangeable Values is a little unclear to me, but it appears to be about automatically generating UIs suitable for visualising function behaviour and exploring functional programming.

Future Work

Currently I have a library that implements the VE GADT to automatically build GTK editors and JSON serialisers. There’s many ways to progress this work. Some of my ideas follow…

Whilst the generated GTK editor is a sensible default, there are only very limited ways in which the editor can be customised. I envisage a model where the uiGTK function takes an extra parameter akin to a style sheet, given extra information controlling the UI layout and formatting, etc.

I can envisage many other useful things that could automatically be derived from VE definitions:

  • equivalent functionality for wxHaskell
  • console GUIs
  • Funky UIs implemented with primitives more interesting than the standard toolkit widgets: eg zoomable UIs, or UIs more suited to table based platforms.
  • web GUIs. This could be done by automatically generating javascript and corresponding server side haskell code.

Finally, It might be worth investigate whether the GHC Generic mechansism might be used to automatically generate VE definitions.

So there’s plenty of directions this work can go, but right now I want to put it to the test and build an application!


by Tim Docker at May 09, 2012 03:10 PM

May 08, 2012

Bryce Verdier

Apache Log IP Counting

It all started with a simple question: “Which single IP shows up the most in an apache log file?” To which I ultimately came up with an answer using a string of Bash commands:

 less apache_log | cut -d '-' -f 1 | sort -r | uniq -c

This produced some text displaying how many times various IP addresses made requests from an apache web server. After seeing the results and thinking for a moment, I remembered that there is a “new” data type – as of Python 2.7 and new to me at least – called “Counter” within the collections module in the standard library that would allow me to recreate this in Python. So I quickly whipped up the code below:

  1. #!/usr/bin/env python
  2.  
  3. from sys import argv
  4. from collections import Counter
  5.  
  6. if __name__ == "__main__":
  7. with open(argv[1]) as f:
  8. c = Counter(i.partition("-")[0] for i in f)
  9.  
  10. for k,v in c.most_common():
  11. print '{:>4} {}'.format(v,k)

VIOLA! I get the same results as if I'd done it in Bash. Of course, like the madman I am, I wasn't happy to just stop at Python; I had to write this up in Haskell too. So I said “damn the torpedoes,” fired up my favorite text editor, and started to hack and slash my way toward a third version. After much documentation reading, internet searching, and trial and error I can proudly proclaim, “MISSION ACCOMPLISHED”:

  1. module Main where
  2.  
  3. import System.Environment (getArgs)
  4. import qualified Data.Text as T hiding (map, head, zip)
  5. import qualified Data.Text.IO as TI (readFile)
  6. import Data.Map hiding (map)
  7.  
  8. type Counter = (T.Text, Int)
  9.  
  10. mapUnpack :: Counter -> String
  11. mapUnpack (k,v) = show v ++ " " ++ T.unpack k
  12.  
  13. sortMapByValue :: Map T.Text Int -> [Counter]
  14. sortMapByValue = rqsort . toList
  15.  
  16. -- copied & modified by LYHGG in the recusion chapter
  17. -- doing in reverse order as to better suit the output
  18. rqsort :: [Counter] -> [Counter]
  19. rqsort [] = []
  20. rqsort ((k,v):xs) =
  21. let smallerSorted = rqsort [(k',v') | (k',v') <- xs, v' <= v]
  22. biggerSorted = rqsort [(k',v') | (k',v') <- xs, v' > v]
  23. in biggerSorted ++ [(k,v)] ++ smallerSorted
  24.  
  25. main :: IO ()
  26. main = do
  27. results <- getArgs >>= TI.readFile . head
  28. let results' = map (\x -> T.replace space nothing $ head $ T.splitOn dash x) $ T.lines results
  29. let results'' = fromListWith (+) . zip results' $ repeat (1 :: Int)
  30. mapM_ (print . mapUnpack) $ sortMapByValue results''
  31. where dash = T.pack "-"
  32. space = T.pack " "
  33. nothing = T.pack ""

The Haskell implementation took me longer than I expected as there were a couple of challenges to figure out:
1) Learning the Data.Map datatype and figuring out how to build that datatype from a list of IP addresses.
2) Sorting the Map by value and not by key.
3) Unpacking the Map to be printed.

The first problem was solved by a lucky internet search that showed me an example of how to use the 'fromListWith' function. This function performed the heavy lifting of sorting and counting the IP addresses in the log file. For the sorting by value instead of by key problem I was able to construct my own solution by tweaking the quicksort example from Learn You a Haskell. The changes I made expanded the tuple, allowing it to compare values and sort them in descending order. For the text holding and manipulation involved in Map unpacking, I've been told by other Haskellers that Data.Text is “the way to go” (as the default String implementation is a little slow and lacking features). While Data.Text did provide me an easy way to split the string and grab the IP addresses, it also required translating the IP addresses back into a String data type before Haskell would print it. Thus my need for a specific function to create a string based on each item in the Map. In the grand scheme of things having to write the mapUnpack function wasn't horrible...it was just one more hoop that I had to jump through before I could call this project complete. After these modifications I was able to put this program together without TOO much hassle, and in the end got the exact same results as both the Bash string and the Python program.

Would I recommend writing these scripts for work instead of using the string of Bash commands? No, especially if you're asked this question in a high-stress situation. However, these little personal challenges were a great way to expand my programming skills, particularly in learning about new libraries like the collections module in Python and the Data.Map module in Haskell. Having this random new knowledge might not seem worthwhile upfront, but might come in handy in the future if I ever encounter a problem that simple strings of Bash commands can't handle.

by Bryce at May 08, 2012 03:55 PM

May 07, 2012

Joachim Breitner

Free Groups in Agda

I must say that I do like free groups. At least whenever I play around with some theorem provers, I find myself formalizing free groups in them. For Isabelle, my development of free groups is already part of the Archive of Formal Proofs. Now I became interested in the theorem prover/programming language Agda,so I did it there as well. I was curious how well Agda is suited for doing math, and how comfortable with intuitionalistic logic I’d be.

At first I wanted to follow the same path again and tried to define the free group on the set of fully reduced words. This is the natural way in Isabelle, where the existing setup for groups expects you to define the carrier as a subset of an existing type (the type here being lists of generators and their inverses). But I did not get far, and also I had to start using stuff like DecidableEquivalence, an indication that this might not go well with the intuitionalistic logic. So I changed my approach and defined the free group on all words as elements of the group, with a suitable equivalence relation. This allowed me define the free group construction and show its group properties without any smell of classical logic.

The agda files can be found in my darcs repository, and the HTML export can be browsed: Generators.agda defines the sets-of-generators-and-inverses and FreeGroups.agda (parametrized by the Setoid it is defined over) the reduction relation and the group axioms. Here are some observations I (disclaimer: Agda-beginer) made:

  • Fun fact: Free groups exist not only in classical logic.
  • Without any automation as in Isabelle, even simple things get quite complicated. A simple substitution of an equality with subst requires me to specify not only the equality and the term I want it to apply, but also to repeat the common part of the terms. Or when using the associativity of list concatenation, I have to pass all three sublists to the lemma. Maybe I am a bit spoiled by Isabelle, but I’d be worried that this would prevent large proofs.
  • The levels are also annoying. Although my theory stays within one level, I have to annotate it everywhere. I’d expect the type inference to figure this out for me.
  • Equality reasoning with begin ... ∎ is quite nice and surprisingly well readable.
  • Besides the additional work, it is nice to be able to do the proof in almost all detail. There is a limitation, though, as some steps are done automatically (if they happen to occur when evaluating/normalizing a term) and the others, even if similar-looking, are not.
  • It’d be great if one would be free in the choice of editor, but vim users generally have a hard time in the field of theorem provers.

If I were to extend this theory, there are two important facts to be shown: That there is a unique reduced word in every equivalence class (norm_form_uniq), and the universal property of the free group. For the former (started in NormalForm.agda) I’m missing some general lemmas about relations (e.g. that local confluence implies global confluence, and even the reflexive, symmetric, transitive hull is missing in the standard library). For the latter, some general notions such as a group homomorphism need to be developed first.

I planned to compare the two developments, Isabelle and Agda. But as they turned out to show quite things in different orders, this is not really possible any more. One motivation to look at Agda was to see if a dependently typed language frees me from doing lots of set-element-checking (see the “mems” lemma in the Isabelle proof of the Ping-Pong-Lemma). So far I had no such problems, but I did not get far enough yet to actually tell.

Thanks to Helmut Grohne for an educating evening of Agda hacking!


Flattr this post

by nomeata (mail@joachim-breitner.de) at May 07, 2012 01:24 PM

Ken T Takusagawa

[spovtkwt] Equiangular pentagon

Finding not many good pictures of a pentagon with equal angles but unequal sides, I drew one. For reference, inside it is a regular pentagon; corresponding sides are parallel.

Equiangular Pentagon

Haskell source code is here.

And in Scalable Vector Graphics (SVG) format:
<object data="http://mit.edu/kenta/www/three/pentagon-equiangular/spovtkwt/pentagon.svg" height="473" type="image/svg+xml" width="600"></object>

Equiangular Pentagon in SVG

Someday, an amoeba-like animation which explores possible shapes of an equiangular pentagon of a constant area.

by Ken (noreply@blogger.com) at May 07, 2012 06:31 AM

Leon P Smith

Announcing postgresql-simple 0.1

A new release of postgresql-simple is now available. Version 0.1 introduces some breaking changes as well as some significant improvements to the interface. I’ll cover the biggest and best change in this post, for other changes you should read the changelog.

A few months ago, Ozgun Ataman suggested that I add some convenience code to simplify writing instances of QueryResults. I ended up liking his suggestion so much I decided to dig deeper into postgresql-simple and rebase the implementation around his interface. This allowed me to eliminate some intermediate data structures, and has some other advantages I’ll get to later. I’m rather pleased with the end result; here is a contrast of the old and the new:

class QueryResults a where
    convertResults :: [Field] -> [Maybe ByteString] -> Either SomeException a
 
instance (Result a, Result b) => QueryResults (a,b) where
    convertResults [fa,fb] [va,vb] = do
        !a <- convert fa va
        !b <- convert fb vb
        return (a,b)
    convertResults fs vs  = convertError fs vs 2



-- RowParser has instances for Applicative, Alternative, and Monad
class FromRow a where
    fromRow :: RowParser a

-- field is exported from Database.PostgreSQL.Simple.FromRow
field :: FromField a => RowParser a

instance (FromField a, FromField b) => FromRow (a,b) where
    fromRow = (,) <$> field <*> field

The new interface not only eliminates a significant amount of syntactic overhead, it also allowed me to automatically force the converted values to WHNF, which eliminates the old caveats about writing QueryResults instances. (Note that there are (still) caveats to writing FromField instances, even if they weren’t documented before.) And perhaps more interestingly, it enables new forms of composition. For example, the Types module defines a pair type for composing FromRow instances as follows:

data a :. b = a :. b 

infixr 3 :.

instance (FromRow a, FromRow b) => FromRow (a :. b) where
   fromRow = (:.) <$> fromRow <*> fromRow

Here’s an example of how you might use this. Let’s pretend we have a simple schema for that might be used for a (small) website where people can post stories and rate them from 0-9:

CREATE TABLE users                     (uid int not null, username text   not null);
CREATE TABLE posts   (pid int not null, uid int not null, content  text   not null);
CREATE TABLE ratings (pid int not null, uid int not null, score    float4 not null);

Then, here’s some example boilerplate that could be used to represent (part of) the schema in Haskell:

data User = User { u_uid :: Int, username :: Text }

instance FromRow User where
  fromRow = User <$> field <*> field

data Post = Post { p_pid :: Int, p_uid :: Int,  content :: Text } 

instance FromRow Post where
  fromRow = Post <$> field <*> field <*> field

type Score = Float

And finally, here is an example of how one could get the data out of the database in a form almost ready to generate a web page:

    rows :: [User :. Post :. Only Score] <- 
       query conn [sql|  SELECT u.*, p.*, avg(r.score) as avg_score
                           FROM users as u JOIN posts as p JOIN ratings as r  
                                ON u.uid = p.uid AND p.pid = r.pid
                          GROUP BY p.pid
                          ORDER BY avg_score DESC                             |] ()
    forM rows $ \(user :. post :. Only score) -> do
        -- generate the HTML for a single post here

Now of course, this isn’t meant to be a serious sketch of such a post/rating system; there are plenty of performance problems that are perfectly obvious to me, and maybe to you too. This is just to give you some ideas about how this functionality might be used: for example, to deal with (a limited class of) joins, or to deal with a computed column that’s been affixed to the table.

Some unresolved issues at this point would be dealing with natural joins and outer joins. I suspect that an adequate solution to these problems may require more breaking changes to this interface. In any case, multiple people have been asking for this kind of functionality, and I do think this interface is a step in the right direction.

For questions, comments, or suggestions relating to postgresql-simple, I suggest joining database-devel, a new mailing list pertaining not only to postgresql-simple, but also database development in Haskell in general. Also I’m often willing to answer questions if you happen to catch me on Freenode.


by lpsmith at May 07, 2012 03:56 AM

May 05, 2012

Russell O'Connor

ACM Followup

I would like to post a follow-up on my recent run in with the ACM’s copyright policies. First I give some links to some of internet cometary related to my incident.

The most interesting response I got was an e-mail from my friend Peter. He wrote that I should have published under a Creative Commons licence and then transferred the copyright to the ACM and that there would be nothing illegal nor misleading about doing this.

More interestingly, he pointed me to an obscure part of U.S. copyright law: section 203. This section describes how to terminate copyright transfers and licences that occurred after January 1st, 1978. Termination of the transfer is allowed between 35 and 40 years after the copyright transfer (there is only a 5 year window). This means that starting January 1st, 2013, academic researches who have transferred copyright in order to publish articles in 1978 may have their copyright returned to them. Then the researchers (or their decedents if the researcher is deceased) can do what they like with their articles, though I would recommend republishing them on the arXiv or some similar open access repository. Alternatively, they could also start suing their U.S. publisher or anyone else in the U.S. who continues to distribute their articles behind a paywall.

Termination of copyright transfer requires advanced notice be given to the publisher between 2 years and 10 years before the termination date, and the termination date must fall between 35 and 40 years after the copyright transfer was made. If people really want to hurt Elsevier and/or other academic publishers, then today we can starting sending them notices of termination of copyright transfer for any article published between 1978 and 1987.

It would be really helpful if a lawyer could draft a fill-in the blank legal document for academic researchers to create these notifications.

May 05, 2012 05:39 PM

Kevin Reid (kpreid)

Done, done, done,

Done.

I have completed my last final exam and my last course project of my last semester. I will be graduating a week from now. I have a job starting in July.

DONE.

by Kevin Reid (kpreid) (kpreid@switchb.org) at May 05, 2012 12:58 AM

May 04, 2012

Shayan Najd Javadipour

Kickoff

Hello dear visitors,

I am excited!
I am excited cause my proposal “Haskell-Type-Exts”, submitted to haskell community, has been accepted for Google Summer of Code 2012!
I am going to have lots of fun this summer.
I am excited cause I am going to write a clean[2] typechecker for ASTs extracted by HSE[1].
HSE is one of the most widely used (if not the most popular) standalone parsers for Haskell 2010 plus a wide variety of extensions.
HSE is written (and maintained) by Niklas Broberg and he is also mentoring this project.

Functional programming is cool, Haskell is awesome and definitely working on its type system would be exciting.

Now that I have this luxury (thanks to Haskell.org committee), I would like to share it with you here, in this weblog.

Stay tuned if you are interested in Haskell and its type system.

Welcome!

/Shayan

[1] http://hackage.haskell.org/package/haskell-src-exts
[2] What I mean of “clean” is so far a mystery :)


by cleantypecheck at May 04, 2012 10:57 AM

Johan Tibell

New major release of the containers package

I'm proud to announce a new major release of the containers package. There has been a whopping 160 commits since the last release. The major improvements are:

  • a clearer distinction between value-lazy and value-strict containers,
  • performance improvements across the board,
  • a big internal clean-up, and
  • new functions for e.g. merging, updating, and searching containers.

The first item is perhaps the most visible. While the old Data.Map and Data.IntMap modules will continue to exist for the foreseeable future, we've abandoned the practice of having the strict and lazy versions of each function distinguished by an apostrophe. The distinction is instead made at the module level, by introducing four new modules:

  • Data.Map.Strict
  • Data.Map.Lazy
  • Data.IntMap.Strict
  • Data.IntMap.Lazy

This split has three benefits:

  • It makes the choice between value-strict and value-lazy containers more declarative; you pick once at import time, instead of having to remember to use the strict or lazy versions of a function every time you modify the container.
  • It alleviates a common source of performance issues, by forcing the user to think about the strictness properties upfront. For example, using insertWith instead of insertWith' is a common source of containers-related performance bugs.
  • There are fewer functions per module, making it easier to get an overview of each module.

Note that the types used in the strict and lazy APIs are the same, so you can still use the same container in a "mixed" manner, if needed.

Contributors to this release:

  • Milan Straka
  • Johan Tibell
  • Joachim Breitner
  • Edward Z. Yang
  • Herbert Valerio Riedel
  • Matt West
  • Max Bolingbroke

by Johan Tibell (noreply@blogger.com) at May 04, 2012 02:51 AM

May 03, 2012

Shayan Najd Javadipour

Inquiry

Now that I am going to start the project, I need you, as potential users, to write me a few lines about your expectations by answering the following two questions:

1. What do you expect from Haskell-Type-Exts? What features (preferably with priorities)?

2. Where are you going to use it? Concrete use-cases are highly appreciated.

Looking forward to hear your comments…
Thanks in advance!


by cleantypecheck at May 03, 2012 08:05 PM

May 02, 2012

Bryn Keller

Outnumbered by Robots

I heard a piece on the radio the other day about robots. One reporter told the story of how he’d driven across the United States almost without speaking to another human being the whole time. Food via self-checkout at the grocery store, hotels via robots check-in systems, etc. This was all very amusing at first. Then the discussion turned to jobs.

It turns out that more and more people are being put out of a job by robots. Not just repetitive factory work, mind. Pathologists. Sports reporters. Soon, drivers.  There are, as we all know, lots of things computers do better than we do. That list is growing all the time, and it’s growing faster. Moore’s law says that computing power doubles every eighteen months, and if that continues to hold true, we’re going to start seeing almost unimaginable science fiction-type robots and computers, very, very soon.

People seem not to be paying attention to this issue. I don’t want to call it a problem, because it’s just reality. However, for many (most? all?) of us, it will feel like a problem. It’s not just a matter of losing your current job, of suffering economic loss until you can retrain and find your balance in this new world. It’s a matter of being completely outmoded.

There’s really nothing special about humans. Unusual, perhaps, but not unique. People routinely claim that this or that trait is “uniquely human,” without the faintest justification. My cats have most of these “uniquely human” characteristics, too.  There are others that the cats don’t have, but they’re not uniquely human, either. Laughter was the one I heard last. Turns out rats laugh too. Tool use is an old one. Chimps. Ravens. Lots of things use tools. Not unique.

There’s also nothing about carbon that makes it uniquely better for supporting life than silicon. True, there are things that we’re better at than computers are, today. We have emotions, better vision, better language processing, a few other advantages. This will change. Computers are getting better, smarter, faster, every single day, and we’re staying basically the same. Evolution is obsolete. Robots will be better than us in every conceivable way, sometime soon. Maybe within my lifetime. My kids, if I ever have any, might well be the last human generation, unless we figure out some kind of way to live in a world where robots are better than us at everything.

We have to figure this out. Nobody’s going to stop improving robots or computers. To keep this from happening, we’d have to have everyone agree to stop. At the same time. Sort of like how we collectively decided to restrain the development of nuclear weapons. The important difference, though, is that the atom bomb looks like the end of the world. Better robots just look like a pile of cash for the people who make them. There’s no stopping piles of cash.

What we need to do is come up with a plan for how to deal with this transition. Can we come up with some way to improve ourselves as quickly as we improve technology? Can we somehow remain as we are, yet stay in control and actually have our lives improved? There are many unknowns. This could be a coming golden age, or it could easily be crisis in the making, as big a problem as clean drinking water or global climate change. Take some time to think about it. How can we prepare for the day when robots do everything better than we do?

by xoltar at May 02, 2012 08:31 PM

Wolfgang Jeltsch

Dependently typed programming and theorem proving in Haskell

Programming languages with dependent types allow us to specify powerful properties of values using the type system. By employing the Curry–Howard correspondence, we can also use these languages as proof languages for higher-order logics. In this blog post, I want to demonstrate that Haskell as supported by the Glasgow Haskell Compiler (GHC) can give us almost the same features.

The complete article can be seen as a kind of literate program. Extracting all code snippets and glueing them together yields a valid Haskell module. For your convenience, you can download the complete code as an ordinary Haskell source code file.

Prerequisites

Plain Haskell 2010 is not enough for emulating dependent types. We need support for generalized algebraic data types and for type synonym families. Furthermore, we want to use operator syntax on the type level for making our code easier to read. We declare the necessary language extensions:

{-# LANGUAGE GADTs, TypeFamilies, TypeOperators #-}

We want to define better typed versions of certain Prelude functions. For convenience, we want them to have the same names as their Prelude counterparts. Therefore, we hide the corresponding Prelude functions:

import Prelude hiding (head, tail, (++), (+), replicate)

Since Haskell does not come with a type for natural numbers, we make use of the natural-numbers package:

import Data.Natural

We are now ready to turn to the actual development.

Dependently typed programming

Dependent types are types that are parameterized by values. In Haskell, however, parameters of types have to be types again. So we represent values by types. Let us look how this works for natural numbers. The usual ADT declaration of natural numbers uses two data constructors Zero and Succ. These data constructors become type constructors now:

data Zero

data Succ nat

Note that it does not matter what values the types Zero and Succ cover. It is most reasonable to leave these types empty.

Now we define a list type with a length parameter almost like we would do in Agda:

data List el len where

    Nil  ::                      List el Zero

    Cons :: el -> List el len -> List el (Succ len)

If we let GHCi compute the type of

Cons 'H' $ Cons 'i' $ Cons '!' $ Nil ,

for example, it will correctly answer with

List Char (Succ (Succ (Succ Zero))) .

We can drop the static length information from lists by converting them to Haskell’s ordinary list type:

toList :: List el len -> [el]
toList Nil           = []
toList (Cons el els) = el : toList els

Using toList, we implement a Show instance for displaying values of type List in GHCi:

instance (Show el) => Show (List el len) where

    show = show . toList

Let us actually make use of the length parameter in list types. We first construct a function head, which can only be applied to non-empty lists and thus cannot fail:

head :: List el (Succ len) -> el
head (Cons el _) = el

The type of the corresponding function tail also shows that the length of a list’s tail is one less than the length of the original list:

tail :: List el (Succ len) -> List el len
tail (Cons _ els) = els

A more advanced example of using static length information is list concatenation. Let us construct a dependently typed version of the well-known ++-operator. Its type should specify that the length of the result is the sum of the lengths of the arguments. So we have to implement addition of type-level naturals. We introduce a type synonym family for this:

infixl 6 :+

type family nat1 :+ nat2 :: *

type instance Zero      :+ nat2 = nat2
type instance Succ nat1 :+ nat2 = Succ (nat1 :+ nat2)

We can now define list concatenation as follows:

infixr 5 ++

(++) :: List el len1 -> List el len2 -> List el (len1 :+ len2)
Nil         ++ list = list
Cons el els ++ list = Cons el (els ++ list)

Note that the type checker guarantees that our implementation of concatenation really fulfills the length constraint expressed in the type.

We want to implement a dependently typed version of the Prelude function replicate, which takes a natural number len and some value val and returns a list that contains len times the value val. Its desired type would be

(len : Nat) -> el -> List el len

in an Agda-like language. This type gives the first argument the name len. Since len is used in the result type, the type of a result depends on the concrete argument.

The problem for our Haskell emulation of dependent types is that the len argument of the function must be a value, while the len parameter of List must be a type. So we have to link value-level naturals and type-level naturals. We do this by defining a type Nat with one parameter such that for each n ∈ ℕ with a type-level representation nat, the type Nat nat contains exactly one value, which represents n:

data Nat nat where

    Zero ::            Nat Zero

    Succ :: Nat nat -> Nat (Succ nat)

If we ask GHCi for the type of the value

Succ (Succ (Succ Zero)) ,

we get the answer

Nat (Succ (Succ (Succ Zero))) ,

which demonstrates that the value is now reflected at the type level.

As in the case of List, we can define a conversion function for forgetting the type parameter of Nat, and a Show instance based on it:

toNatural :: Nat nat -> Natural
toNatural Zero       = 0
toNatural (Succ nat) = succ (toNatural nat)

instance Show (Nat nat) where

    show = show . toNatural

Analogously to the implementation of (++), we can implement addition for values of type Nat:

infixl 6 +

(+) :: Nat nat1 -> Nat nat2 -> Nat (nat1 :+ nat2)
Zero      + nat2 = nat2
Succ nat1 + nat2 = Succ (nat1 + nat2)

However, we actually wanted to implement a dependently typed replicate. We can do this as follows:

replicate :: Nat len -> el -> List el len
replicate Zero       _  = Nil
replicate (Succ len) el = Cons el (replicate len el)

The definition of Nat ensures that the first argument of replicate and the type variable len represent the same natural number. So the type of replicate guarantees that the first argument determines the length of the result list.

Theorem proving

Theorem proving in a dependently typed programming language works by exploiting the Curry–Howard correspondence. Propositions correspond to types and their proofs correspond to expressions of the corresponding types. In this blog post, we will only discuss how to prove equations. Proving other things works analogously.

Let us first define the equality predicate. We can define a predicate by introducing a corresponding GADT. The data constructors of this GADT serve as atomic proofs, their types correspond to axioms. A type constructor (:==) that implements equality is defined as follows:

infix 4 :==

data val1 :== val2 where

    Refl :: val :== val

We implement a Show instance for (:==) that enables us to show the only possible normal form of an equality proof, which is Refl:

instance Show (val1 :== val2) where

    show Refl = "Refl"

We will now prove that 0 is a right unit of addition, that is,

n ∈ ℕ . n + 0 = n .

It might seem reasonable to give the proof the polymorphic type

nat :+ Zero :== nat .

However, the proof has to be done by induction, so that we would have to do pattern matching on the natural number parameter. We cannot do pattern matching on a type, so we have to pass the natural number represented by nat as a value of type Nat nat. We implement the proof as follows:

rightUnit :: Nat nat -> nat :+ Zero :== nat
rightUnit Zero       = Refl
rightUnit (Succ nat) = case rightUnit nat of
                           Refl -> Refl

As a second and last example, we proof that addition is associative, that is,

n₁, n₂, n₃ ∈ ℕ . (n₁ + n₂) + n₃ = n₁ + (n₂ + n₃) .

The Haskell code is as follows:

assoc :: Nat nat1                                          ->
         Nat nat2                                          ->
         Nat nat3                                          ->
         (nat1 :+ nat2) :+ nat3 :== nat1 :+ (nat2 :+ nat3)
assoc Zero        nat2 nat3 = Refl
assoc (Succ nat1) nat2 nat3 = case assoc nat1 nat2 nat3 of
                                  Refl -> Refl

Note that we pass all three natural numbers involved as values of Nat, although we only do pattern matching on the first one. The reason is that we have to specify the second and third number in the recursive call. If we would not do this, they would have to be inferred, which turns out to be impossible.

So have we shown that Haskell can replace Agda if we are willing to write a bit of boilerplate code? Not really. A crucial property of Agda, which is important for proving theorems, is that all values are fully defined. In Haskell, however, values may be partially or completely undefined. In particular, the value ⊥, which denotes undefinedness, is an element of every type. So every proposition has at least one proof, which makes our logic inconsistent.

A solution is to accept a value as a proof only if it does not contain ⊥. We can check if ⊥ is present by evaluating the value completely. This succeeds if and only if there is no ⊥. There are two problems with this approach. First, it causes a runtime overhead, which is not necessary when using a language like Agda. Second, complete evaluation is not possible for arbitrary functions, since it would mean applying the functions in question to all possible arguments. Since function types correspond to implications, and implications occur often in general statements, this is a serious drawback.

So theorem proving is obviously something that should be left to languages that can handle it better than Haskell can. That said, I think that it is nevertheless fun to see how far we can go with present-day Haskell.


Tagged: Agda, Curry–Howard correspondence, dependent type, functional programming, GADT, GHC, Haskell, higher-order logic, literate programming, logic, natural-numbers (Haskell package), theorem proving, type family, type-level programming

by Wolfgang Jeltsch at May 02, 2012 09:38 AM

May 01, 2012

Wolfgang Jeltsch

Category theory crash course

I recently gave an introduction to category theory at the Institute of Cybernetics. This crash course introduced several of the basic categorical notions that are of interest from a computer science and logic perspective, while assuming no previous knowledge about category theory. Everything necessary to understand my upcoming talk about categorical models of modal logics was covered.

The slides of the crash course are now online. If you have questions regarding the material, please ask; for example, via a comment to this blog post.


Tagged: categorical logic, category theory, logic, modal logic, temporal logic, TTÜ Küberneetika Instituut

by Wolfgang Jeltsch at May 01, 2012 07:56 AM

Tom Moertel

Test like you're betting for your life

In software, as in life, every problem you solve is actually part of a larger problem, an optimization problem in which you are trying to squeeze as much as you can from a single scarce resource: yourself. Every hour you spend solving one problem is an hour you no longer have to solve other problems. At the end of your life, when you receive the final tally of how you spent your scant allotment of hours, you probably don’t want to discover that you wasted most of them on problems that didn’t matter.

Which brings us to David Heinemeier Hansson’s recent post on Testing like the TSA, in which he advises against wasting your time writing tests that are unlikely to find problems. He’s right.

Whether you realize it or not, every time you write a test, you are placing a bet. You are betting that the test’s expected payoff will outweigh the test’s expected opportunity cost. Instead of writing that test, would you be better off doing something else? For every test you write, you are betting that the answer is no.

But being right about this answer – reliably right – is tricky. It requires that you have a good understanding of which bets are likely to pay off and which are not, which are cheap to place and which are expensive. In other words, it requires that your internal probability distributions be well calibrated for the kind of code you write and the environment in which you code.

New programmers, unfortunately, are poorly calibrated. They have little experience to draw upon and end up imitating other people’s probability distributions. These they pick up, for better or worse, from folklore – best practices, development methods, expert opinion, the guy down the hall. As a result, new programmers concentrate their nascent probability distributions around the things they learn first and, consequently, over-bet on those things.

But even the best programmers over-bet on the things they know. That’s because they under-bet in their blind spots, and we all have blind spots.

In general, we never bet on the things we don’t know. If all you know about getting your code right is TDD, you’ll never bet on types or proofs or constructive correctness because you don’t know how to place those bets. But those bets are often dirt cheap and pay in spades. If you’re not betting on them at least some of the time, whatever you are betting on probably costs more and pays less. You could be doing better.

But to do better, you have to know what other bets are out there. You have to know what it costs to place them and how likely they are to pay off. To gain this knowledge, you’ll have to go beyond what is comfortable; you’ll have to explore those dark, distant parts of the casino and place some trial bets.

If you’ve never written a unit test, write some. If you always write your tests after your main logic, write them before. If you always write them before, write them after. If all you ever do is unit testing, try integration and system tests, too. If you’ve never done QuickCheck-like property checking, do it. If you don’t know how to prove things about your code, write some proofs. If you don’t know how to represent important properties as types so that your computer will prove them for you, now’s the time to learn. If you don’t know how to design APIs so that certain classes of problems become impossible, start practicing.

And, after you place your bets, pay attention to what happens down the road. Learn from what you observe. Learn which bets are likely pay off now, pay off later, and pay off never.

The more bets you know how to place, and the more you know about their expected payoffs, the better you’ll place your bets. And the better you place your bets, the more problems you’ll solve, problems that actually matter, before you run of out of time.

Remember: Your supply of hours is finite. If you care about making those hours matter, learn how to bet them wisely. Both in software, and in life.

Edited 2012-04-30 16:56 to add note about integration and system testing and also about paying attention to what happens with your bets so that you learn about expected payoffs. Thanks to grauenwolf for the editorial feedback.

by Tom Moertel at May 01, 2012 05:07 AM

April 30, 2012

David Amos

CHAs IV: Hopf Algebra Morphisms


In the last few posts, we've been looking at combinatorial Hopf algebras, specifically:
- SSym, a Hopf algebra with a basis of (indexed by) permutations
- YSym, with a basis of binary trees
- QSym, with a basis of compositions

It turns out that these three Hopf algebras are quite closely related (and indeed, there are a few others in the same "family"). Let's start with SSym and YSym.

Given a permutation p of [1..n], we can construct a binary tree called the descending tree as follows:
- Split the permutation as p = ls ++ [n] ++ rs
- Place n at the root of the tree, and recursively place the descending trees of ls and rs as the left and right children of the root
- To bottom out the recursion, the descending tree of the empty permutation is of course the empty tree

For example, the following diagram shows the descending tree of [3,5,1,4,2].


Here's the Haskell code:

descendingTree [] = E
descendingTree [x] = T E x E
descendingTree xs = T l x r
    where x = maximum xs
          (ls,_:rs) = L.break (== x) xs
          l = descendingTree ls
          r = descendingTree rs

> :m Math.Algebras.Structures Math.Combinatorics.CombinatorialHopfAlgebra
> descendingTree [3,5,1,4,2]
T (T E 3 E) 5 (T (T E 1 E) 4 (T E 2 E))

Now you'll recall that for the Hopf algebra YSym, although we sometimes carry around the node labels to help us see what is going on, we're really only interested in the shapes of the trees. So here's a function to erase the labels:

shape :: PBT a -> PBT ()
shape t = fmap (\_ -> ()) t

> shape $ T (T E 3 E) 5 (T (T E 1 E) 4 (T E 2 E))
T (T E () E) () (T (T E () E) () (T E () E))

Thus (shape . descendingTree) is a function from permutations to unlabelled trees. We can consider this as a map between the fundamental bases of SSym and YSym, which therefore induces a linear map between the corresponding Hopf algebras:

descendingTreeMap :: (Eq k, Num k) => Vect k SSymF -> Vect k (YSymF ())
descendingTreeMap = nf . fmap (\(SSymF xs) -> (YSymF . shape .  descendingTree) xs)

Now it turns out that this map is in fact a Hopf algebra morphism. What does that mean? Basically it means that descendingTreeMap plays nicely ("commutes") with the unit, mult, counit, comult, and antipode maps in the two Hopf algebras.

For example, for an algebra morphism, we require:
f (x1*x2) = f x1 * f x2
f . unit = unit


It's not immediately clear why descendingTreeMap should have these properties. The unit property is clear:
> descendingTreeMap 1 == 1
True
or put another way
> descendingTreeMap (ssymF []) == ysymF E
True

But what about the mult property?

Recall that in SSymF, we multiply permutations by shifting the right operand, and then summing all possible shuffles of the two lists:

> ssymF [3,2,1] * ssymF [1,2]
F [3,2,1,4,5]+F [3,2,4,1,5]+F [3,2,4,5,1]+F [3,4,2,1,5]+F [3,4,2,5,1]+F [3,4,5,2,1]+F [4,3,2,1,5]+F [4,3,2,5,1]+F [4,3,5,2,1]+F [4,5,3,2,1]

On the other hand, in YSymF, we multiply by multi-splitting the left operand, and then grafting the parts onto the leafs of the right operand, in all possible ways. The following diagram shows one possible multi-split-grafting, corresponding to one of the summands in [3,5,1,4,2] * [1,2]:

The numbers along the tops of the trees are the node labels generated by the descending tree construction. We can see from these that there is an exact correspondence between a shifted shuffle in SSymF and a multi-split grafting in YSymF. The asymmetry in the mult for YSymF, where we graft multi-splits of the left operand onto the right operand, corresponds to the asymmetry in the mult for SSymF, where we shift the right operand. This shifting in SSymF ensures that the nodes for the right operand end up at the root of the descending tree, as required by the grafting in YSymF. When we defined shuffling, we gave a recursive definition, but it's fairly clear that the grafting of the parts of the multi-split onto the right tree is accomplishing the same thing.


Similarly, a coalgebra morphism is a linear map which commutes with counit and comult:

In SSymF, counit is 1 on the empty permutation, 0 on anything else. In YSymF, counit is 1 on the empty tree, 0 on anything else. The descendingTreeMap sends the empty permutation to the empty tree, so it's clear that it commutes with counit in the required way.

What about comult? In SSymF, the comult of a permutation is the sum of its two-part deconcatenations (with each part flattened back to a permutation).

> comult $ ssymF [3,5,1,4,2]
(F [],F [3,5,1,4,2])+(F [1],F [4,1,3,2])+(F [1,2],F [1,3,2])+(F [2,3,1],F [2,1])+(F [2,4,1,3],F [1])+(F [3,5,1,4,2],F [])

In YSymF, the comult of a tree is the sum of its two-part splits. Now it's clear that flattening makes no difference to the descending tree. Then, it's also clear that if you take descending trees of the two parts of a deconcatenation, this corresponds to a split of the descending tree of the whole.

Finally, a Hopf algebra morphism is a bialgebra morphism which in addition commutes with antipode.

In SSymF and YSymF, we didn't give an explicit expression for the antipode, but instead derived it from mult and comult using the fact that they are both graded connected bialgebras. So it's actually kind of obvious that descendingTreeMap will be a Hopf algebra morphism, but just to check:

prop_HopfAlgebraMorphism f x = (f . antipode) x == (antipode . f) x

> quickCheck (prop_HopfAlgebraMorphism descendingTreeMap)
+++ OK, passed 100 tests.

Given any tree, there are many ways to label the nodes so that the tree is descending (ie such that the label on each child node is less than the label on its parent). For example, we could first label all the leaf nodes [1..k], and then all their immediate parents [k+1..], and so on. (For our example tree, this would lead to the alternative labelling [1,5,2,4,3].)

This shows that the descending tree map is surjective, but not injective. Hence YSym is a quotient of SSym.

by DavidA (noreply@blogger.com) at April 30, 2012 08:35 PM

Brandon Simmons

Converting HTML5 canvas elements to images

I recently needed to do some conversions/processing to a bunch of HTML files, which contained html canvas images, so I needed a way of converting all the canvas elements on the page to PNGs.

After a bit of research and tweaking, the following worked well enough for me:

<script src="https://gist.github.com/2561736.js?file="></script> <noscript>
$('canvas').each(function(i,e){ var img = e.toDataURL("image/png"); $(e).replaceWith(  $('<img src="'+img+'"/>').attr({width: $(e).attr("width"), height: $(e).attr("height"), style: $(e).attr("style") }) ) });
</noscript>

Run it in your browser's JS console on the page you want to process (we assume jQuery is already loaded on the page).

April 30, 2012 06:47 PM

Philip Wadler

What Must be Said

Gunter Grass continues to face accusations of antisemitism for his poem `What Must Be Said', which predicts that very reaction.
But why do I forbid myself
to name that other land in which
for years—although kept secret—
a usable nuclear capability has grown
beyond all control, because
no scrutiny is allowed.

The universal silence around this fact,
under which my own silence lay,
I feel now as a heavy lie,
a strong constraint, which to dismiss
courts forceful punishment:
the verdict of “Antisemitism” is well known.
...
Why is it only now I say,
in old age, with my last drop of ink,
that Israel’s nuclear power endangers
an already fragile world peace?

by Philip Wadler (noreply@blogger.com) at April 30, 2012 11:01 AM

JP Moresmau

EclipseFP progress report April 2012

There's not going to be a new EclipseFP release at the end of April, I'm not going to keep the monthly release rhythm I had over the past couple of months. In fact, I have a few simple things I could have done that warranted a release (thanks tobbebex for all the suggestions), but I had already started the next big step, so I want to finish what I'm on before moving to other things.
The goal now is to have a database of usages over all projects in the workspace, so you can see where each module is imported, where each function is used, and then potentially do things like workspace-wide renames and such.
The way it's implemented at the moment:
- on the buildwrapper side, I use the GHC API to load all the modules for a given cabal component in a project, and extract usage information from the RenamedSource.
- the usage information is serialized as JSON to a file in the buildwrapper temporary folder
- these files will be read by EclipseFP after it's called buildwrapper generateUsage call
- the information will then be stored in a sqlite database stored in the workspace plugin folder.

So it will use sqlite as scion-browser does, but on the Eclipse side for performance reasons.

Most of the Haskell side of things is done, and I've started the Eclipse side. So far, this has allowed me the humble but useful "open a module" dialog, similar to "Open Type" in the JDT: it lists all the Haskell modules defined in the workspace and let you select one or several for opening the corresponding file in the editor. Selecting a module in the list shows in which project the file belongs. Wildcards are supported of course.
Hopefully all of this will prove useful.

Happy Haskell Hacking!

by JP Moresmau (noreply@blogger.com) at April 30, 2012 06:32 AM

April 29, 2012

Darcs

darcs weekly news #95

News and discussions

  1. Darcs 2.8 was released:
  2. Report of the seventh hacking sprint was put online:
  3. BSRK Aditya was accepted as a Summer of Code student to work on the patch index optimization:

Issues resolved in the last week (4)

issue1166 Guillaume Hoffmann
issue2065 Owen Stephens
issue2120 Adam Wolk
issue2139 Florent Becker

Patches applied in the last week (111)

See darcs wiki entry for details.

by guillaume (noreply@blogger.com) at April 29, 2012 06:35 PM

April 28, 2012

Keegan McAllister

Scheme without special forms: a metacircular adventure

A good programming language will have many libraries building on a small set of core features. Writing and distributing libraries is much easier than dealing with changes to a language implementation. Of course, the choice of core features affects the scope of things we can build as libraries. We want a very small core that still allows us to build anything.

The lambda calculus can implement any computable function, and encode arbitrary data types. Technically, it's all we need to instruct a computer. But programs also need to be written and understood by humans. We fleshy meatbags will soon get lost in a sea of unadorned lambdas. Our languages need to have more structure.

As an example, the Scheme programming language is explicitly based on the lambda calculus. But it adds syntactic special forms for definitions, variable binding, conditionals, etc. Scheme also lets the programmer define new syntactic forms as macros translating to existing syntax. Indeed, lambda and the macro system are enough to implement some of the standard special forms.

But we can do better. There's a simple abstraction which lets us define lambda, Lisp or Scheme macros, and all the other special forms as mere library code. This idea was known as "fexprs" in old Lisps, and more recently as "operatives" in John Shutt's programming language Kernel. Shutt's PhD thesis [PDF] has been a vital resource for learning about this stuff; I'm slowly making my way through its 416 pages.

What I understand so far can be summarized by something self-contained and kind of cool. Here's the agenda:

  • I'll describe a tiny programming language named Qoppa. Its S-expression syntax and basic data types are borrowed from Scheme. Qoppa has no special forms, and a small set of built-in operatives.

  • We'll write a Qoppa interpreter in Scheme.

  • We'll write a library for Qoppa which implements enough Scheme features to run the Qoppa interpreter.

  • We'll use this nested interpreter to very slowly compute the factorial of 5.

All of the code is on GitHub, if you'd like to see it in one place.

Operatives in Qoppa

An operative is a first-class value: it can be passed to and from functions, stored in data structures, and so forth. To use an operative, you apply it to some arguments, much like a function. The difference is that

  1. The operative receives its arguments as unevaluated syntax trees, and

  2. The operative also gets an argument representing the variable-binding environment at the call site.

Just as Scheme's functions are constructed by the lambda syntax, Qoppa's operatives are constructed by vau. Here's a simple example:

(define quote
(vau (x) env
x))

We bind a single argument as x, and bind the caller's environment as env. (Since we don't use env, we could replace it with _, which means to ignore the argument in that position, like Haskell's _ or Kernel's #ignore.) The body of the vau says to return the argument x, unevaluated.

So this implements Scheme's quote special form. If we evaluate the expression (quote x) we'll get the symbol x. As it happens, quote is used sparingly in Qoppa. There is usually a cleaner alternative, as we'll see.

Here's another operative:

(define list (vau xs env
(if (null? xs)
(quote ())
(cons
(eval env (car xs))
(eval env (cons list (cdr xs)))))))

This list operative does the same thing as Scheme's list function: it evaluates any number of arguments and returns them in a list. So (list (+ 2 2) 3) evaluates to the list (4 3).

In Scheme, list is just (lambda xs xs). In Qoppa it's more involved, because we must explicitly evaluate each argument. This is the hallmark of (meta)programming with operatives: we selectively evaluate using eval, rather than selectively suppressing evaluation using quote.

The last part of this code deserves closer scrutiny:

(eval env (cons list (cdr xs)))

What if the caller's environment env contains a local binding for the name list? Not to worry, because we aren't quoting the name list. We're building a cons pair whose car is the value of list... an operative! Supposing xs is (1 2 3), the expression

(cons list (cdr xs))

evaluates to the list

(<some-value-representing-an-operative> 2 3)

and that's what eval sees. Just like lambda, evaluating a vau expression captures the current environment. When the resulting operative is used, the vau body gets values from this captured static environment, not the dynamic argument of the caller. So we have lexical scoping by default, with the option of dynamic scoping thanks to that env parameter.

Compare this situation with Lisp or Scheme macros. Lisp macros build code which refers to external stuff by name. Maintaining macro hygiene requires constant attention by the programmer. Scheme's macros are hygienic by default, but the macro system is far more complex. Rather than writing ordinary functions, we have to use one of several special-purpose sublanguages. Operatives provide the safety of Scheme macros, but (like Lisp macros) they use only the core computational features of the language.

Implementing Qoppa

Now that you have a taste of what the language is like, let's write a Qoppa interpreter in Scheme.

We will represent an environment as a list of frames, where a frame is simply an association list. Within the vau body in

( (vau (x) _ x) 3 )

the current environment would be something like

( ;; local frame
((x 3))

;; global frame
((cons <operative>)
(car <operative>)
...) )

Here's a Scheme function to build a frame from some names and the corresponding values.

(define (bind param val) (cond
((and (null? param) (null? val))
'())
((eq? param '_)
'())
((symbol? param)
(list (list param val)))
((and (pair? param) (pair? val))
(append
(bind (car param) (car val))
(bind (cdr param) (cdr val))))
(else
(error "can't bind" param val))))

We allow names and values to be arbitrary trees, so for example

(bind
'((a b) . c)
'((1 2) 3 4))

evaluates to

((a 1)
(b 2)
(c (3 4)))

(If you'll recall, (x . y) is the pair formed by (cons 'x 'y), an improper list.) The generality of bind means our argument-binding syntax — in vau, lambda, let, etc. — will be richer than Scheme's.

Next, a function to find a (name value) entry, given the name and an environment. This just invokes assq on each frame until we find a match.

(define (m-lookup name env)
(if (null? env)
(error "could not find" name)
(let ((binding (assq name (car env))))
(if binding
binding
(m-lookup name (cdr env))))))

We also need a representation for operatives. A simple choice is that a Qoppa operative is represented by a Scheme procedure that takes the operands and current environment as arguments. Now we can write the Qoppa evaluator itself.

(define (m-eval env exp) (cond
((symbol? exp)
(cadr (m-lookup exp env)))
((pair? exp)
(m-operate env (m-eval env (car exp)) (cdr exp)))
(else
exp)))

(define (m-operate env operative operands)
(operative env operands))

The evaluator has only three cases. If exp is a symbol, it refers to a value in the current environment. If it's a cons pair, the car must evaluate to an operative and the cdr holds operands. Anything else evaluates to itself: numbers, strings, Booleans, and Qoppa operatives (represented by Scheme procedures).

Instead of the traditional eval and apply we have "eval" and "operate". Thanks to our uniform representation of operatives, the latter is very simple.

Qoppa builtins

Now we need to populate the global environment with useful built-in operatives. vau is the most significant of these. Here is its corresponding Scheme procedure.

(define (m-vau static-env vau-operands)
(let ((params (car vau-operands))
(env-param (cadr vau-operands))
(body (caddr vau-operands)))

(lambda (dynamic-env operands)
(m-eval
(cons
(bind
(cons env-param params)
(cons dynamic-env operands))
static-env)
body))))

When applying vau, you provide a parameter tree, a name for the caller's environment, and a body. The result of applying vau is an operative which, when applied, evaluates that body. It does so in the environment captured by vau, extended with arguments.

Here's the global environment:

(define (make-global-frame)
(define (wrap-primitive fun)
(lambda (env operands)
(apply fun (map (lambda (exp) (m-eval env exp)) operands))))
(list
(list 'vau m-vau)
(list 'eval (wrap-primitive m-eval))
(list 'operate (wrap-primitive m-operate))
(list 'lookup (wrap-primitive m-lookup))
(list 'bool (wrap-primitive (lambda (b t f) (if b t f))))
(list 'eq? (wrap-primitive eq?))
; more like these
))

(define global-env (list (make-global-frame)))

Other than vau, each built-in operative evaluates all of its arguments. That's what wrap-primitive accomplishes. We can think of these as functions, whereas vau is something more exotic.

We expose the interpreter's m-eval and m-operate, which are essential for building new features as library code. We could implement lookup as library code; providing it here just prevents some code duplication.

The other functions inherited from Scheme are:

  • Type predicates: null? symbol? pair?

  • Pairs: cons car cdr set-car! set-cdr!

  • Arithmetic: + * - / <= =

  • I/O: error display open-input-file read eof-object

Scheme as a Qoppa library

The Qoppa interpreter uses Scheme syntax like lambda, define, let, if, etc. Qoppa itself supports none of this; all we get is vau and some basic data types. But this is enough to build a Qoppa library which provides all the Scheme features we used in the interpreter. This code starts out very cryptic, and becomes easier to read as we have more high-level features available. You can read through the full library if you like. This section will go over some of the more interesting parts.

Our first task is a bit of a puzzle: how do you define define? It's only possible because we expose the interpreter's representation of environments. We can push a new binding onto the top frame of env, like so:

(set-car! env
(cons
(cons <name> (cons <value> null))
(car env)))

We use this idea twice, once inside the vau body for define, and once to define define itself.

((vau (name-of-define null) env
(set-car! env (cons
(cons name-of-define
(cons (vau (name exp) defn-env
(set-car! defn-env (cons
(cons name (cons (eval defn-env exp) null))
(car defn-env))))
null))
(car env))))
define ())

Next we'll define Scheme's if, which evaluates one branch or the other. We do this in terms of the Qoppa builtin bool, which always evaluates both branches.

(define if (vau (b t f) env
(eval env
(bool (eval env b) t f))))

We already saw the code for list, which evaluates each of its arguments. Many other operatives have this behavior, so we should abstract out the idea of "evaluate all arguments". The operative wrap takes an operative and returns a transformed version of that operative, which evaluates all of its arguments.

(define wrap (vau (operative) oper-env
(vau args args-env
(operate args-env
(eval oper-env operative)
(operate args-env list args)))))

Now we can implement lambda as an operative that builds a vau term, evals it, and then wraps the resulting operative.

(define lambda (vau (params body) static-env
(wrap
(eval static-env
(list vau params '_ body)))))

This works just like Scheme's lambda:

(define fact (lambda (n)
(if (<= n 1)
1
(* n (fact (- n 1))))))

Actually, it's incomplete, because Scheme's lambda allows an arbitrary number of expressions in the body. In other words Scheme's

(lambda (x) a b c)

is syntactic sugar for

(lambda (x) (begin a b c))

begin evaluates its arguments in order left to right, and returns the value of the last one. In Scheme it's a special form, because normal argument evaluation happens in an undefined order. By contrast, the Qoppa interpreter implements a left-to-right order, so we'll define begin as a function.

(define last (lambda (xs)
(if (null? (cdr xs))
(car xs)
(last (cdr xs)))))

(define begin (lambda xs (last xs)))

Now we can mutate the binding for lambda to support multiple expressions.

(define set! (vau (name exp) env
(set-cdr!
(lookup name env)
(list (eval env exp)))))

(set! lambda
((lambda (base-lambda)
(vau (param . body) env
(eval env (list base-lambda param (cons begin body)))))
lambda))

Note the structure

((lambda (base-lambda) ...) lambda)

which holds on to the original lambda operative, in a private frame. That's right, we're using lambda to save lambda so we can overwrite lambda. We use the same approach when defining other sugar, such as the implicit lambda in define.

There are some more bits of Scheme we need to implement: cond, let, map, append, and so forth. These are mostly straightforward; read the code if you want the full story. By far the most troublesome was Scheme's apply function, which takes a function and a list of arguments, and is supposed to apply the function to those arguments. The problem is that our functions are really operatives, and expect to call eval on each of their arguments. If we already have the values in a list, how do we pass them on?

Qoppa and Kernel have very different solutions to this problem. In Kernel, "applicatives" (things that evaluate all their arguments) are a distinct type from operatives. wrap is the primitive constructor of applicatives, and its inverse unwrap is used to implement apply. This design choice simplifies apply but complicates the core evaluator, which needs to distinguish applicatives from operatives.

For Qoppa I implemented wrap as a library function, which we saw before. But then we don't have unwrap. So apply takes the uglier approach of quoting each argument to prevent double-evaluation.

(define apply (wrap (vau (operative args) env
(eval env (cons
operative
(map (lambda (x) (list quote x)) args))))))

In either Kernel or Qoppa, you're not allowed to apply apply to something that doesn't evaluate all of its arguments.

Testing

The code we saw above is split into two files:

  • qoppa.scm is the Qoppa interpreter, written in Scheme

  • prelude.qop is the Qoppa code which defines wrap, lambda, etc.

I defined a procedure execute-file which reads a file from disk and runs each expression through m-eval. The last line of qoppa.scm is

(execute-file "prelude.qop")

so the definitions in prelude.qop are available immediately.

We start by loading qoppa.scm into a Scheme interpreter. I'm using Guile here, but I've actually tested this with a variety of R5RS implementations.

$ guile -l qoppa.scm
guile> (m-eval global-env '(fact 5))
$1 = 120

This establishes that we've implemented the features used by fact, such as define and lambda. But did we actually implement enough to run the Qoppa interpreter? To test this, we need to go deeper.

guile> (execute-file "qoppa.scm")
$2 = done
guile> (m-eval global-env '(m-eval global-env '(fact 5)))
$3 = 120

This is factorial implemented in Scheme, implemented as a library for Qoppa, implemented in Scheme, implemented as a library for Qoppa, implemented in Scheme (implemented in C). Of course it's outrageously slow; on my machine this (fact 5) takes about 5 minutes. But it demonstrates that a tiny language of operatives, augmented with an appropriate library, can provide enough syntactic features to run a non-trivial Scheme program. As for how to do this efficiently, well, I haven't got far enough into the literature to have any idea.

by keegan (noreply@blogger.com) at April 28, 2012 07:09 AM

April 27, 2012

Paolo Capriotti

Applicative option parser

There are quite a few option parsing libraries on Hackage already, but they either depend on Template Haskell, or require some boilerplate. Although I have nothing against the use of Template Haskell in general, I’ve always found its use in this case particularly unsatisfactory, and I’m convinced that a more idiomatic solution should exist.

In this post, I present a proof of concept implementation of a library that allows you to define type-safe option parsers in Applicative style.

The only extension that we actually need is GADT, since, as will be clear in a moment, our definition of Parser requires existential quantification.

> {-# LANGUAGE GADTs #-}
> import Control.Applicative

Let’s start by defining the Option type, corresponding to a concrete parser for a single option:

> data Option a = Option
>   { optName :: String
>   , optParser :: String -> Maybe a
>   }
> 
> instance Functor Option where
>   fmap f (Option name p) = Option name (fmap f . p)
> 
> optMatches :: Option a -> String -> Bool
> optMatches opt s = s == '-' : '-' : optName opt

For simplicity, we only support "long" options with exactly 1 argument. The optMatches function checks if an option matches a string given on the command line.

We can now define the main Parser type:

> data Parser a where
>   NilP :: a -> Parser a
>   ConsP :: Option (a -> b)
>         -> Parser a -> Parser b
> 
> instance Functor Parser where
>   fmap f (NilP x) = NilP (f x)
>   fmap f (ConsP opt rest) = ConsP (fmap (f.) opt) rest
> 
> instance Applicative Parser where
>   pure = NilP
>   NilP f <*> p = fmap f p
>   ConsP opt rest <*> p =
>     ConsP (fmap uncurry opt) ((,) <$> rest <*> p)

The Parser GADT resembles a heterogeneous list, with two constructors.

The NilP r constructor represents a "null" parser that doesn’t consume any arguments, and always returns r as a result.

The ConsP constructor is the combination of an Option returning a function, and an arbitrary parser returning an argument for that function. The combined parser applies the function to the argument and returns a result.

The definition of (<*>) probably needs some clarification. The variables involved have types:

<notextile> <figure class="code">
1
2
3
opt :: Option (a -> b -> c)
rest :: Parser a
p :: Parser b
</figure> </notextile>

and we want to obtain a parser of type Parser c. So we uncurry the option, obtaining:

<notextile> <figure class="code">
1
fmap uncurry opt :: Option ((a, b) -> c)
</figure> </notextile>

and compose it with a parser for the (a, b) pair, obtained by applying the (<*>) operator recursively:

<notextile> <figure class="code">
1
(,) <$> rest <*> p :: Parser (a, b)
</figure> </notextile>

This is already enough to define some example parsers. Let’s first add a couple of convenience functions to help us create basic parsers:

> option :: String -> (String -> Maybe a) -> Parser a
> option name p = ConsP (fmap const (Option name p)) (NilP ())
> optionR :: Read a => String -> Parser a
> optionR name = option name p
>   where
>     p arg = case reads arg of
>       [(r, "")] -> Just r
>       _       -> Nothing

And a record to contain the result of our parser:

> data User = User
>   { userName :: String
>   , userId :: Integer
>   } deriving Show

A parser for User is easily defined in applicative style:

> parser :: Parser User
> parser = User <$> option "name" Just <*> optionR "id"

To be able to actually use this parser, we need a "run" function:

> runParser :: Parser a -> [String] -> Maybe (a, [String])
> runParser (NilP x) args = Just (x, args)
> runParser (ConsP _ _) [] = Nothing
> runParser p (arg : args) =
>   case stepParser p arg args of
>     Nothing -> Nothing
>     Just (p', args') -> runParser p' args'
> 
> stepParser :: Parser a -> String -> [String] -> Maybe (Parser a, [String])
> stepParser p arg args = case p of
>   NilP _ -> Nothing
>   ConsP opt rest
>     | optMatches opt arg -> case args of
>         [] -> Nothing
>         (value : args') -> do
>           f <- optParser opt value
>           return (fmap f rest, args')
>     | otherwise -> do
>         (rest', args') <- stepParser rest arg args
>         return (ConsP opt rest', args')

The idea is very simple: we take the first argument, and we go over each option of the parser, check if it matches, and if it does, we replace it with a NilP parser wrapping the result, consume the option and its argument from the argument list, then call runParser recursively.

Here is an example of runParser in action:

> ex1 :: Maybe User
> ex1 = fst <$> runParser parser ["--name", "fry", "--id", "1"]
> {- Just (User {userName = "fry", userId = 1}) -}

The order of arguments doesn’t matter:

> ex2 :: Maybe User
> ex2 = fst <$> runParser parser ["--id", "2", "--name", "bender"]
> {- Just (User {userName = "bender", userId = 2}) -}

Missing arguments will result in a parse error (i.e. Nothing). We don’t support default values but they are pretty easy to add.

> ex3 :: Maybe User
> ex3 = fst <$> runParser parser ["--name", "leela"]
> {- Nothing -}

I think the above Parser type represents a pretty clean and elegant solution to the option parsing problem. To make it actually usable, I would need to add a few more features (boolean flags, default values, a help generator) and improve error handling and performance (right now parsing a single option is quadratic in the size of the Parser), but it looks like a fun project.

Does anyone think it’s worth adding yet another option parser to Hackage?

April 27, 2012 12:40 PM

Ivan Lazar Miljenovic

Announcing planar-graph

I’ve been working on a new planar graph library on and off for just over the past year.

I realise that this might not be exactly a much-sought-after library, but I’ve been using this as a test-bed for various ideas I’ve been having for a “normal” graph library. I’m going to discuss various aspects of the design of this library and some ideas I’ve had for an extensible graph library.

What is a graph?

The standard definition of a graph is as follows:

a graph is an ordered pair G = (V, E) comprising a set V of vertices or nodes together with a set E of edges or lines, which are 2-element subsets of V (i.e., an edge is related with two vertices, and the relation is represented as unordered pair of the vertices with respect to the particular edge).

However, this definition is rather limiting: by assuming that an edge is comprised of a two-element subset of the vertices, we make it harder for us to consider it computationally: in practice we don’t have a data-structure that represents a two-element set. In practice, we instead tend to use something like (Node,Node). However, this isn’t ideal:

  • Using this representation implicitly makes graphs directed rather than undirected, unless you do a lot more bookkeeping by checking both elements of the tuple. In practice this may not be too much of a problem, as people want directed graphs, but this doesn’t make it perfect.

  • The directionality of an edge is now part of its definition rather than being a property of the edge: that is, whether the edge is meant to be directed or not should be a value that can be determined from the edge; currently all edges are directed and undirected graphs are simulated with two inverse edges.

  • Multiple edges become difficult to handle: if you want to delete an edge between node n1 and node n2 and there are three edges there, how do you know which one to delete? In practice, the answer seems to be “all of them”.

A more preferable definition was stated by W. T. Tutte in a 1961 paper:

A graph G consists of a set E(G) of edges and a (disjoint) set V(G) of vertices, together with a relation of incidence which associates with each edge two vertices, not necessarily distinct, called its ends. An edge is a loop if its ends coincide and a link otherwise.

Note that no indication is given of E being a set of two-sets: instead, a mapping exists between every e ∈ E to the two endpoints of that edge.

Planar graphs

A planar graph is a graph that can be drawn on a specified surface (usually a plane or a sphere) such that no edges intersect/cross except at their endpoints.

When considering a planar graph programmatically, we also want to take into account their embedding (i.e. where all the edges adjacent to a node are in relation to each other). As such, just using an approach of identifying edges solely by their end points fails completely if there are multiple edges. As such, using a unique identifier for each edge is preferable.

But the difficulty of endpoint identification (i.e. distinguishing between (n1,n2)) remains. As such, several implementations of planar graphs use two identifiers for each edge. More about this later.

Library implementation

As I said earlier, I’ve been using the development of this library as a way to experiment with various approaches to how to design a graph library, all of which I intend to use in a non-planar graph library.

Abstract node identifiers

Most existing graph libraries (e.g. Data.Graph and fgl) use a type alias on Int values to represent vertices/nodes. Furthermore, when considering how to create a graph, it requires that you:

  1. Explicitly come up with new node identifier for each new node;

  2. Make sure you don’t re-use an existing identifier.

Whilst this isn’t a big problem if considering a bulk creation of a graph (i.e. you have some arbitrary [a] representing the nodes and edges represented as [(a,a)], in which case a zip-based solution can be used to assign node identifiers, etc. though it would be a tad messy), it isn’t ideal for adding new nodes on after the fact and it is also open to abuse.

As such, planar-graph does not permit users to create node identifiers: the constructor isn’t exported, it isn’t an instance of Enum or Bounded, etc. Instead, you provide the label for the new node you want to add and the function returns the updated graph and the identifier for the new node. When the node identifiers are changed for some reason (e.g. merging two graphs), a function is returned that allows you to update the values of any node identifiers you’ve been storing elsewhere.

Show and Read instances are available and leak some of the internals out, but you have to really be persistent to try and abuse them to create your own identifier values: you need to explicitly call read on the String to get it to parse as the result isn’t valid Haskell code (as the instances exist solely for debugging).

Half-edges

As intimated earlier, each edge is actually represented by two half-edges: an edge from n1 to n2 is “stored” twice: one half-edge n1 -> n2 and its inverse n2 -> n1 (this also includes loops). Each half-edge has its unique identifier (which is abstract, just as with nodes) and mapping function exists that lets you determine a half-edge’s inverse.

Most graph implementations are something like a newtyped version of:

type Graph = Map Node [Node]

where each node has information on its adjacent nodes. Or, if we consider a graph with labels, we have:

type LabelledGraph n e = Map Node (n, [(Node,e)])

Instead, the definition of planar-graph looks more like (just considering the labelled version):

type PlanarGraph n e = ( Map Node (n, [Edge])
                       , Map Edge (Node, e, Node, Edge)
                       )

where a mapping exists between a half-edge identifier and the corresponding node that it comes from, the label of that half-edge, the node that it is going to and its inverse half-edge. This definition matches the mathematical ones stated earlier much more closely.

Now, this half-edge usage might be a requirement for a planar graph data structure, but it is also viable for non-planar graphs. First of all, if we wished to allow multiple edges between two nodes, then the traditional representation must be altered slightly:

type LabelledGraph' n e = Map Node (n, [(Node,[e])])

Each edge representation now keeps a list of edge labels, one per edge between the two nodes. Extra bookkeeping is required about what to do when that list becomes empty (and in fact previous fgl versions had an implementation where this list wasn’t considered at all and thus multiple edges would silently fail).

Also, consider how fgl-style graphs are implemented:

type FGLGraph n e = Map Node ([(Node,e)], n, [(Node,e)])

Here, each node stores not only the outgoing edges but also the incoming edges for efficiency reasons (otherwise we’d need to traverse all edges in the graph to determine what the incoming edges are). This also leads to possible data corruption issues as each edge label is stored twice.

However, with our half-edge implementation, neither of these points need any change: each multiple edge has its own unique identifier, and to obtain the incoming edges we just determine the inverse of all the outgoing edges (though technically this point isn’t quite valid when considering directed graphs, as planar-graph treats them differently; see the next section).

Distinguishing between structure and content

Most graph implementations conflate the structure of the graph (i.e. which nodes and edges there are) with the information that graph is representing. One example is the question of graph orientation: in fgl, a graph can be considered to be undirected if each edge is represented twice; however, it is quite possible that such a graph is not undirected but just happens to have each directed edge having an inverse (e.g. some kind of flow algorithm).

Whilst it is not fully formalised as yet, in planar-graph the orientation of a graph is dictated by its half-edge labels: in my use case that prompted the development of this library, I had a need for mixed-orientation of a graph: one half-edge pairing might have had a fixed direction on one half-edge whilst its inverse had a label of “InverseEdge“; other pairings might both have some kind of partial edge label.

But the actual edge identifiers didn’t change: I could apply a mapping function to transform all edge labels to () and thus make the graph “undirected”, but I didn’t need to change the actual structure of the graph to do so.

I believe this is a much more useful way of considering graphs, where the information that the graph represents can be found in the node and edge labels, not the identifiers.

Serialisation and encoding

I needed to be able to encode my planar graphs using various binary encodings (e.g. PLANAR_CODE, described in Appendix A here). Now, I could have written custom encoding functions from a PlanarGraph to a ByteString for every possible encoding; however, since they all follow the same basic underlying structure, I decide to utilise an intermediary list-based representation.

However, I then realised that this representation could also be used for Show and Read instances for the graphs as well as pretty-printing functions. The definition of the representation is:

[( node index
 , node label
 , [( edge index
    , node index that this edge points to
    , edge label
    , inverse edge index
   )]
)]

For Show, Read, etc. this is basically a raw dump of the graph (which means it is technically open to abuse as the internals of the abstract identifiers are accessible this way, but I had to draw the line somewhere); the deserialise function that is utilised by Read also ended up being useful for an internal function rather than manually trying to construct a graph!

For encoding/decoding of binary representations, a re-numbering of the identifiers according to a breadth-first traversal is first undertaken (as many require that the identifiers be 0 ... n-1 and for decoding the order of edges for each node is important) and then the same structure is used. A class is then used to convert the graph into this representation and then convert it to the encoding of your choice.

However, not all of the encodings require that the graph be planar: whilst a breadth-first traversal doesn’t make as much sense for non-planar graphs, the same framework could be used for other graph types.

Plans for the new graph library

I haven’t even started to prototype this, as some of the ideas I’m listing below I only started to mentally flesh out over the past few days. However, I think that it should work and will provide a useful approach to dealing with graphs in Haskell.

The root of my idea is that we often have different properties that we want graphs to hold: should it be a directed graph? What should the identifier types be? Is there any way to automatically determine all nodes with a certain label (e.g. for graph colouring)? What kind should the graph have?

The current methods of dealing with such a thing is to have a graph implementation, and just live with it. This “solution” clearly has problems, not least of which is that if you try to do anything else you have to re-implement everything yourself.

However, consider this: we have a method of generalising monads via monad transformers: why not do the same thing with graphs?

Now, I’m not the first person to think of this; Edward Kmett has already released a library that has this kind of characteristic (though his classes differ from how I’m planning on structure/distinguish them by having them more implementation-based than property-based IMO).

What my plans entail is this:

  • Define a GraphTransformer class. Not only will graph transformers be instances of this class, but so will the actual graph types (with identities for the different lift/unlift functions):

    class (Graph (SubGraph gt)) => GraphTransformer gt where
        type SubGraph gt :: *
    
        .....
    
  • The Graph class then requires that the instance type also be an instance of GraphTransformer, and has default definitions of all methods using the lift/unlift functions. These default definitions will resolve down to f = f definitions for actual graph types, but for transformers will just apply the function down the stack.

    This class only defines “getters”: e.g. determine the size of the graph, get all its nodes or edges, etc.

  • Other classes are defined as sub-classes of Graph, again using lift/unlift functions from GraphTransformer for default definitions of the methods.

  • Most classes (except for where it’s necessary, e.g. a class defining mapping functions) will assume that the graph is of kind *.

  • Most transformers will assume that the underlying graph is of kind * -> * -> * (i.e. that you can specify node and edge labels) so that you can make the transformer an instance of any class that requires kind * -> * -> *, but it should be possible to make transformers that take in a graph of kind *.

  • Because of the default definitions using lift/unlift functions, most class instances for the graph transformers will be of the form instance Graph (MyTransformer ExistingGraph a b); this means that if you want to newtype your graph transformer stack, writing the instances will be trivial (albeit rather repetitive and boring).

    As such, if a transformer only effects one small aspect of graph manipulation (e.g. a transformer that keeps a Map of node labels to node IDs so you can more efficiently look up all nodes that have a particular label), then you only need to provide explicit definitions for those classes and methods (in this case, adding and deleting nodes and looking up node identifiers based upon labels rather than filtering on the entire list of nodes).

    However, this does mean that any kind of unique operation you can think of (e.g. in the example above the ability to find all identifiers for nodes that have a particular label), you will need to create a new class and make appropriate instances for underlying types (if possible) and existing transformers (so that if you put extra transformers on the stack you can still use the improved definitions for this transformer).

  • Usage of the serialisation and encoding functionality for all graphs. This will provide Show/Read instances for graphs, pretty-printing, easy Binary instances (using the serialised form) and any available encodings specified.

    The actual method of this may change from what I’ve described above, as whilst a breadth-first traversal of a planar graph is unique up to the first edge chosen, for non-planar graphs this isn’t the case. However, for encodings that don’t assume a planar graph this shouldn’t be a problem.

  • Whilst the provided underlying graph types might use abstract node identifiers, it will not be required for instances of Graph to do so (and a transformer will be provided to let you specify your own type, that under-the-hood maps to the generated abstract identifiers). However, I can’t see a way around having edges using some kind of abstract edge identifier, as it isn’t as common to attach a unique label to each edge.

Generally, transformers will utilise the node and edge labels of the underlying graph stack to store extra metadata (e.g. directionality of the edges); this is why the transformers will typically require that the underlying type is of kind * -> * -> *. However, the question then arises: should users be aware of these underlying type transformations? For example, should this information leak out with a Show instance?

My current thinking is that it shouldn’t: the output from Show, prettyPrint, etc. should be as if there were no transformers being used. The main counter-example I can think of is to have some kind of indicator whether each listed half-edge is the “real” one or not, especially when using a transformer that makes it a directed edge (though in this case it can be solved by only listing the “primary” half-edges and not their inverses; this again works for most graph types but not planar ones, as all half-edges need to be listed for the graph to be re-created due to the embedding of edge orders).

Assuming this all works (I plan on starting playing with it next week), I think this approach will do quite well. As you’re writing your code, if you use a newtype/type alias for your graph type, you can just add or remove a transformer from your stack without affecting usage (in most cases: some transformers might change which classes a stack can be used in; e.g. a transformer that lets you specify node identifiers will require using a different class for adding nodes than one that generates and returns identifiers for you). Then at the end if you want to try and tweak performance, you can always go and write custom transformers (or indeed your own graph type if you want to merge all the functionality in to the actual type without using transformers) without having to change your code.

If this all works as I think/hope it will… ;-)


Filed under: Graphs, Haskell

by Ivan Miljenovic at April 27, 2012 10:22 AM

April 24, 2012

Audrey Tang

開源之道

(這是 Allison Randal 在 OSDC.tw 的演講中譯本。請參原文錄影。)

這幾年來,我慢慢覺得,我們參與開源社群,就像是在一條道路上並肩而行:這不僅讓我們成為更好的程式設計者,也讓我們通過與人合作,而成為更好的人。

您可以將它想成一條修行之道,讓身而為人的我們能夠不斷成長。接下來,我想談談我對開源世界的個人觀點,希望能與您分享。

首先,人是一切開源專案的核心。程式碼是很重要,但最核心的永遠是人。

人們透過各種不同的方式來參與專案:有人寫程式,有人寫文件,有人寫測試。而使用軟體的人,同樣也是專案裡不可或缺的一部分。

您的專案也許會用到別人開發的軟體,而因此接觸到上游的專案,或許偶爾也會向他們提出建議和修正。

又或許您開發的是一套程式庫或模組,提供給其他專案的人使用。此時,您就是他們的上游專案,他們也會用相同的方式來與您溝通。

所以,人們到底為什麼要做開源軟體呢?如果您想理解開源模式如何運作,這是一個很關鍵的問題。

許多人在日常工作中,可能已經常常和軟體打交道了。我們為什麼要花額外的心力,來參與開源專案呢?一部分的原因,是因為這能夠讓人迅速接觸到刺激、有趣的新鮮技術。

能夠與人分享,也是一個主因:透過與人分享,我們可以認識開源專案裡的同好,來提升彼此的樂趣。

投入開源專案的人,往往也帶著分享奉獻的精神。能夠伸出雙手幫助別人,是身而為人很重要的一部份。

除了這些內在因素,參與開源專案工作,也可以得到許多回報。其中一項,是獲得別人的敬重:當我們創造新的事物與人分享,進而吸引人們一同合作時,人們自然會認識我們的人品與才能,從而為我們自己帶來成就感。

換個角度來看,這也意味著:我們應當對於加入專案的人表示尊重,這樣人們才會願意繼續參與專案的活動。

欣賞別人的作品也很重要。當人們發表自己的作品,而您有機會與他們交流時,即使是一封簡單的電子郵件感謝函,說「您的專案對我很重要」,也足以營造出一種正向的文化,讓大家都能保有繼續創造的動力。

懂得讚美也很重要。當您介紹專案時,不要忘了讚賞您身邊的人,讓大家認識這些人是誰、做了多麼棒的貢獻,以建立社群的認同感。

之所以有那麼多人持續對開源專案保持興趣,其中一個原因是這樣的:在合力工作時,我們的能力會愈來愈強,能做的事也愈來愈多。

光用簡單的算數來想:如果我們有兩倍的人,至少就可以寫兩倍多的程式,有三倍的人就可以寫出三倍的程式。不過,我們的能力遠遠不止這些。

在一起合作時,我們可以透過彼此鼓勵,讓彼此變得更好更強大。當您看到其他人正在解決艱難的問題時,您不妨鼓勵他們,跟他們說:「你做得很好,而且我看得出來,你在未來會做得更棒。」

僅僅是透過談話和分享,您就可以為他人培力,讓對方變得更好。

還有一點就是,當許多人聚在一起的時候,每個人都有不同的能力。一起工作時,可能您知道專案需要的五樣東西,而其他人知道另外五樣東西,您們互補長短,就有了一整套技能足以完成專案,而這是單打獨鬥時做不到的事情。

所以在多人合作時,不只是生產力倍增,還可以達到互相加乘的效果。

另一件很重要的事,是鼓勵彼此放眼未來、看得更遠。我們可以給其他人靈感,幫助他們解決有意思的問題。有時,只要說「我有這個想法...」,別人就可以將它化為現實。

有些時候,您只要看看別人在做些什麼,然後告訴他們您想到的關鍵之處,不必自己跳下去實作,也可以幫助他們走得更好更遠。

在做開源工作時,我們得時常提醒自己,我們並不是孤身一人。由於需要和許多人合作,我們最需要注意的,就是不斷改進自己的溝通技巧。

我們經常會彼此溝通對未來的規劃,例如軟體專案的發展藍圖,以及我們的個人計劃,像是接下來想要實作哪些功能等等。

在開源社群中,我注意到一件事情:人們對如何做軟體往往有很好的規劃,可是卻由於缺乏良好的溝通,而讓彼此的計劃互相衝突。如果您朝向某個規劃埋頭開發,而沒有與人溝通的話,很可能會傷害到其他朝向不同方向開發的人。

我們就像一窩在蜂巢裡的蜜蜂,要經常發出嗡嗡聲,才能讓彼此持續發揮功能。

此外,我們還會不時討論技術問題,嘗試找出最好的解決方案。在面對技術問題的時候,人們可能會互相爭論、甚至大動肝火,讓事情難以獲得實質的進展。

所以,我們在工作過程裡,要逐漸學會接受各種各樣的可能性。對於您自己想到的解法,您當然應該持續努力,但也不妨對別人所提出的其他可能性,抱持開放的態度。

而在您自己的工作有所進展時,也可以透過各種通訊管道,讓大家知道您做了些什麼。發電郵、寫推特… 有很多方法能讓人們知道您的進度。

有時候我們可能會覺得害羞,或是不想被別人認為自己在吹噓。但其實事情完全不是這樣!多溝通對專案有好處,對專案裡的人也是好事,因為他們可以從您所作的事情裡學到東西。

溝通的另一個重點是問問題。有社群的好處,就是可能有人已經解決過您正在面對的問題。透過論壇或聊天室主動發問,可以為您省去很多時間。

同樣的道理,當別人想要學習時,您也可以認真回應,而不是對簡單的問題拋下一句「RTFM(去看該死的說明書)」就算了。

如果您回答「RTFM」,的確可以為自己省些時間,但是您一旦這麼做,同時也是在告訴別人說,他們一開始就不應該問問題。而這絕對不是您想要的效果,您要的是培養對方溝通的意願。

學著如何去給別人有幫助的答案,幫助他們一同走上這條開源之道,日後他們才能把這條路走得更長、更遠。

有些時候,批評別人是必要的。雖然我們對各種可能性抱持開放的態度,但針對特定的技術問題,確實可能有某種解法比其他的都要正確。即使如此,當您想要讓別人改變他們的看法,最好的方式是用友善的態度提出回應,對方才會用開放的胸懷來向您學習。

即使對方態度惡劣,也請保持優雅。難免有些人會對您很不客氣,但這也是參與開源的必經之路。有時候,臉皮厚一點也有好處。雖然有些人的溝通方式有待加強,但他們說的內容或許也有可取之處,您還是可以從中學到東西。

從這個角度來看,就算人們說話的時候不禮貌,您還是可以禮貌地回應他們。

溝通的另一部分不是說話,而是傾聽。有時我們須要做的,不是告訴別人我們的想法,而是靜靜地坐好,讓別人暢所欲言。

光是聆聽是不夠的,我們還需要有同理心。英文有句俗話說:「如果您真想瞭解某人的話,請穿上他的鞋走一哩路。」 — 或許只有這樣,您才能明白別人所經過的煎熬。

有些人以為,能夠從事開源軟體工作的人,個個都得是天才。事實絕非如此。的確有 Larry、Guido、Linus 這樣的人物,但其實任何一個專案,都需要各方面具有不同才能的人加入。

重要的是,無論您有多聰明,都要保持謙虛。因為只有謙虛的人,才能以開放的態度面對其他人,學會用新方法來做事。謙遜的心態,讓您能歡迎其他人加入您的專案。相反的,抱持驕傲自大的態度,就等於是在跟其他人說:「我不需要你們,我用自己的方法做事就夠了。」

也是因為謙遜,我們才能歡迎各種性別、各種文化的人加入社群,為開源軟體帶來多元而豐富的人才。

就像各個國家有不同的語言和文化一樣,相同的多元性,也體現在各式各樣的開源專案裡。舉例來說,Linux 社群、Perl 社群、Ruby 社群和 Python 社群,都各自用獨特的方式來交流合作。

只要我們懷著一顆謙卑的心,就可以看到自己專案所屬的社群並不是唯一的途徑,也才能夠欣賞其他社群裡的合作方式。

另外,做開源專案並不只是享受樂趣而已。樂趣當然是有,但同時也有責任。當您承諾參與一個專案時,您是讓雙肩扛上了重量。這是件好事,因為責任能讓我們進步,變成更好的人。

但是人生中還有其他的事情,像是您的伴侶、父母、孩子、職業等等。對於開源專案,我們可能會承擔一段時間的責任,但到了某天,我們可能會發現,自己不能再負起那麼多的責任了。

我們要意識到這是一個循環。一開始我們加入社群,然後逐漸負起越來越多的責任。但當人生到達某個階段之後,您總會逐漸減少所負的責任。這個過程完全是自然的,而且在專案的生命週期裡一定會發生。

所以我們不妨想想:「哪天我無法再付出那麼多心力的時候,誰來繼續我的工作呢?」

為了確保其他人能繼續我們的工作,我們可以創造出某種持續前進的過程:盡力教導與分享我們所學到的一切,同時也向其他人學習更多的事物。這是一個不斷吸收與分享知識的過程。

最後,當您在為開源工作的時候,請保持快樂吧,讓您的臉上帶著笑容,讓其他人分享您的喜悅!因為正是這種樂趣給予我們力量,讓我們能創造出偉大的事物。

您現在更快樂了嗎?:-)

by audreyt at April 24, 2012 12:40 PM

April 23, 2012

David Amos

CHAs III: QSym, a combinatorial Hopf algebra on compositions


The compositions of a number n are the different ways that it can be expressed as an ordered sum of positive integers. For example, the compositions of 4 are 1+1+1+1, 1+1+2, 1+2+1, 2+1+1, 1+3, 2+2, 3+1, 4. Equivalently, we can forget about the plus signs, and just consider the ordered lists of positive integers that sum to n. Here's the Haskell code:

compositions 0 = [[]]
compositions n = [i:is | i <- [1..n], is <- compositions (n-i)]

> :m Math.Algebras.Structures Math.Combinatorics.CombinatorialHopfAlgebra
> compositions 4
[[1,1,1,1],[1,1,2],[1,2,1],[1,3],[2,1,1],[2,2],[3,1],[4]]

We will define a CHA QSym whose basis elements are (indexed by) compositions:

newtype QSymM = QSymM [Int] deriving (Eq)

instance Ord QSymM where
    compare (QSymM xs) (QSymM ys) = compare (sum xs, xs) (sum ys, ys)

instance Show QSymM where
    show (QSymM xs) = "M " ++ show xs

qsymM :: [Int] -> Vect Q QSymM
qsymM xs | all (>0) xs = return (QSymM xs)
         | otherwise = error "qsymM: not a composition"

We will use QSymM as the basis for a Hopf algebra, indexed by compositions. (In practice, we're going to stick with smallish compositions, as the calculations would take too long otherwise.) We form the free vector space over this basis. An element of the free vector space is a linear combination of compositions. For example:


> qsymM [1,2] + 2 * qsymM [3,1]
M [1,2]+2M [3,1]

The algebra structure on QSymM is similar to the algebra structure we saw last time on SSymM. Instead of shifted shuffles, we will use overlapping shuffles or quasi-shuffles. This means that when shuffling (x:xs) and (y:ys), then instead of just choosing between taking x first or taking y first, we also have a third choice to take them both and add them together.

quasiShuffles (x:xs) (y:ys) = map (x:) (quasiShuffles xs (y:ys)) ++
                              map ((x+y):) (quasiShuffles xs ys) ++
                              map (y:) (quasiShuffles (x:xs) ys)
quasiShuffles xs [] = [xs]
quasiShuffles [] ys = [ys]

For example:

> quasiShuffles [1,2] [3]
[[1,2,3],[1,5],[1,3,2],[4,2],[3,1,2]]

For our algebra structure, we say that the product of two compositions is the sum of all quasi-shuffles of the compositions:

instance (Eq k, Num k) => Algebra k QSymM where
    unit x = x *> return (QSymM [])
    mult = linear mult' where
        mult' (QSymM alpha, QSymM beta) = sumv [return (QSymM gamma) | gamma <- quasiShuffles alpha beta]

It's fairly obvious that this is associative and satisfies the algebra requirements. (And there are quickCheck tests in the package to confirm it.)

The coalgebra structure on QSymM is also similar to the coalgebra structure on SSymM. (We'll see in due course that SSym and QSym are closely related.) For the comultiplication of a composition, we'll just use the sum of the deconcatenations of the composition (without the flattening that we did with SSymM):

instance (Eq k, Num k) => Coalgebra k QSymM where
    counit = unwrap . linear counit' where counit' (QSymM alpha) = if null alpha then 1 else 0
    comult = linear comult' where
        comult' (QSymM gamma) = sumv [return (QSymM alpha, QSymM beta) | (alpha,beta) <- deconcatenations gamma]

For example:

> comult $ qsymM [1,2,3]
(M [],M [1,2,3])+(M [1],M [2,3])+(M [1,2],M [3])+(M [1,2,3],M [])

(Recall that (x,y) should be read as x⊗y.)

This comultiplication, along with those we have seen for other combinatorial Hopf algebras, is obviously coassociative - but perhaps I should spell out what that means. Coassociativity says that:
(comult⊗id) . comult = (id⊗comult) . comult

In other words, if you split a composition in two, and then split the left part in two - in all possible ways - then you get the same sum of possibilities as if you had done the same but splitting the right part in two. This is obvious, because it's just the sum of possible ways of splitting the composition in three (modulo associativity).

Then for a coalgebra we also require:
(id⊗counit) . comult = id = (counit⊗id) . comult

But this is clear. For example:
((id⊗counit) . comult) [1,2,3] =
(id⊗counit) ([]⊗[1,2,3] + [1]⊗[2,3] + [1,2]⊗[3] + [1,2,3]⊗[]) =
0 + 0 + 0 + [1,2,3]
(modulo the isomorphism C⊗k = C)

Then, as we did previously, we can quickCheck that the algebra and coalgebra structures are compatible, and hence that we have a bialgebra. (See the detailed explanation last time: mainly it comes down to the fact that deconcatenating quasi-shuffles is the same as quasi-shuffling deconcatenations.)

instance (Eq k, Num k) => Bialgebra k QSymM where {}

As before, this is a connected graded bialgebra in an obvious way (using the sum of the composition as the grading), and hence it automatically has an antipode. In this case we can give an explicit expression for the antipode. The coarsenings of a composition are the compositions which are "more coarse" than the given composition, in the sense that they can be obtained by combining two or more adjacent parts of the given composition.

coarsenings (x1:x2:xs) = map (x1:) (coarsenings (x2:xs)) ++ coarsenings ((x1+x2):xs)
coarsenings xs = [xs] -- for xs a singleton or null

For example:

> coarsenings [1,2,3]
[[1,2,3],[1,5],[3,3],[6]]

Then the antipode can be defined as follows:

instance (Eq k, Num k) => HopfAlgebra k QSymM where
    antipode = linear antipode' where
        antipode' (QSymM alpha) = (-1)^length alpha * sumv [return (QSymM beta) | beta <- coarsenings (reverse alpha)]

Why does this work? Remember that - in the case of a combinatorial Hopf algebra, where counit picks out the empty structure - antipode has to perform the disappearing trick of making everything cancel out.

That is, we require that
mult . (id⊗antipode) . comult = unit . counit
where the right hand side is zero for any non-empty composition.

Now, in order to try to figure out how it works, I decided to go through an example. However, in the cold light of day I have to admit that perhaps this is one of those times when maths is not a spectator sport. So the rest of this blog post is "optional".

Ok, so let's look at an example:

[1,2,3]

-> (comult = deconcatenations)

[]⊗[1,2,3] + [1]⊗[2,3] + [1,2]⊗[3] + [1,2,3]⊗[]

-> (id⊗antipode = reversal coarsening of right hand side, with alternating signs)

- []⊗([3,2,1] + [5,1] + [3,3] + [6])
+ [1]⊗([3,2] + [5])
- [1,2]⊗[3]
+ [1,2,3]⊗[]

-> (mult = quasi-shuffles)

- [3,2,1] - [5,1] - [3,3] - [6]
+ [1,3,2] + [4,2] + [3,1,2] + [3,3] + [3,2,1] + [1,5] + [6] + [5,1]
- [1,2,3] - [1,5] - [1,3,2] - [4,2] - [3,1,2]
+ [1,2,3]

= 0

(The way I like to think of this is:
- comult splits the atom into a superposition of fragment states
- antipode turns the right hand part of each fragment pair into a superposition of anti-matter states
- mult brings the matter and anti-matter parts together, causing them to annihilate
However, I think that is just fanciful - there's no flash of light - so I'll say no more.)

So why does it work? Well, in the final sum, the terms have to cancel out in pairs. If we look at some of the terms, it appears that there are three possibilities:

Case 1:

Consider the case of [4,2]. It arises in two ways in the final sum:
123 -> 1,23 -> 1,32 -> 42
123 -> 12,3 -> -12,3 -> -42

In both cases, the 1 and the 3 are combined during the quasi-shuffle phase to make a 4. In order to be combined in this way, they have to end up on opposite sides of the split during the deconcatenation phase. Because there is a 2 separating them, there are two ways this can happen, with the 2 ending up on either the left or right hand side of the split. And then the alternating signs ensure that the two outcomes cancel out at the end.

(In this case there was just one part separating them, but the same thing happens if there are more. For example:
1234 -> 1,234 -> -1,432 -> -532 + other terms
1234 -> 12,34 -> 12,43 -> 532 + 523
1234 -> 123,4 -> -123,4 -> -523)

Case 2:

Consider the case of [3,3]. This arises in two ways in the final sum:
123 -> [],123 -> -[],33 -> -33
123 -> 1,23 -> 1,32 -> 33

In this case it is the 1 and the 2 that combine. Unlike the 42 case, in this case the parts that combine are adjacent to one another at the beginning. In the top interaction, they combine during the reverse coarsening phase, in the bottom interaction, they combine during the quasi-shuffle phase. If x and y are adjacent, then the only way they can combine during the quasi-shuffle phase is if the split happens between them during the deconcatenation phase. This will always cancel with the term we get by splitting just before both x and y, and then combining them during coarsening.

wxyz -> w,xyz -> +/- w,z(y+x) -> {wz}(y+x)
wxyz -> wx,yz -> -/+ wx,zy -> {wz}(x+y)

Case 3:

Finally, it may happen that x and y don't combine. This can happen in two different ways, depending whether x and y are adjacentparts or not.

If they're not adjacent, then we get a failed case 1. For example, the [3,1,2] and [1,3,2] terms arise when the 1 and 3 fail to combine into a 4:
123 -> 1,23 -> 1,32 -> 132 + 312
123 -> 12,3 -> -12,3 -> -132-312

If they're adjacent, we get a failed case 2. For example, the [3,2,1] terms arise when the 1 and 2 fail to combine into a 3:
123 -> [],123 -> -[],321 -> -321
123 -> 1,23 -> 1,32 -> 321


This isn't quite a proof, of course. In a longer composition, there might be an opportunity for more than one of these cases to happen at the same time, and we need to show how that all works out. Also, I'm not sure that the [1,2,3] term is either a failed case 1 or a failed case 2. I hope at least though that this analysis has shed some light on why it works. (Exercise: Complete the proof.)

by DavidA (noreply@blogger.com) at April 23, 2012 09:01 PM

Ketil Malde

Constructing An Assembly Evaluation Pipeline

Warning: Although this software works well for me, it is in current development, and hasn’t seen a lot of testing. Thus, I’d be surprised if it installs smoothly and works without a hitch. If you try anyway, please let me know how it goes.

Part of my day job has recently been working on a de novo sequencing project. As we are are exploring many methods for genome assembly, it is imporant to be able to evaluate the resulting assembly in order to eliminate poor methods, improve on the more promising ones, and of course in the end select the best assembly. I have implemented a pipeline that performs this evaluation, and this is an attempt to briefly describe its goals and implementation.

Measures and input data

In order to assess quality, we rely on both internal and external measures. For instance, we can calculate the N50 contig size from the genome assembly directly, making it an internal measure. Alternatively, we can count the number of ESTs we are able to align to the assembly, making it an external measure.

For external measures, we will make use of many different types of data:

  • High-throughput DNA-sequencing reads
  • Paired-end and mate-pair reads
  • EST reads
  • Known DNA fragments, including the mitochondrion genome

Defining qualities

The pipeline addresses four aspects of assembly quality that ideally are independent of each other:

  1. Completeness: does the genome assembly contain every bit of the genome, or is a part of it missing?

  2. Fragmentation: does the genome come as a few, long, sequences, or as a zillion tiny fragments? And if the former, are the sequences correctly put together?

  3. Accuracy: does the assembled genome sequence contain small-scale errors, from e.g. base calling errors?

  4. Redundancy: is there a one-to-one correspondence between locations in the assembly and the actual genome, or does the assembly contain multiple copies of parts of the genome?

Note that it isn’t necessary that there is a fixed, optimal value for each measure that indicates a perfect assembly, it is only important that we can compare the scores for different assembly in order to determine which assembly is superior for that particular measure.

Completeness

Genomic (DNA) reads are aligned to the genome, and the proportion having matches is reported. We use BWA to align Illumina and 454 reads, and rely on ‘samtools flagstats’ to report the matches.

Also, various other sequences are mapped using BLAT, below we have a set of transcripts assembled from ESTs and RNASeq data, the mitochondrion genome, and fosmid ends. Here we report the average size of each match (as a percentage of the lenght of the query sequence), as well as the total number of sequences matching.

COMPLETENESS:
  Illumina mapped:      77.0411%
  PSL hits:
    all_transcripts_165.rna.psl.uniq:   92.0361%, 28398 matching sequences
    l.salmonis-mtDNA.dna.psl.uniq:      90.1952%, 3 matching sequences
    fosmids_fwd.dna.psl.uniq:   66.002%, 9411 matching sequences
    fosmids_rev.dna.psl.uniq:   63.7143%, 5886 matching sequences

Fragmentation

Fragmentation is measured by the standard N50 contig size, as well as N25 and N75. In order for it to be comparable, we use a fixed estimate for the genome size, not the size of the assembly itself.

Also, we count the proportion of Illumina reads that have the ‘proper pair’ flag set by BWA. Note that in order to make fragmentation independent of completeness, this is measured as a fraction of the matching reads, not all reads.

FRAGMENTATION:
  N25/50/75:    76308   46116   24010   using 600000000 as total size.
  Illumina pairs proportion:    67.7091%

Accuracy

We measure accuracy by looking at individual scores of alignments, in this case, genomic reads are aligned with BWA, and the alignments score is averaged.

ACCURACY:
  SFF average match:    148.565
  Illum average match:  46.7718

Redundancy

Redundancy looks at the total size of the assembly, comparing it to the estimated size. Then, we count the number of alignments of 454 reads and compare it to the number of query sequences.

Finally, we make use of the BLAT alignments, summing up the alignment scores as a proportion of the best scores for each sequence.

REDUNDANCY:
  Total size: 675758373, 112.626% of estimated size.
  SFF reads mapped:     117.235%
  PSL score, total vs unique:
    all_transcripts_165.rna.psl:            233.564%    67439222/28873929
    l.salmonis-mtDNA.dna.psl:       179.406%    76115/42426
    fosmids_fwd.dna.psl:            1046.08%    67459110/6448777
    fosmids_rev.dna.psl:            963.377%    37459791/3888385

Implementation

The pipeline is built as a makefile, which uses a mix of shell, AWK, and the occasional Haskell executable to do most of the heavy lifting.

Make is the widely used build system traditionally used in the Unix world. Essentially, the build process is described using a declarative language, and the make tool works out the process needed to constuct what’s desired.

Some features that makes it especially suitable is that:

  • it avoids redoing work when it’s not necessary
  • it automatically parallelizes the build process
  • it is ubiquitously available

Make

The basic syntax of make is pretty simple (but never fear, it quickly gets complicated enough!). Here’s an example rule:

foo: bar
	process bar > foo

This tells make that, in order to build the file foo, it first needs the file bar, and when that is in order – either because it exists already, or because make can use some other rule to construct it – foo will be built by running the command process bar.

Make supports pattern-based rules, the archetypal example is probably something like:

%.o: %.c
	$(CC) $<

Which means that we can build any file ending in .o from a prerequisite having the same name, except ending with .c, by running the command line from the variable CC (presumably the C compiler) on the prerequisite (from the built-in variable intuitively named $<).

While this makes it easy to set up rules for building stuff from other stuff, the caveat here is that it’s all pattern based, typically using a wildcard (%) and a suffix. This means that, for instance, paired sequence data in FASTQ must be named something.1.txt and something.2.txt.1

In addition, make has a set of built-in string manipulation functions, they typically look somewhat like:

STATS    := $(patsubst %.bam,%.stats,$(BAMS))

This populates the variable \$(STATS) with the contents of the variable \$(BAMS), except that it substitutes .stats for .bam for each element.

Shell

The commands in make files are executed by the shell, so it’s not too difficult to do things like loops. For instance, you can do:

all-fstats: $(FSTATS)
	for a in $(FSTATS); do echo "$a	`grep % $a | tr '\n' '\t'`"; done > $@

I.e., to build the file all-fstats, we first make sure we have everything in the \$(FSTATS) variable, then we loop over them, doing some grep and tr stuff, and directing output into \$@, yet another intuitively named variable, this time referring to the target of the current rule, or in this case all-fstats. Note that to refer to shell variables, they must be prefixed with an extra \$, a single dollar refers to makefile variables.

AWK

AWK is useful to collect data and perform arithmetic. For instance, we construct a table summarizing the interesting stats from all-fstats like so:

all-fstats.tab: all-fstats
	sed -e 's/\t[^\t]*(\([0-9][0-9]*\.[0-9]*%\))/\t\1/g' < $< | \
		awk ' BEGIN { OFS="\t" } { print; mapped += $2; paired += $3; single += $4 } \
		END { print "totals:    ",mapped/NR "%",paired/NR "%",single/NR "%" }' > $@

AWK is used here to accumulate sums of the various variables, and to calculate and print the averages in the end.

Haskell

There are also some tools written in Haskell that performs various processing. This includes psluniq, which extracts the best hit from PSL files, atcounts, which calculates GC/AT-ratio for each contig.

The whole thing is distributed as a Cabal library, so the required Haskell tools will be compiled or pulled in as dependencies on installation. Note that you’ll probably have to add $HOME/.cabal/bin to your $PATH.

Some thoughts

Make is Unix’s answer to democracy: it’s clearly the worst build tool out there, except for the others. It is often criticised for not being a full programming language, but I think this is a Good Thing: a more restrictive language is easier to get right. In fact, most of the problems of make stems from it having all kinds of extra functionality bolted on top.

One difficulty is the need to juggle between different paradigms, for instance, the various string processing functions in make go a long way in spite of all their messy syntax. But you still can’t escape some munging with AWK or Perl or similar tools. Wouldn’t it be nice if this were unified? Similarly, you have different layers with different sets of variables. Perhaps the ideal tool is make rules with Perl implementations?

An interesting alternative is Neil Mitchell’s shake, which does all of this in Haskell. In addition, it traces dependencies indirectly, so no more problems with forgotten dependencies. If or when I get around to trying out the alternatives, that’s clearly at the top of my list.

In practice

Getting it

The pipeline is available as a darcs repository, you should be able to get it simply by doing

darcs get http://malde.org/~ketil/asmeval

Of course, you’ll also need a bunch of tools, including the GHC compiler, the biopsl library, aligners BWA and BLAT, samtools, etc. Hopefully, you already have standard Unix tools like sed and awk.

On debian-derived Linux distributions, you should be able to get at least some of the way doing:

sudo apt-get install cabal-install darcs bwa
cabal update
darcs get http://malde.org/~ketil/asmeval
cd asmeval
cabal install

It looks like you need to install BLAT separately, and I may be missing some stuff - do let me know and I’ll update this post.

Configuring it

The configuration essentially consists of setting a bunch of make variables. This can be done on the command line, e.g.:

asmeval GENOME=mygenome.fasta ESTS=transcripts.fasta

However, asmeval will also look for a file called CONFIG in its current directory, which also can set these variables. It needs to be in makefile format, and is just included by the main makefile. The above translates to a CONFIG file containing:

GENOME := mygenome.fasta
ESTS   := transcripts.fasta

Note that this lets you use make’s string processing facilities in all their byzantine glory. There is an example (CONFIG.example) provided in the repository that can be used as a template.

Running it

After configuration, it should be possible to run everything by executing asmeval, which invokes make with some default options. These include -r to avoid the built-in rules (which only slow things down), and turning on warnings for uninitialized variables, in case you forgot something in the CONFIG file. You can add further options, for instance you may feel inclined to add -j to get parallel processing.

In order to run in the background and collect various information, I often type:

nice nohup time asmeval -j8 &

Note that j allows make to spawn multiple processes in parallel, but does nothing to take advantage of multi-threaded executables like BWA. There is a configuration variable, THREADS, you can set for this, so if an alternative to the above could be

nice nohup time asmeval THREADS=4 -j2 &

This will speed up some of the sub-processes, but run fewer of them in parallel.

The report

The default summary is output in a file called report, which contains the main statistics collected about the assembly. Hopefully it is self-explanatory enough to be useful.


  1. If they aren’t, one solution is to add symbolic links to fix this.

April 23, 2012 09:00 PM

Yesod Web Framework

More Client Side Yesod: todo sample

Following up on our previous post outlining a possible client-side approach for Yesod, I wanted to follow up with a slightly more concrete example. It seems the rage today is to provide a todo list example, so here's mine. All of the code is available from the yesod-js Github repo.

This is all still very much experimental, and no one has made any decisions yet regarding whether we'll be pursuing this approach (or any others). The discussion is still completely open, I'm just adding some more fuel to the fire.


Let's put together a simple example app that will keep a server-side list of todo items, and allow the client to update them. First, our language extensions and imports.

{-# LANGUAGE OverloadedStrings, QuasiQuotes, TemplateHaskell, KindSignatures,
             TypeFamilies, FlexibleContexts, GADTs, MultiParamTypeClasses,
             FlexibleInstances, TypeSynonymInstances
  #-}
import Yesod.Core
import Yesod.Persist
import Data.Text (Text)
import Database.Persist.Sqlite
import Network.Wai.Handler.Warp (run)
import Yesod.Json
import Yesod.Form.Jquery (YesodJquery)
import Network.HTTP.Types (status201, status204)
import Yesod.Javascript
import Data.Aeson hiding (object)
import Control.Applicative ((<$>), (<*>))
import Text.Lucius (lucius)

We'll store our data in Persistent, and set up some serialization with JSON. This is all pretty boilerplate.

share [mkPersist sqlSettings, mkMigrate "migrateAll"] [persistLowerCase|
Todo
    text Text
    done Bool
|]

instance ToJSON (Entity Todo) where
    toJSON (Entity tid (Todo text done)) = object
        [ "id" .= tid
        , "text" .= text
        , "done" .= done
        ]
instance FromJSON Todo where
    parseJSON (Object o) = Todo
        <$> o .: "text"
        <*> o .: "done"
    parseJSON _ = fail "Invalid todo"

Now our app. We'll have a pretty standard REST interface, allowing GET, PUT and DELETE.

data App = App ConnectionPool

mkYesod "App" [parseRoutes|
/ HomeR GET
/todo TodosR GET PUT
/todo/#TodoId TodoR GET DELETE
|]

And a bit more boilerplate.

instance Yesod App
instance YesodPersist App where
    type YesodPersistBackend App = SqlPersist
    runDB f = do
        App pool <- getYesod
        runSqlPool f pool
instance YesodJquery App

Let's implement our RESTful interface. This is all standard Yesod stuff, nothing client-side specific. However, we're not providing any HTML representations for now. That could be easily changed so we could have a more robot-friendly site, but that's not our purpose here.

getTodosR :: Handler RepJson
getTodosR =
    runDB (selectList [] []) >>= jsonToRepJson . asTodoEntities
  where
    asTodoEntities :: [Entity Todo] -> [Entity Todo]
    asTodoEntities = id

putTodosR :: Handler ()
putTodosR = do
    todo <- parseJsonBody_
    tid <- runDB $ insert todo
    sendResponseCreated $ TodoR tid

getTodoR :: TodoId -> Handler RepJson
getTodoR tid = runDB (get404 tid) >>= jsonToRepJson . Entity tid

deleteTodoR :: TodoId -> Handler ()
deleteTodoR tid = do
    runDB (delete tid)
    sendResponseStatus status204 ()

Now we'll start our client side stuff. All client-side values get wrapped in JSValue, with a phantom type to indicate the actual datatype. Let's generate some getters to access the fields of a todo item.

jsTodoText :: JSValue (Entity Todo) -> JSValue Text
jsTodoText = jsGetter "text"

jsTodoDone :: JSValue (Entity Todo) -> JSValue Bool
jsTodoDone = jsGetter "done"

We'll also set up a constructor to generate a new todo item. We have to use jsCast to erase the phantom types of the individual values.

jsTodo :: JSValue Text -> JSValue Bool -> JSValue Todo
jsTodo text done = jsonObject
  [ ("text", jsCast text)
  , ("done", jsCast done)
  ]

In theory, this kind of code could automatically be generated as part of Persistent (along with the ToJSON/FromJSON instance generation, which Persistent can already handle). One thing to note is that the underlying jsGetter function is not typesafe. By giving explicit signatures here, we're adding back type safety.

Next, we'll implement our homepage. Our interface doesn't (yet) allow you to mark items as done, but we provide the CSS class anyway.

getHomeR :: Handler RepHtml
getHomeR = defaultLayout $ do
    toWidget [lucius|
.done {
    color: #999;
}
|]
    runJS $ do

Let's pull in the todo items via Ajax. ajaxJson takes a type-safe route and returns two values: the data pulled from the server, and the name of the function that can be used to reload the data. We'll use that later, when we add new items.

        (todos, reload) <- ajaxJson TodosR

Now we'll format our todo items into HTML. There are a number of functions here worth mentioning:

  • htmlOutput will take a JSValue Html and produce a Widget.
  • wrapTag will wrap a JSValue Html with an extra tag. wrapTagClass does the same, but will also apply a class-attribute.
  • jsjoin will concatenate a `JSValue [Html] into a JSValue Html. This is performed client side with the Javascript join` function.
  • jsfor maps over a list of items. Under the surface, it uses Underscore.js's map function.
  • jsif is essentially Javascript's ternary operator ?:.
  • jsToHtml turns text into HTML. Notice that we're getting the same XSS-protection that Yesod is known for, even at the client side.

I'm not sure how I feel about this block, but it certainly works. The end result is a simple unordered list containing all of the TODO items.

        list <- htmlOutput $ wrapTag "ul" $ jsjoin $ jsfor todos $ \todo ->
            wrapTagClass "li"
                (jsif (jsTodoDone todo) "done" "notdone")
                $ jsToHtml $ jsTodoText todo

We want to display how many items we have total, so we use jslength and jsShowInt to get a textual representation of the item count, and then get a widget with textOutput.

        countWidget <- textOutput $ jsShowInt $ jslength todos

Now we start on the interface for adding new tasks. We'll use textInput to get an input widget, and the JSValue being input.

        (taskWidget, taskValue) <- textInput

Now we'll set up a Javascript function body which will use Ajax to submit the new todo item. Notice that we don't have to explicitly pull the value from the input field; using taskValue automatically gets the latest value for us.

Our last parameter is the reload function we got earlier. This says that, each time we submit the form, we want the data to be reloaded from the server. We could theoretically do a local update here instead, but for future features it will be important to have the server-generated ID available.

        putter <- putJson TodosR (jsTodo taskValue jsFalse) reload

And now we create a submit button, which will call putter whenever it's clicked.

        submitWidget <- button [whamlet|Add Item|] putter

Finally, we just piece it all together:

        lift [whamlet|
<h2>Item count: ^{countWidget}
^{list}
<form>
    Enter new task: #
    ^{taskWidget}
    ^{submitWidget}
|]

And more boilerplate...

main :: IO ()
main = do
    pool <- createSqlitePool "todo.db3" 5
    runSqlPool (runMigration migrateAll) pool
    toWaiApp (App pool) >>= run 3000

April 23, 2012 04:01 PM

Philip Wadler

Commercial Uses of Functional Programming 2011


Kudos to Anil Madhavapeddy (U. Cambridge), Yaron Minsky (Jane Street Capital), and Marius Eriksen (Twitter) for producing an excellent summary of CUFP 2011 and publishing it in JFP (22(1):1--8).  CUFP is valuable, and this readable and compact summary increases its value.  I hope it becomes a JFP tradition.

by Philip Wadler (noreply@blogger.com) at April 23, 2012 10:14 AM

April 21, 2012

Snap Framework

Announcing: Snap Framework v0.8.1

The Snap team would like to announce the release of version 0.8.1 of the Snap Framework.

New features

  • Snap.Http.Server.Config now has a new extendedCommandLineConfig function, which should make it easier to add your own command-line flags while still re-using most of the Snap command-line processing logic.

Bugfixes / minor improvements

  • Added a MIME type for favicons to Snap.Util.FileServe.

  • Switched the internals of the tries used in our routing tables from Data.Map to a hashmap.

  • Minor fixes to the snap tutorial projects.

Dependencies

  • All Snap dependencies have been bumped to work with the latest Hackage packages.

by Gregory Collins (greg@gregorycollins.net) at April 21, 2012 03:27 PM

Manuel M T Chakravarty

Using the Glasgow Haskell Compiler (GHC) on OS X Lion with Xcode 4.3

Here is what you need to do if you want to use the Glasgow Haskell Compiler (GHC) on OS X Lion with a clean installation of Xcode 4.3.

Command line tools

You need to install the command line tools from Apple. You may do that in two ways (the second is faster as it is a much smaller download):

  1. Install all of Xcode:
    • Install Xcode from the Mac App Store.
    • Launch Xcode.
    • In the Preferences dialog of Xcode, select the “Downloads” pane and install “Command line tools”.
  2. Install the command line tools only:

In both cases, you need to register as an Apple developer. (This is a free registration.)

Install GHC 7.4.1 (or later)

Older versions of GHC —including GHC 7.0.4, which is part of the Haskell Platform 2011.4.0.0 (December 2011)— won’t work. They complain about not being able to execute ‘/Developer/usr/bin/gcc’.

Download and install GHC 7.4.1 (or a later version) — or install the Haskell Platform once a 2012 release has been made.

Using GHC’s LLVM backend (optional)

If you like to use GHC’s LLVM backend —which is more efficient for array-based and other loop-oriented code— you need to seperately install LLVM. (This is despite the Apple tools being based on LLVM as well.)

  • Install Homebrew (as per the instructions on that webpage).
  • Then, execute the following: ‘brew install llvm’

Compiling GHC from the sources in the Git repositories (GHC developers only)

If you plan to work on GHC itself and you need to compile the develeopment version of GHC straight from the Git repositories, you need to install the GNU auto tools as well (which are no longer distributed by Apple).

  • Install Homebrew (as per the instructions on that webpage) — if you didn’t do that already to install LLVM above.
  • Then, execute the following: ‘brew install autoconf automake’

The GHC Trac has more information on Building and Porting GHC.

Posted via email from Just Testing | Comment »

April 21, 2012 12:25 PM

April 19, 2012

Chris Smith

Juggling in Haskell and Gloss

Just sharing what we’ve been talking about in my classes at LSV (Little School on Vermijo) the past few days.  It all started when a math teacher I follow on Google+ posted a link to this video on the mathematics of juggling, by Cornell professor Allen Knutson.

So we all watched the video, and talked about it… and then about midnight last night, I realized this would make a perfect introduction to Gloss simulations, which we’re learning about in our Haskell programming class!  Fast forward about 20 minutes of hacking, and we have this short program:

import Graphics.Gloss
thePattern = [5,2,5,1,2]
initial g = (-1, 0.0, [], cycle thePattern)
step dt (hand, time, balls, pattern) = (newhand, newtime, newballs, newpattern)
    where (throw, newtime) = properFraction (time + dt)
          newhand    = if throw == 1 then -hand else hand
          thrown     = if throw == 1 then [ (hand, newtime, head pattern) ]
                                     else []
          newpattern = if throw == 1 then tail pattern else pattern
          newballs   = [ (bhand, btime + dt, height)
                       | (bhand, btime, height) <- balls,
                         btime + dt < fromIntegral height ]
                       ++ thrown
draw (hand, time, balls, pattern) = pictures [
    juggler,
    pictures [ ball b | b <- balls ]
    ]
ball (bhand, btime, height) = translate (50*x) (50*y) (circleSolid 10)
    where t = 1 - 2 * (btime / fromIntegral height)
          x = if even height then bhand else bhand * t
          y = if height < 3 then 0 else fromIntegral (height - 1) * (1 - t^2)
juggler = pictures [
    line [(-50, 0), (0, 25), (50, 0)],
    line [(-30, -100), (0, -50), (30, -100)],
    line [( 0, 25), (0, -50)],
    translate 0 50 (circle 25)
    ]

Feel free to check it out.

This isn’t the best code in the world, because I deliberately write it to avoid as many ideas as I can that we haven’t talked about in my Haskell class.  Among those, for example, are any kind of user-defined data types, or any structured data other than tuples and lists!  I’ve also used list comprehensions when maps and filters might be clearer, for the same reason: we’ve done list comprehensions in class!  Those are coming soon, but for the time being, it’s not too hard to see what’s going on here.  A couple notes about the world type:

  • The state of world comprises four pieces of information: which hand will throw next, how much time has passed since the last throw, what balls are in the air, and what pattern is coming up next.  Since we don’t have user-defined types and it makes the math easy, I’ve described hands as -1 for left, and +1 for right.
  • The balls in the air form a list, but each element of that list is another tuple, containing the hand the ball was thrown from, how long it has been in the air, and its height.

So like all Gloss simulations, we need to specify three things:

  1. The initial state.  This is the left hand, no time since the last throw, no balls in the air, and the pattern sitting ahead of us.  The cycle function turns a finite list into an infinite repeating one.
  2. The step function.  This is where most of the logic sits… there are a lot of bits, but it does what you’d expect!  Every second, a ball is thrown, and the hand switches sides.  We update the balls to add to their time in the air, and keep only the ones that haven’t landed.  When a ball is thrown, we add it.  And each time we drop a number from the pattern.
  3. The draw function.  This draws the juggler and all of the balls.  The balls are drawn with some faux-physicsy thing that, frankly, is just the result of fiddling and isn’t an accurate physics simulation.  I can get away with that when teaching middle schoolers!  Hey, at least they follow parabolas; just not all with the same acceleration.  In class I just skimmed over the math for where the ball appears, since the class right now is about the state type, and drawing is something they’ve been doing all year anyway.

Definitely a lot of fun.  I encourage you to copy and paste the code above into http://dac4.designacourse.com:8000/sim and fiddle with the patterns and see what they look like!


by cdsmith at April 19, 2012 08:49 PM

Why Do Monads Matter?

(A Side Note: I’ve been formulating the final thoughts on this post for about a week now. In an entirely unrelated coincidence, a good friend of mine and fellow Haskell programmer, Doug Beardsley, ended up writing two posts about monads over the weekend as well. Weird! But don’t fret; this isn’t really the same thing at all. I’m not writing to teach Haskell programmers how to use monads. I’m writing about a kind of intuition about why these concepts turn out to matter in the first place. You won’t find much here by way of how to program in Haskell.)

Category Theory for Software Development?

Match made in heaven? Or abstraction distraction?

If you’re a software developer, have you heard about monads and wondered what they were? Have you tried to learn Haskell, and struggled with them? Wondered why people worry so much about them? Have you watched the videos from Microsoft’s “Channel 9″ and heard a bunch of researchy Microsoft folk talk about them, but had trouble relating them to your day-to-day programming experience?

Or if you’re interested in mathematics, have you heard murmurs in the past about how category theory interests computer science people? Looked for some clear statement of why we care, and what problems we might be interested in? Wondered if it’s really true at all? Perhaps you are like a friend of mine (and a first-rate algebraist, too, so it’s entirely reasonable to have these questions) who asked me about this a year or so ago, remembered hearing a lot of excitement in the early 90s about category theory and computer science, but never heard whether it had really panned out or was a dead end?

These are the kinds of questions I begin with. My goal is to demonstrate for you, with details and examples:

  • Where category-based intuition and ideas, and monads in particular, come from in computer programming.
  • Why the future of programming does lie in these ideas, and their omission in today’s mainstream languages has cost us dearly.
  • What the state of the art looks like in applying category-based ideas to problems in computer programming.

If you’re coming into this without a knowledge of category theory, never fear; this may be one of the gentlest introductions to the idea of categories and monads that you will find. But you’ll want to slow down and take a moment to understand the definition of a category and related ideas like function composition; these are absolutely crucial. Then you want to completely skip or just skim through the section called “What’s This Got To Do With Monads?” where I tell you how what we’re talking about here relates to the traditional math meaning of monads. Don’t worry, you don’t need to know that at all.

On the other hand, if you’re a mathematician, you may want to skim the bits where I review basic category theory, and just dig in where I am talking about the computer programming perspective. Just be forewarned, my introduction to monads will be via Kleisli categories, so take a minute when we get to that part and make sure you’re familiar with how the relationship works out.

Ready? Here goes!

Computer Programming and Functions: A Tenuous Relationship

Quick quiz: Do computer programmers use functions?

Ask any computer programmer you know, and you will hear: YES! Functions are some of the most basic tools computer programmers use. Then you’ll get odd looks, for asking such a silly question. Of course computer programmers use functions. That’s like asking if carpenters use nails! Right?

The truth, though, is a bit more complicated. To a mathematician, a function is just an association of input values to output values… and that is all! Any two functions that associate the same input values to the same output values are the same. Yes, you can represent functions by formulas (sometimes, anyway), but you can also represent them with just tables of inputs and outputs, or if they are functions between real numbers, as graphs. If you ask computer programmers for examples of functions, though, you will start hearing about some pretty bizarre things. I call these the “I must have skipped that day of calculus” functions. These are things that computer programmers are quite happy referring to as functions, but that to a mathematician are not really functions at all!

  • “Functions” that return randomly chosen numbers… and if evaluated several times, will give a different answer each time.
  • “Functions” that return one answer on Sundays, but a different answer on Mondays, yet another on Tuesdays, and so on.
  • “Functions” that cause words to appear on some nearby computer screen every time you calculate their values.

What’s going on here? Most computer programmers go about their lives happily calling these things functions, but really they are something different. But wait a second! They do have quite a lot in common with functions. Namely, they have: (a) parameters, representing their domain; and (b) return values, representing their range. (Many computer programmers are happy to talk about functions that have no parameters, or no return values… but there’s no need to be overly picky here. We can just regard their domains and ranges as one-element sets, so that no actual information is conveyed, but we can keep up appearances.)

Even more importantly, these “functions” share one more thing with the functions of mathematicians: they are constantly being composed by taking the result from one function and passing it along as a parameter to another function. When I say composed, I mean it almost exactly in the basic mathematics sense of function composition: (f · g)(x) = f(g(x)). In fact, the whole reason our “functions” exist at all is to be composed with each other! Once upon a time, in the early days of computers, we liked to keep track of information by just sticking it in known places in the computer’s memory; but all this shared knowledge about where to find information made it hard to write parts separately and fit them together, so we mostly switched to this idea of functions and composition instead.

Here’s the executive summary so far:

  1. When computer programmers talk about functions, they do not mean exactly what mathematicians do.
  2. What they do mean is the idea of having inputs (domains), outputs (ranges), and most importantly composition.

Along Came The Category…

So in the previous section, we ended up with our hands full of things that sort of look like functions. They have domains and ranges, and they can be composed. But at the same time, they are not functions in the mathematics sense. Baffling? No, not really. Mathematicians deal with stuff like that a lot. They have a name for systems of function-esque things of exactly that form. That name is… cue the drumroll, please… CATEGORIES!

In math-speak, categories are:

  1. collections of “objects” (you should think of sets),
  2. and “arrows” (you should think of functions between sets),
  3. where each arrow has a domain and a range,
  4. each object has an “identity” arrow (think of the identity function, where f(x) = x)
  5. and arrows can be composed when the domains and ranges match up right.

Before we agree to call something a category, we also throw in a few rules, such as if you compose any function with an identity, it doesn’t actually change, and composing functions obeys the associative property. These should be unsurprising, so if they seem strange to you, please take a moment, grab a pencil, and try working it out using the definition of function composition earlier: (f · g)(x) = f(g(x)), and simplifying.

The nice thing about categories is this: it’s not just some pointless abstraction that a bunch of mathematicians made up. Categories are defined that way because people have looked at literally hundreds of things that all look sort of like functions with domains and ranges and compositions. Things from algebra, like groups and rings and vector spaces; things from analysis, like metric spaces and topological spaces; things from combinatorics, like elements of partially ordered sets and paths in graphs; things from formal logic and foundations, like proofs and propositions. Almost without fail, they can be described using all the ideas we just looked at! In short, categories are the right intuition for talking about composing things with domains and ranges, which is exactly the situation we’re in.

The Four Horsemen of the Catapocalypse

Now you can see why categories come into the picture: they are the right intuition for things that maybe aren’t functions, but can be composed like functions. But just because a category exists doesn’t mean it’s worth talking about. What makes this worth talking about is that the category-related ideas aren’t just there, but actually express common concerns for computer programmers.

It’s now time to get a little more specific, and introduce the four examples that will guide us the rest of the way through this exploration. Each example highlights one way that the “functions” used by computer programmers might be different from the functions that mathematicians talk about. These examples represent actual kinds of problems that computer programmers have run into and solved, and we’ll look more at the practical side of them later. For now, we’ll just be happy getting familiar with the general ideas.

The First Horseman: Failure

The first problem is failure. Computer programmers do lots of things that might fail. Reading from files (they might not exist, or on a computer with more than one user, they might not be set to allow you to read them), talking over the internet (the network might be broken or too slow), even just doing plain old calculations with a large amount of data (you might run out of memory). Because of this, dealing with failure is a constant concern.

In general, in modern computer programming tools, it’s always understood that a function might fail. You may get an answer, but you also may get a reason that the task could not be completed. When that happens, programmers are responsible for dealing with it responsibly: letting someone know, cleaning up the leftover mess in computer memory from a half-complete task, and just otherwise putting the pieces back together. A major factor in programming techniques or tools is how easy they make it for programmers to cope with the constant possibility of failure.

The Second Horseman: Dependence

The second problem is dependence on outside information. While functions of mathematics are nice and self-contained, computer programmers often don’t have that luxury. Computer programs are messes of configuration. Even simple mobile phones have pages and pages of settings. What language does the user speak? How often should you save off a copy of their work? Should you encrypt communication over the network? Rare is the application today that doesn’t have a “Settings” or “Preferences” menu item. In many other contexts, too, computer programs depend on information that is a sort of “common knowledge” throughout the application, or some part of the application.

Ways of dealing with this have progressed through the ages. When everything was stored in well-known memory locations anyway, it was easy enough to just look there for information you need; but that led to problems when different parts of a program needed different information and sections of programs could step on each other’s toes. The massively influential technique known as object-oriented programming can be seen as partly an attempt to solve exactly this problem by grouping functions into a context with information that they depend on. The simplest and most flexible answer would be to just pass the information around to all the functions where it is needed… but when that’s a lot of places, passing around all those parameters can be very, very inconvenient.

The Third Horseman: Uncertainty

The third problem is uncertainty, also known as non-determinism. A normal function associates an input to an output. A non-deterministic function associates an input to some number of possible outputs. Non-determinism is less well-known than the first two problems, but possibly only because it hasn’t yet seen a convincing solution in a general purpose language! Consider:

  • Theoretical computer science talks about non-determinism all the time, because it’s the right approach for discussing a lot of computational problems, ranging from parsing to search to verification.  That language just hasn’t made its way into the programming practice.
  • Non-determinism comes up when querying, searching, or considering many possible answers. These are precisely the places that programmers end up relying on a variety of domain specific languages, ranging from SQL to Prolog, and more recently language-integrated technologies like LINQ.
  • Even with specialized languages for heavy-duty querying and search tasks, we still end up writing a lot of our own nested looping and control structures for the purpose of looking through possibilities when it’s not worth crossing that language barrier.  This kind of thing is responsible for some of the more complex code structures you find these days.

While the first two problems of failure and dependence are at least partly solved by current mainstream programming languages, non-determinism is as yet solved mostly by special-purpose sub-languages, with LINQ as the notable exception.

The Fourth Horseman: Destruction

Finally, the fourth problem is destruction.  Evaluating a math-type function is observable only in that you now know the answer.  But in computer programming, functions can have permanent effects on the world: displaying information, waiting on responses from other computers or people, printing documents, even quite literally exploding things, if they are running on military systems!  Because of this, things that aren’t specified in mathematics, like the order in which evaluation happens, matter quite a lot here.

The destructive nature (by which we just mean having effects that can’t be undone) of computer programming functions has plenty of consequences. It makes programming more error-prone. It makes it harder to divide up a task and work on different parts simultaneously, such as you might want to do with a modern multi-core computer, because doing the parts in the wrong order might be incorrect. But at the same time, these destructive effects are in a sense the whole point of computer programming; a program that has no observable effects would not be worth running! So in practically all mainstream programming languages, our functions do have to cope with the problem of destruction.

Back To The Function

Now we’ve seen the faces of some problems we find in the computer programming world. We build software that might fail, has to deal with a ton of extra context, models non-deterministic choice, and sometimes has observable effects on the world that constrain when we can perform the computation.

It may now seem that we’ve left the nice and neat world of mathematical functions far behind. We have not!  On closer inspection, we’ll see that if we can just squint hard enough, each of these quasi-functions can actually be seen as true, honest-to-goodness functions after all.  There is a cost, though. To turn them into real functions, we need to change the range of those functions to something else.  Let’s see how it works for each of our function types in turn:

Functioning With Failure

Our first example of pseudo-functions were those that might fail. It’s not hard to see that a function that could fail is really just a function whose results include two things:

  • successes, which are the intended possible results; and
  • failures, which are descriptions of why the attempt failed.

So for any set A, we’ll define a new set called Err(A) to be just A together with possible reasons we might have failed. Now a possibly failing function from a set A to a set B is really just an ordinary function from A to Err(B).

Functioning With Dependence

Our second type of pseudo-functions were those that depended on information that they got from the world around them: perhaps preferences or application settings. We play a similar trick here, but for a set A, we will define the set Pref(A) to be the set of functions from application settings to the set A. Now watch closely: a function in context from A to B is just an ordinary function from A to Pref(B). In other words, you give it a value from the set A, and it gives you back another function that maps from application settings to the set B.

As confusing as that might sound, a function whose range is another function is really just a function of two parameters, except that it takes its parameters one at a time! Take a minute to convince yourself of this. The conversion between these two equivalent ideas is sometimes called “currying”. So by changing the range of our function, we actually effectively added a new parameter, and it now receives the application settings as a parameter. Remember that except for being inconvenient (we’ll deal with that later), that’s exactly what we wished for.

Functioning With Uncertainty

This is perhaps the most obvious example of all. Our third type were those that represent non-determinism: instead of one specific answer, they have many possible answers. This is easy enough: for each set A, define P(A) to be the power set of A, whose members are themselves sets of values of A. Then a non-deterministic function from A to B is just an ordinary function from A to P(B).

Functioning With Destruction

Our final trick is to deal with functions that have destructive effects. Here we’ll need to be a bit more elaborate in constructing a new range: for each set A, we define IO(A) (standing for input/output, which captures the notion of effects that interact with the rest of the world). An element of the set IO(A) is a list of instructions for obtaining a member of A. It is not a member of A, merely a way to obtain one, and that procedure might have any number of observable effects.

Now we play the same trick and change the range: a destructive function from A to B is just an ordinary plain old mathematical function from A to IO(B). In other words, if you give me an A, then as a plain old function I can’t actually do the steps to get a B, but I can certainly tell you what they are.

But what about composition? It’s great to be back in the world of plain functions, but remember what got us here in the first place? We liked functions because we liked composition; but it seems we’ve now lost composition! If I have a possibly failing function from A to B, and another from B to C, well now I’ve turned them into functions from A to Err(B) and then B to Err(C). Those function domains and ranges don’t match up, and I can’t compose them!

Oh no…

Hold Your Horses, Heinrich Kleisli to the Rescue!

Well, all is not lost. I just haven’t yet told you how to compose these “special” functions.

Because some math dude found these things before us, we call our “special” functions by a name: Kleisli arrows.  There are two things going on here at once, so keep your eyes open: first, Kleisli arrows are just plain old ordinary functions, but with weird-looking ranges. Since they are just functions, you can compose them as functions, and that’s just fine.  But at the same time, they are “special”, and we can compose them as Kleisli arrows, too.

Remember what we decided earlier? The right way to think about composition is by talking about a category. Sets are a category, and that’s fine if you want plain function composition. But now we want a new kind of category, too. It’s called the Kleisli category. If you don’t remember what all the parts of a category are, take a second to review them. To define a category, I need objects, arrows, identities, and composition.

  • To keep things simple, the objects in this new category will be the same: they are just sets of things.
  • The arrows in this category are, unsurprisingly, the Kleisli arrows.
  • I haven’t told you yet what the identities and composition look like, so let’s do that next.

First, we look at failure. We’re given a failure Kleisli arrow from A to B, and one from B to C. We want to compose them into a Kleisli arrow from A to C. In other words, we have an ordinary function from A to Err(B), and a function from B to Err(C), and we want one from A to Err(C).  Take a minute to think about what to do.

The central idea of error handling is that if the first function gives an error, then we should stop and report the error. Only if the first function succeeds should we continue on to the second function, and give the result from that (regardless of whether it’s an error or a success).

To summarize:

  1. If g(x) is an error, then (f · g)(x) = g(x).
  2. If g(x) is a success, then (f · g)(x) = f(g(x)).

To complete the definition of a category, we also need to decide about the identity Kleisli arrows. These are the ones that don’t do anything, so that if you compose them with any other Kleisli arrow, it doesn’t change the other one. Identities are functions from A to Err(A), and it turns out these are just the functions f(x) = x, just like for sets. Notice that means they never return an error; only a successful result.

I’ll run more briefly through the remaining three examples, but I encourage readers who aren’t yet clear on how this will work to write them out in more detail and use this opportunity to become more comfortable with defining a second category of Kleisli arrows.

Next we have Klesli arrows for dependence, which are functions from A to Pref(B).  Recall that adding the Pref to the range is equivalent to adding a new parameter for the application preferences. The key idea here is that if I have two functions that both need to know the application preferences, I should give the same preferences to both. Then composing two of these Kleisli arrows just builds a new function that gets the extra preferences parameter, and passes the same one along to the two component parts. And identities? A Kleisli identity will get that extra preferences parameter, but will ignore it and just return its input anyway.

The Kleisli arrows for uncertainty, or non-determinism, are functions from A to P(B), the power set of B. The key idea for non-determinism is that at each stage, we want to try all possible values that might exist at this point, and collect the results from all of them. So the composition calculates the second function for each possible result of the first, then the resulting possibilities are merged together with a set union. The identities, of course, aren’t really non-deterministic at all, and just return one-element sets containing their input.

Finally, Kleisli arrows for destructive effects are functions from A to IO(B). The key idea here is to combine instructions by following them in a step-by-step manner: first do one, then the next. So the composition writes instructions to perform the first action, look up the second action from the result, and then perform the second action, in that order. A Kleisli identity here is just an instruction to do nothing at all and announce the input as the result. So for each of the four motivating examples, we created a new category, the Kleisli category.

These new categories have their own function-like things, and related ideas of composition and identities, that express the unique nature of each specific problem. By using the appropriate notion of composition in the right Kleisli category, you can solve any of these long-standing computer programming problems in a nice composable way.

And that’s why you should care about monads.

Monads?!? Oh yes, I should mention that we’ve just learned about monads. We simply forgot to use the word.

What’s This Got To Do With Monads?

This section is for those of you who want to know how the stuff we said earlier are related to monads as they are understood in mathematics.  If you open Wikipedia, or most category theory textbooks, and look up monads, they won’t look very much like what we just did. You’ll see something about an endofunctor, and two natural transformation, and properties about commuting triangles and squares.

We haven’t talked about functors at all, much less natural transformations… so how could we have possibly learned about monads? It turns out there’s more than one way to describe monads.  The one we’ve just gone through is an entirely valid one. The shifts we made to the ranges of our functions earlier – Err, Pref, P, and IO – are actually examples of monads. To make sure they are monads in the conventional math way, we’d have to work pretty hard: first, prove that they are functors. Then build two natural transformations called η and µ, and prove that they are natural. Finally, prove the three monad laws.

But wait, there’s an easier way!  Heinrich Kleisli, whom we’ve already met from the categories earlier, pointed out that if you can build a category like the ones we did in the last section, whose arrows are just functions with a modified range, then your category is guaranteed to also give you a monad. That’s quite convenient, because as computer programmers, we tend to care a lot more about our Kleisli arrows than we do about a mathematician’s idea of monads.  Remember, those Kleisli arrows are exactly the modified notion of functions that we were already using, long before we ever heard a word about category theory!  And Kleisli tells us that as long as composition works the way we expect with our Kleisli arrows (namely, that it’s associative and the identities act like identities), then all that other stuff we’re supposed to prove to show we have a monad just happens for us automatically.

Still, it’s an interesting side question to look at the relationship between the two. I won’t give all the details, but I’ll give the structure, and then leave the interested reader with some familiarity with category theory to fill in the proofs of the relevant properties. We’ll use Err as our monad, just to pick a specific example, but nothing here is specific to Err.

  1. We start with Err, which is already a map from objects to objects. But the traditional definition of a monad also requires that it be a functor. That is, given a function f from A to B, I need a way to construct a function Err(f) from Err(A) to Err(B). I do it as follows: in the underlying category (not the Kleisli category, just the category of sets), I find an identity function from Err(A) to Err(A). Then I find a Kleisli identity from B to Err(B). I compose that Kleisli identity in the underlying category with f, and get a function from A to Err(B). I can now do a Kleisli composition of the identity from Err(A) to Err(A) and the function from A to Err(B), and get a function from Err(A) to Err(B). That’s the one I’ll call Err(f).
  2. Next, I need a natural transformation η, from the identity functor to Err. This is easy: the components of η are the Kleisli identities.
  3. Finally, I need a natural transformation µ from Err² to Err. To get the component of µ at A, I take the identity functions in the underlying category from Err (Err A) to Err (Err A), and then from Err A to Err A, and I combine them with Kleisli composition to get a function from Err (Err A) to Err A. This is the component of µ.

The construction in the opposite direction is easier. Given a monad Errwith ? and µ, the Kliesli category is constructed as follows.

  1. The identities are just the components of η.
  2. Given a function f from A to Err(B) and a function g from B to Err(C), I compose the two as µ · Err(g) · f.

Again, the details and the proofs of the appropriate monad and category laws are left to the reader. I hope this brief aside has been useful. I now return to using the word “monad” but talking about monads via Kleisli categories.

Joining The Monadic Revolution

Once again, let’s pause to sum up.

  • Computer programmers like to work by composing some things together, which we call functions.
  • They aren’t functions in the obvious way… but they do make up a category.
  • Actually, they are functions after all, but only if you squint and change the ranges into something weirder.
  • The category that they form is called a Kleisli category, and it’s basically another way of looking at monads.
  • These monads / Kleisli categories nicely describe the techniques we use to solve practical problems.

It’s not just about those four examples, either.  Those are typical of many, many more ideas about programming models that can be described in the same framework. I think it’s fair to sum up and say, at this point, that someone interested in studying and analyzing programming languages and models should be familiar with some ideas from category theory, and with monads in particular.

But still, what about the humble computer programmer, who is not designing a new language, is not writing research papers analyzing programming languages, but just wants to solve ordinary everyday problems?  That’s a fair question.  As long as monads remain just a mathematical formalism for understanding what computer programmers mean by functions, the practicing computer programmer has a good claim to not needing to understand them.

It’s becoming clear, though, that monads are on their way into practical programming concerns, too.  In the past, these Kleisli arrows, the modified notions of “function” used by computer programmers, were built into our programming languages.  Functions in C used a Kleisli arrow, and C++ functions used a different one.  The language specification would tell us what is and what is not possible using a function in this language, and if we wanted something different, too bad.  Maybe once a decade, we’d make the swap to a brand new programming language, and bask in the warm rays of some new language features for a while.

The Past: Error Handling

Consider the Err monad, which gave us functions that might fail and report their failure in structured ways.  Modulo some details and extensions, this is basically structured exception handling. Looking to history, programmers worked without exception handling in their programming languages for many years. Of course, languages like C are all Turing complete, and can solve any possible computational problem, proper error handling included. But we don’t apply categories to think about possible computations; categories are for thinking about composition. Without exception handling in the notion of a “function” that’s provided by languages like C, programmers were left to do that composition by hand.

As a result, any C function that could fail had to indicate that failure using a return value.  In many cases, conventional wisdom built up saying things like “return values are for indicating success or failure, not for giving back answers”. Coding conventions called for most if not all function calls to be followed with if statements checking for failure, and the resulting code was borderline unreadable.  This was the heyday of flowcharts and pseudo-code, because no one expected to be able to understand real code at a glance!  In reality, though, programmers only checked for errors when they thought they was possible, and a lot of errors went undetected. Programs were often unreliable, and likely untold billions of dollars spent on extra development work and troubleshooting.

What was the reason for this?  It’s quite simple: the C programming language and others of its time provided an insufficient kind of Kleisli arrow!  If their Kleisli arrow had included the functionality from the Err monad we defined above, this could have been avoided.  But the notion of what a function means in C is fixed, so the answer was to deal with it, and eventually migrate to a different programming language, rewriting a lot of software, and likely costing another untold billions of dollars.

The Present: Global Variables and Context

What about the Pref monad, and others like it? As discussed earlier, this is about defining computations in a larger context of available information and state of the world.

In the past, we had global variables, the slightly more modern equivalent of just storing information at a known place in computer memory. Quick and dirty, but even 30 years ago, programmers knew they were the wrong answer, and wouldn’t be manageable for larger programs. Object oriented programming tried to alleviate the problem a little, by having functions run in a specific “object” that serves as their context, and that was implicitly passed around at least within the implementation of the object itself. To get this, everyone effectively had to change programming languages to get a better Kleisli arrow again.  But even so, object-oriented languages don’t give a perfect answer to this problem.

The Near Future (/ Present): Purity, Effects, Parallelism, Non-Determinism, Continuations, and More!

This point is about the future, but I’ll start out by pointing out that everything here is already possible, but just requires an appropriate choice of programming language!

One current challenge for the computer programming community is finding effective ways to handle parallelism. Ironically, while past examples have focused on the problem of putting too little power into a language’s Kleisli arrow, the problem this time is too much!  Plain (also known as “pure”) functions present lots of opportunities for parallelism. When code is executed in parallel, it may run faster, or if the parallelism is poorly designed it may even run slower, but in any case it will certainly still give the same answer. But when the Kleisli arrow incorporates destructive updates, that is no longer the case. Now parallelism is risky, and might give unexpected or incorrect results due to so-called race conditions.

We can’t just remove destructive updates from a language’s Kleisli arrow, though.  A program that has no observable effects at all isn’t useful. What is useful is the ability to separate the portions of code that perform destructive update from those that just compute pure functions.  So for the first time, we need a language with more than one kinds of Kleisli arrow, in the same language!

There is already at least one language that offers precisely this. Programmers in the Haskell language can build their own monads, and work in the Kleisli category of a monad of their choosing.  The programming language offers a nice syntax for making this approach readable and easy to use.  If something might fail, you can throw it in Err.  If it needs access to the application settings, throw it in Pref.  If it needs to do input or output, throw it in IO.  Haskell web application frameworks and similar projects start by defining an appropriate monad with the appropriate features for that kind of application.

Another current trend in the computer programming community is toward building more domain-specific programming models. The language Erlang became popular specifically for providing a new programming model with advantages for parallelism.  Microsoft’s .NET framework incorporates LINQ, which offers a programming model that’s better for bulk processing and querying of collections of data.  Rails popularized domain-specific languages for web applications.  Other languages offer continuations as a way to more easily build specify computations in a more flexible way.  All of these are examples of working in new and different Kleisli arrows that capture exactly the model appropriate for a given task.

It comes down to this: If we believe that there is one single notion of “function” that is most appropriate for all of computer programming, then as practical programmers we can find a language that defines functions that way, and then forget about the more general idea of monads or Kleisli arrows as a relic of theoreticians.  But it’s not looking that way.  The programming community is moving quickly toward different notions of what a function means for different contexts, for different tasks, even for different individual applications.  So it’s useful to have the language, the tools, and the intuition for comparing different procedural abstractions.  That’s what monads give us.

Abstraction Over Monads

Using a language with a choice of monads offers some other advantages here, too.  It gives us back our abstraction.  In Haskell, for example, it’s possible to write code that is applicable in multiple different monads. A surprising amount of the programming done with one monad in mind actually has meaning in very different monads! For example, consider the following Haskell type:

sequence :: Monad m => [m a] -> m [a]

What this means is that for any monad, which we’ll call M, sequence converts from a list of values of M(A) into M(List(A)), the monad applied to lists themselves.  Let’s take a minute to consider what this means for each of our four examples. For Err, it takes a list of results that might be failures, and if any of them are failures, it fails; but if not, then it gives back a list of all the results.  It’s basically a convenient way to check a whole list of computations for a failure. For Pref, it takes a single set of application preferences, and distributes that to everything in the list, giving back a list of the results. For the power-set monad, P, it would take a list of sets, and give back a set of all the ways to choose one item from each set.  And for IO, it takes a list of instruction cards, and gives back the single card with instructions for doing all of them in turn.  Amazingly, this one function, which had only one implementation, managed to make sense and do something useful for all four of our examples of monads!

Along with a choice of monads comes the ability to abstract over that choice, and write meaningful code that works in any monad that you do end up choosing.

Between all of these forces, I predict that within the next ten years, software developers will be expected to discuss monads in the same way that most developers currently have a working vocabulary of design patterns or agile methodologies.

Beyond Monads: More Categorical Programming

While most of this has been about monads, I don’t want to leave anyone with the impression that monads are the only influence of categories in computer programming.  All of the following ideas have found their way into programming practice, mostly (so far) within the Haskell programming language community because of its flexibility and a deep academic culture and history.

  • Monad transformers are a powerful technique for combining the effects of more than one monad to build rich and powerful programming models.
  • Functors and applicative functors (a.k.a. strong lax monoidal functors for mathematicians) are weaker than monads, but more widely applicable.
  • Other kinds of categories that are not Kleisli categories can often be defined and composed to solve specific problems. Freyd categories are also useful.

I’ll stop there, but only as an encouragement to look more into the various abstractions from category theory that programmers have found useful. A good starting point is the (Haskell-specific) Typeclassopedia by Brent Yorgey. That’s just a door into the many possibilities of applying category-based ideas and intuitions in computer programming.

But I hope I was able to convey how these ideas aren’t just made up, but are actually the natural extension of what computer programmers have been doing for decades.


by cdsmith at April 19, 2012 08:12 PM

Well-Typed.Com

Parallel Haskell Digest 9

The Google Summer of Code is upon us and students have already submitted their proposals. There are a couple potential projects on concurrent data structures, which we'll have a look at below.

We will also be continuing our tour of Haskell concurrency abstractions with our word month, transaction. This digest is brought to you by the Parallel GHC project, an MSR-sponsored effort to push parallel Haskell technologies out into the real world. Check our project news below to see how we're doing in that front.

Finally, you may heard Functional Propaganda from a Simon or two. But how would the same message affect you if it came from a hardcore C++ hacker? If you haven't seen it making the rounds, have a quick look at Bartosz Milewski's The Downfall of Imperative Programming, and maybe tell your imperative friends? The FP monster is hatching from its academic egg; best be prepared!

News

Let's have a quick look at some of those GSoC proposals, particularly those with a parallel Haskell theme. It's all about performance this year. Two of the proposals involve using or improving parallellism in Haskell, and four are related to high-performance concurrency.

  • Parallel CSG engine

    Constructive Solid Geometry (CSG) is the common approach to define complex bodies in engineering applications, ray tracing engines and physical simulators. Dmitry Dzhus' proposal is to deliver a fast parallel CSG engine using the Accelerate/Repa libraries for vectorised computations on GPU.

  • NUMA supporting features for GHC

    Sajith Sasidharan wants to reach into the GHC RTS, with the aim of “extracting that last ounce of performance from NUMA systems, by firing all CPUs if/when necessary and by ensuring a suitably NUMA-aware memory allocation behavior.” Work in this area would benefit folks doing scientific computing, who may need great amounts of task and data parallelism.

  • Windows support for the new GHC I/O manager

    The new I/O manager (GHC 7) brings great scalability improvements to Haskell — 10k simultaneous connections to your web server? no sweat. Unfortunately for Haskellers on Windows, these improvements are currently only available on Unix. Mikhail Glushenkov (who worked on GSoC last year to make improvements to Cabal) proposes to remedy this, making the new manager available on Windows. The challenge is that Windows I/O completion ports have slightly different semantics than their epoll/kqueue analogues on Unix.

  • Lock-free hash table and priority queue

    More along the theme of high performance concurrency, Florian Hartwig's project aims to use GHC's new-ish atomic compare-and-swap (CAS) primitive to implement a high-performance lock-free hash table and a lock-free priority queue in Haskell. The CAS instruction is a bit of hardware support for concurrency: it compares a value in memory to an expected value, and iff they match, replaces it with a new value.

  • Implement Concurrent Hash-table / Hashmap

    Loren Davis also proposes a thread-safe mutable hash table implementation in Haskell. Loren is still weighing some of the alternative approaches suggested in the Haskell community. He is currently leaning towards a lock-stripping approach as it would fulfill an immediate need in the community.

  • Concurrent Datastructures with good Performance

    Mathias Bartl aims to implement two concurrent data types in Haskell, along with the usual battery of automated unit tests and benchmarks. The two that Mathias has in mind are a lock-free concurrent bag, and a concurrent priority queue.

Parallel GHC project update

We have been continuing our work to make ThreadScope more helpful and informative in tracking down your parallel and concurrent Haskell performance problems. We now have the ability to collect heap statistics from the GHC runtime system and present them in ThreadScope. These features will be available for users of a recent development GHC (7.5.x) or the eventual 7.6 release. In addition to heap statistics, we have been working on collecting information from hardware performance counters, more specifically adding support for Linux Perf Events. This could be useful for studying IO-heavy programs, the idea being to visualise system calls as being distinct from actual execution of Haskell code.

Speaking of performance, we are also continuing work on the new Cloud Haskell implementation (see Duncan Coutts' Fun in the Afternoon Talk), and have lately been focused on reducing message latency. This consists of work in three areas: improving binary serialisation, investigating the implications of using Chan and MVar to pass messages between threads, and perhaps improving the Haskell network library implementation to compete better with a direct C implementation.

Word of the month

Lately, we've been investigating the various ways Haskell helps us to get to grips with concurrency. We talked about how the MVar, the Haskell variant on locks, allows us to share mutable variables between threads, with some safeguards to help ensure consistency. MVar's may provide a nice high-level packaging around locks, but as we mentioned in the last digest, they can still go horrifically wrong, just like locks and synchronized methods in other languages.

We could go through the usual litany of reasons why locks are bad news, but maybe a healthier approach would be for us to focus on the positive. What do we want as programmers? One possibility is what Simon PJ (Beautiful Concurrency) calls “modular programming”, the ability to “[build] large programs by gluing together smaller programs”. Locks fall short of helping us to meet this desire. First, because the mere act of combining two locky programs may be inherently incorrect; withdraw acct1 amt >> deposit acct2 amt is bad because of the gap between the two actions where the money is in neither account. Second, because they seal off programs that we may otherwise like to moosh together; if process p1 waits for input on a pipe, process p2 waits for input on another pipe, how do wait for either of p1 or p2? So how do we wrestle back this modularity from our locky masters? And how do we make programming fun again?

Our word of the month today is “transaction”. Software transactional memory (STM) takes this idea of a transaction (a sequence of operations that can be treated as a single atomic block) from database design. The Haskell implementation of STM was introduced in the 2005 paper Composable Memory Transactions by Harris et. al. If programming fun is what you're after, this is a paper that comes with its own war-cry: “compositionality: a programmer can control atomicity and blocking behaviour in a modular way that respects abstraction barriers.”

Here are some quick highlights of the stm library. You may notice a couple of things, first that this library introduces its own notion of variable, the TVar (MVar,IVar; what, no pirate joke?) and second that STM involves a new monad of its own. Unlike the MVar that we saw in the last digest, TVar's do not have the same notion of being full or empty; they just hold values plain and simple. As for the STM monad, we will see why it matters when we first try to do some IO.

 -- Control.Concurrent.STM
 data STM a
 instance Monad STM
 
 atomically :: STM a -> IO a
 
 data TVar a
 newTVar   :: a -> STM (TVar a)
 readTVar  :: TVar a -> STM a
 writeTVar :: TVar a -> a -> STM ()
 
 retry  :: STM a
 orElse :: STM a -> STM a -> STM a

To get a rough idea how some of this is used, let's look at the transactional hello world, safely wiring money from one bank account to another. For the purposes of our example, a bank account is just a balance. To get some money from an account, we read the balance, subtract the amount, and write the new balance. Making a deposit is just withdrawing negative-money.

 type Account = TVar Int
 
 withdraw :: Account -> Int -> STM ()        
 withdraw acc amount = do
     bal <- readTVar acc
     writeTVar acc (bal - amount)
 
 deposit :: Account -> Int -> STM ()
 deposit acc amount = withdraw acc (- amount)

These primitive operations (withdraw and deposit) bring us to the question of modularity. How do we know that it's safe to combine these mini-programs into a bigger one? In other words, if we write something like withdraw from 42 >> deposit to 42, how do we avoid the possibility of running into some twilight zone state where the money is neither here nor there? If people do strange things like simultaneously transfering money in the other direction, will our program still work?

The answer lies in the distinction between STM (transactions) and IO (actions). So long as we remain in STM, we are simply assembling transactions, piecing smaller ones (“withdraw from a”) into larger ones (“withdraw from a and deposit it to b”), but not actually performing them! Having composed our transactions, we can use the function atomically to turn them into IO actions.

 -- still just a transaction
 transfer :: Account -> Account -> Int -> STM ()
 transfer from to amount = do
     deposit to amount
     withdraw from amount
 
 -- now we have an action!
 doTransfer :: Account -> Account -> Int -> IO ()
 doTransfer from to amount =
     atomically $ transfer from to amount

And atomically does what it says on the tin: it runs the transaction in a way that renders it indivisible, no twlight zones. Lest there is any confusion, even though the transaction is indivisible, we can still have concurrency during the course of the transaction, even simultaneously read the affected TVars if we want to. The indivisibility simply means that we never catch our transactions with their pants down. We neither read nonsense mid-transactional values (simultaneous reads would either get the before or after value), nor injecting values into a transaction mid-stream.

To get a feel for how these guarantees are possible, it could be useful to take a peek under the hood. For each transaction that is run, GHC maintains a thread-local log with an entry for each TVar accessed in that transaction. Each entry contains both the old value and the new value that would be committed if the transaction is succesful. This may be easier to see with a silly example:

main = do
    v1 <- atomically $ newTVar "Joe"
    v2 <- atomically $ newTVar "Bob"
    done <- atomically $ newTVar 0
    -- thread A (you can just pretend forkDelayIO == forkIO)
    forkDelayIO . atomically $ do
                              -- transaction log if A runs first
        x <- readTVar v1      -- v1: Joe -> Joe
        y <- readTVar v2      -- v1: Joe -> Joe, v2: Sue -> Sue 
        writeTVar v1 "Sue"    -- v1: Joe -> Sue
        writeTVar v2 x        -- v1: Joe -> Sue, v2: Bob -> Joe 
        writeTVar v1 y        -- v1: Joe -> Bob, v2: Bob -> Joe
        modifyTVar done (+1)  -- (stm 2.3 but easy to define)
    -- thread B 
    forkDelayIO . atomically $ do
                              -- (if A runs first)
        writeTVar v1 "Jean"   -- v1: Bob -> Jean
        writeTVar v2 "Paul"   -- v1: Bob -> Jean, v2: Joe -> Paul
        modifyTVar done (+1)
    waitThreads 2 done
    people <- atomically $ do -- (if A runs first)
        p1 <- readTVar v1     -- v1: Jean -> Jean
        p2 <- readTVar v2     -- v1: Jean -> Jean, v2: Paul -> Paul
        return (p1, p2)
    print people -- if A runs first, (Jean, Paul)
                 -- if B runs first, (Paul, Jean).

-- boring details just for this example
forkDelayIO job = forkIO $
    randomRIO (1, 1000000) >>= threadDelay >> job
waitThreads n v = atomically $
    do { d <- readTVar v;  when (d < n) retry }

In the above, we fork off two threads, A which swaps a pair of names and, B which overwrites them with other names. Atomicity here means that other threads never see any intermediary states and state changes from other threads don't affect the current thread. For example, thread B should never see v1 being set to "Sue". Likewise, if thread A should still read "Joe" from v1 even if B simultaneously writes "Jean".

This is made possible by validation of the transaction logs. Validation normally occurs at the end of a transaction (we won't cover the two other cases here: exceptions, and thread wake-ups). It consists of checking that all the expected “before” values for TVars still match reality. If the logs are good, we commit the new values; if not, we simply discard them and try the transaction again, taking the new reality into account. This validate-and-commit model allows us to run transactions simultaneously, safely, but with the occasional rollback and retry to ensure atomicity.

The notion of a transaction log brings us to the notion of cost. Good things don't always come cheap, and using a good thing like STM may require a little familiarity with the cost model behind it. Basically, it's important to keep in mind that the values we write to TVar's may come from some arbitrary expression, and that arbitrary expressions may be arbitrarily expensive. So being forced to retry transactions may involve redoing something expensive. If the transactions affect many variables, the chances of hitting a retry go up. Likewise, if the transaction takes a long time to run, the chance goes up of some other thread making a change that triggers a retry. In the pathological worst case, you can have some transactional behemoth that never manages to commit; because some smaller faster transaction keeps stealing its thunder. So keep an eye out for starvation and the more general problem for retries being expensive.

Cost may be a bit of a bummer, but there's also a Haskell-related silver lining behind all this. Because we have a purely functional language and the enforced separation between pure functions and side-effecting actions, STM is actually quite practical in Haskell. The number of things we need to track in a transaction log is limited to handful of explicit TVars rather that just about everything. If you are coming from other languages, you may have a memory of STM as being nice, but wildly impractical. Not so in Haskell. Eminently viable.

Aside from making STM practical, this sort of separation is also good for general peace of mind. Suppose for example that we coded up a feature in our banking software to send our users a text message alert whenever their balances fall below a threshold. If we were in the middle of a complicated transaction, we might be tempted to just slap that logic right in the middle of the transaction; however, the Haskell implementation makes this deliberately impossible. This can be a bit frustrating at first (and new Haskellers are sometimes faced with the “how do I get this out of the monad” puzzle), but saves us the greater danger of bombarding our users with spurious retry-induced text messages.

The guarantees that STM offer make it a great place to get started with Haskell concurrency. After all, why make software any buggier than it needs to be? If you do want to get started, have a look at Simon Peyton Jones' Beautiful Concurrency. It's a particularly good idea to do so, because there's some really interesting ground that we've not covered here (briefly, blocking, the retry function aborts the current transaction, and causes it to be retried when appropriate; and choice: a orElse b tries a, and if that should retry, then b, and if that should also retry, the whole expression again). Other great STM resources are Simon Marlow's tutorial on parallelism and concurrency and the Real World Haskell chapter on STM. With the four resources combined, you'll see a nice range of examples from the usual bank-account one to concurrently shuffling windows between desktops.

Blogs

  • The Downfall of Imperative Programming (9 Apr)

    Take a hardcore C++ veteran with imperative programming in his bloodstream and loads of experience under his belt. Now give him a passion for concurrency and what do you get? First, you get a keen awareness that the future is massively multicore. Second, you get a hard-won appreciation for how difficult concurrent programming can be; for all jokes we make in the Haskell community about firing the missiles, the consequences of data races can sometimes be deadly. Third, you get the conviction that functional programming is the inevitable way forward.

    Bartosz Milewski sums up the situation thus: Sooner or later you’ll have to face the multicore reality. You will be forced to learn functional methods to the extent to which your imperative language supports them. Despite that, data races will infest your code and leak into released products. So you might as well take a shortcut and embrace a full blown functional language now.

    See what you make of his blog post if you have not done so already. There's quite a bit of buzz about this post, so you may also be interested in the programming reddit discussion around it as well.

  • Building A Concurrent Web Scraper With Haskell (10 Mar)

    Let's make a concurrent web scraper! This blog post by Aditya Bhargava presents a hands-on introduction to both arrows (via hxt) and concurrency (via parallel-io). Aditya builds from the bottom-up, showing us little pieces of program that we might cobble together, culminating in a 52 line Haskell program that crawls web sites and fetches images within their pages. The parallel-io library used in this tutorial provides a thread pool which minimises contention by guaranteeing a limit to the number of unblocked threads running at the same time. It uses lock based concurrency with MVar's under the hood.

  • 0MQ and Haskell (6 Mar)

    Magnus Therning could not find any excuses to look into 0MQ. But “to hell with reason”, Magnus ended up deciding to just poke around without any specific goal in mind. He found a nice tutorial based on Python and translated its mini examples into Haskell. Magnus wonders why the API for subscribe is String rather than ByteString based. Also, he's finding that his client mysteriously dies after receiving a few messages. Any comments?

  • SIMD Support for the vector library (27 Mar)

    Single instruction, multiple data (SIMD) is the sort of thing you might be interested in if you're into data parallelism: hardware that can perform the same instruction on multiple data simultaneously. Geoffrey Mainland posted about his efforts to bring SIMD support to GHC, the bigger picture being that you ought to be able to write nice high-level Haskell and have it work as fast the low-level Haskell or C that you might otherwise crank out. To try things out, Geoff benchmarks taking the dot product of two vectors in various Haskell and C versions. No happy ending yet, unfortunately: while the low-level Haskell version is competitive with C, the high-level is not. Check the post out for dissection of the results down the Core and assembly level. Hopefully better news in a follow-up posting.

  • Adding SIMD Support to Data Parallel Haskell (18 Apr)

    Hopefully better news? Not as such, but perhaps something more interesting. In the previous post, we could make use of SIMD support by issuing some explicit instructions from the vector library. OK, but what about people who writing parallel code, say, by using a parallel arrays framework? And what if you could get this SIMD support virtually for free — no syntax attached? Geoff makes this possible by extending the Data Parallel Haskell framework so you would only have to tweak a single import statement, and exploitation of SIMD instructions would be automatic. See the posting for some nice benchmarks and also a brief introduction to Data Parallel Haskell.

  • Work Efficient Higher-Order Vectorisation (24 Mar)

    Found on Manuel Chakravarty's tumblr: Our new draft paper on Work Efficient Higher-Order Vectorisation introduces a novel representation for nested, irregular parallel arrays that enables the work-efficient SIMD-ification of nested data parallelism — i.e., nested parallelism is transformed into flat parallelism, while maintaining the work complexity of a naive pointer-based representation of nested arrays. This solves a long standing problem that dates back to the original implementation of the language NESL.

Talks, tutorials, and packages

  • stm-chans 1.3.1 (1 Mar)

    The stm-chans package offers a collection of channel types, similar to Control.Concurrent.STM.TChan but with additional features. This latest update by wren ng thornton takes advantage of optimisations in the newly released stm-2.3. It's highly recommended that all users bump their minimum stm-chans requirement to version 1.3.1

  • accelerate 0.10.1 (12 Apr)

    Manuel Chakravarty has just released version 0.10.0.0 of Accelerate, an embedded language for GPU-accelerated array computations in Haskell that targets NVIDIA's CUDA framework and also has an experimental (and partial) OpenCL backend. A considerable amount of example code is in the companion package accelerate-examples. The main user-visible changes in this release are frontend bug fixes.

  • Parallel Functional Programming course

    Students at Chalmers and Gothenburg University are currently 6 lectures into a course on parallel functional programming. The course has so far covered parallelism with par/pseq, Strategies and monad-par, using ThreadScope, and skeletons as a means to structure parallel computations. The course page has lecture notes and exercises which could be of interest even if you aren't currently following the course.

  • Cloud Haskell (Fun in the Afternoon)

    Well-Typed's Duncan Coutts was at the recent Fun in the Afternoon, a termly seminar on Functional Programming in the UK). Duncan presented some of the motivation behind Cloud Haskell (”Erlang for Haskell”) and distributed programming, along with the Cloud Haskell design, and our work on a new implementation to follow on the initial protoype by Jeff Epstein. Our new implementation adds a swappable network transport layer. If you're happy with TCP/IP, don't wait for the new implementation; just cabal install remote and give Jeff's prototype a try.

Mailing lists

Concurrency

  • Transactional memory going mainstream with Intel Haswell (9 Feb)

    Ben was wondering if any STM experts would comment on this recent Ars Technica article on the Intel Haswell chip. Austin Seipp pointed us to a comment by Duncan Coutts in the Reddit discussion (unfortunately not; the new extension would sledgehammer all instructions between the XBEGIN/XEND instructions). Ryan Ingram suggests that maybe the extension could be used to optimise the existing implementation, perhaps by wrapping transaction commits with XBEGIN/XEND)

  • Behavior of -threaded in GHC 7.4.1? (14 Feb)

    Mike Craig just debugging a recent issue with GHC 7.4.1 and the zeromq3-haskell library, which provides a provides an FFI binding to libzmq. Unfortunately, code which used to run when compiled on GHC 7.0.4 dies with “operation on non-socket” when built with GHC 7.4.1. With the latter GHC, Mike can only run the code if he omits -threaded, or if he uses the -V0 flag to turn off the RTS clock and associated signals. After more debugging, he tracked the problem down to a addFinalizer on a Socket tuple. The finalizer was being run prematurely, perhaps because the Socket type was being optimised away. Putting the finalizer on the Ptr () in the tuple seems to solve the problem.

  • Question about concurrency, threads and GC (2 Mar)

    Paul Graphov is trying to implement a networked application that supports bidirectional conversations, ie. not just request/response, but also sending notifications to clients. Paul is particularly interested in STM, but he's stuck on a bit of a design problem. His thinking so far is that he'll need to start 3 threads for each client, one to read data from the socket, one that sends queued messages to that socket, and one for the main behaviour loop.

    Joey Adams noticed that this was the same sort of problem we reported in the previous digest. Joey was grappling with making making asynchronous I/O composable and safe. He wound up not using the stm-channelize package that he wrote, and recommends instead a 3-thread solution, using a thread each for receiving, sending, and coordination. Check out the small Haskell chat server that Joey wrote to illustrate the idea.

    Alexander V Vershilov suggests a data-driven behaviour based on conduits and stm channels. He's also provided an example chat server, which you can compare against Joey's version. The two examples take a similar approach, and `could perhaps be combined to good effect.

  • "killThread" hangs! (ironic) (25 Feb)

    Ryan Newton is gathering information in preparation for a possible bug report. He's testing the new network transport layer in distributed-process (Cloud Haskell) and gets hangs in killThread. Strangely, the pattern for hanging goes: GHC 6.12.3 [OK], 7.0.2 [HANGS], 7.2.1 [HANGS], 7.4.1 [OK]. Any ideas? Simon Marlow suggests it may be a bug in the RTS asychronous exception handling code, fixed with commit fa71e6c.

  • Synchronizations in memory allocation? (21 Mar)

    Following up on the recent scaling bottleneck thread, Ryan Newton wondered: “What is the reason for GHC managing all this pinned memory for foreign pointers itself rather than using an external C malloc/free implementation and thus keeping disjoint Haskell and C heaps?” Simon Marlow says it's basically because GHC's mallocForeignPtrBytes is much faster than malloc()/free()

    Ryan was asking because he is looking to how to do better on NUMA platforms. “We've got a NUMA-aware work-stealing scheduler now for monad-par, but it isn't really helping much yet. So we need to answer the question of how well our memory is being localized to socket-preferred physical addresses.” NUMA isn't something the GHC team have looked into for the RTS yet. He has some ideas for improvements to the block allocator; more details in the thread.

  • Haskell for BigData (16 Mar)

    Andrei Varanovich observes that while Haskell has a lot to offer in the world of parallel/concurrent programming (from DPH to Cloud Haskell), it still lacks two important components for working with Big Data: integration with a distributed file system, such as Hadoop distributed file system, a data aggregration framework (eg. MapReduce, but of course something much richer; this being Haskell and all).

    Andrei was interested in submitting a Google Summer of Code proposal to build a big data framework for Haskell on top of Cloud Haskell. I didn't see a proposal this year, but maybe next time? See the thread for technical suggestions, supportive comment, and pointers for a succesful Haskell GSoC project.

Parallelism

  • Help wanted! Parallelism causes space leaks (23 Mar)

    Yavuz Yetim posted a small chunk of code using Strategies for parallelism. When he enables his parList rdeepseq strategy, though, he gets a stack overflow on smallish input (1 MB file), even if he allows GHC to use a 1GB stack. Switching to parMap, parListChunk and other strategies don't seem to help either.

  • Data.Array.Accelerate initialization timingsmore (20 Feb)

    Paul Sujkov is finding that array initialisation in Data.Array.Accelerate takes 10x the amount of time than either Data.Array and bare C++ CUDA array initialisation. Is there anything Paul might be doing wrong in particular? The accelerate package currently provides two backends, an interpreter (reference implementation) and a CUDA backend generating code for CUDA-capable NVDIA GPUs. Martin Dybdal comments that Paul should use Data.Array.Accelerate.use to generate hints to transfer arrays to GPU, and Data.Array.Accelerate.CUDA.run to actually perform the transfer. Manuel Chakravarty adds that the the fromList function is really just meant for testing, or for initialising small arrays. For anything bigger, going from vanilla lists is a bad idea, so have a look at Data.Array.Accelerate.IO.

  • Reasons for Super-Linear Speedup (5 Mar)

    Burak Ekici has parallelized RSA decryption and encryption schemes by using second generation strategies. He's getting 10 times performance improvements… on a quad-core CPU (with an 8MB cache). Is this just mismeasurement, or are there some differences in how GHC handles serial/parallel of computation, say with respect to cache usage? Bardur Arantsson replies that the usual explanation for this sort of thing is that the working data suddenly fits within the per-CPU L2 cache when split up.

StackOverflow and Reddit

Help and Feedback

If you'd like to make an announcement in the next Haskell Parallel Digest, then get in touch with me, Eric Kow, at parallel@well-typed.com. Please feel free to leave any comments and feedback!

by eric at April 19, 2012 11:32 AM

April 18, 2012

TypLAB

The new Silk blog runs on Silk

The day we founded this company, we started blogging. We set up a Wordpress blog and started discussing a variety of technical topics.

by Sander Koppelaar at April 18, 2012 10:00 PM

Jeremy O'Donoghue

wxHaskell News

A great deal has happened in wxHaskell land over the past few weeks, so I thought a summary was worthwhile

wxHaskell 0.90 Released

A significant update to wxHaskell was released on April 14th. This brings in all of the work done by Dave Tapley, Eric and many others to provide support for wxWidgets 2.9.

Supporting wxWidgets 2.9 is important for quite a number of reasons – not least because at some point it will become wxWidgets 3.0 and will be the new stable version. However in the short term the main benefit is support for 64bit OS platforms – notably MacOS X Snow Leopard and Lion.

The slightly odd version numbering convention was chosen to allow wxHaskell 0.13 to evolve without being excessively constrained over version numbering. In any case, it would be nice to get to version 1.0 at some time soon – perhaps when wxWidgets 3.0 is released.

Most of the future wxHaskell development effort will go on the new branch.

wxHaskell 0.13 Branch Created

On many systems, particularly almost all Linux distributions, wxWidgets 2.8.x remains the standard ‘package’ for wxWidgets, so we continue to support this for those who would prefer to use the packages provided by their distro. It also allows Windows users without C++ development environments to use the wxPack binary installers for wxWidgets.

Experimental GitHub Repository

I have created an experimental GitHub repository. It is right up to date at the time of writing and contains two active branches: master is the wxWidgets 2.9 repo and WXWIDGETS_2_8 is (unsurprisingly) the wxWidgets 2.8 branch.

I’m especially interested in feedback on whether moving definitively to GitHub would be a good thing – the main criteria for judgement being whether it makes it easier to receive contributions from others.


Filed under: Haskell, wxHaskell

by jeremyodonoghue at April 18, 2012 02:37 PM

Johan Tibell

The ekg package can now export string labels

I'm happy to announce a new minor release of the ekg package. The ekg package lets you monitor running executables using your web browser, or any monitoring tool that can speak HTTP. New in this release is support for exporting labels.

A label is an arbitrary text string, set dynamically by the running executable. Labels are more flexible than counters and gauges (which can only represent numerical values.) Typical uses of labels include exporting the command line arguments the executable was started with, the host it's running on, etc. Since labels are arbitrary strings you can use labels to export any type of internal state that can be represented as a string. Here's a simple example:

{-# LANGUAGE OverloadedStrings #-}
module Main where

import Control.Concurrent
import Control.Exception
import qualified Data.Text as T
import System.Environment
import qualified System.Remote.Label as Label
import System.Remote.Monitoring

mean :: Fractional a => [a] -> a
mean xs = sum xs / fromIntegral (length xs)

main :: IO ()
main = do
    handle <- forkServer "localhost" 8000
    label <- getLabel "args" handle
    args <- getArgs
    Label.set label $ T.intercalate " " $ map T.pack args
    -- Busy work:
    let loop n = do
            evaluate $ mean [1..n]
            threadDelay 2000
            loop n
    loop 1000000

This functionality was contributed by Iustin Pop.

by Johan Tibell (noreply@blogger.com) at April 18, 2012 04:51 AM

mightybyte

LTMT Part 2: Monads

In part 1 of this tutorial we talked about types and kinds. Knowledge of kinds will help to orient yourself in today's discussion of monads.

What is a monad? When you type "monad" into Hayoo the first result takes you to the documentation for the type class Monad. If you don't already have a basic familiarity with type classes, you can think of a type class as roughly equivalent to a Java interface. A type class defines a set of functions involving a certain data type. When a data type defines all the functions required by the type class, we say that it is an instance of that type class. When a type Foo is an instance of the Monad type class, you'll commonly hear people say "Foo is a monad". Here is a version of the Monad type class.

class Monad m where
    return :: a -> m a
    (=<<) :: (a -> m b) -> m a -> m b

(Note: If you're the untrusting type and looked up the real definition to verify that mine is accurate, you'll find that my version is slightly different. Don't worry about that right now. I did it intentionally, and there is a method to my madness.)

This basically says that in order for a data type to be an instance of the Monad type class, it has to define the two functions return and (=<<) (pronounced "bind") that have the above type signatures. What do these type signatures tell us? Let's look at return first. We see that it returns a value of type m a. This tells us that m has the kind signature m :: * -> *. So whenever we hear someone say "Foo is a monad" we immediately know that Foo :: * -> *.

In part 1, you probably got tired of me emphasizing that a type is a context. When we look at return and bind, this starts to make more sense. The type m a is just the type a in the context m. The type signature return :: a -> m a tells us that the return function takes a plain value a and puts that value into the context m. So when we say something is a monad, we immediately know that we have a function called return that lets us put arbitrary other values into that context.

Now, what about bind? It looks much more complicated and scary, but it's really pretty simple. To see this, let's get rid of all the m's in the type signature. Here's the before and after.

before :: (a -> m b) -> m a -> m b
after  :: (a -> b) -> a -> b

The type signature for after might look familiar. It's exactly the same as the type signature for the ($) function! If you're not familiar with it, Haskell's $ function is just syntax sugar for function application. (f $ a) is exactly the same as (f a). It applies the function f to its argument a. It is useful because it has very low precedence is right associative, so it is a nice syntax sugar that allows us to eliminate parenthesis in certain situations. When you realize that (=<<) is roughly analogous to the concept of function application (modulo the addition of a context m), it suddenly makes a lot more sense.

So now what happens when we look at bind's type signature with the m's back in? (f =<< k) applies the function f to the value k. However, the crucial point is that k is a value wrapped in the context m, but f's parameter is an unwrapped value a. From this we see that the bind function's main purpose is to pull a value out of the context m and apply it to f, which does some computation, and returns the result back in the context m again.

The monad type class does not provide any mechanism for unconditionally pulling a value out of the context. The only way to get access to the unwrapped value is with the bind function, but bind does this in a controlled way and requires the function to wrap things up again before the result is returned. This behavior, enabled by Haskell's strong static type system, provides complete control over side effects and mutability.

Some monads do provide a way to get a value out of the context, but the choice of whether to do so is completely up to the author of said monad. It is not something inherent in the concept of a monad.

Monads wouldn't be very fun to use if all you had was return, bind, and derived functions. To make them more usable, Haskell has a special syntax called "do notation". The basic idea behind do notation is that there's a bind between every line, and you can do a <- func to unwrap the return value of func and make it available to later lines with the identifier 'a'.

You can find a more detailed treatment of do notation elsewhere. I hear that Learn You a Haskell and Real World Haskell are good.

In summary, a monad is a certain type of context that provides two things: a way to put things into the context, and function application within the context. There is no way to get things out. To get things out, you have to use bind to take yourself into the context. Once you have these two operations, there are lots of other more complicated operations built on the basic primitives that are provided by the API. Much of this is provided in Control.Monad. You probably won't learn all this stuff in a day. Just dive in and use these concepts in real code. Eventually you'll find that the patterns are sinking in and becoming clearer.


by mightybyte (noreply@blogger.com) at April 18, 2012 02:25 AM

April 17, 2012

mightybyte

The Less Travelled Monad Tutorial: Understanding Kinds

This is part 1 of a monad tutorial (but as we will see, it's more than your average monad tutorial). If you already have a strong grasp of types, kinds, monads, and monad transformers, and type signatures like newtype RST r s m a = RST { runRST :: r -> s -> m (a, s) } don't make your eyes glaze over, then reading this won't change your life. If you don't, then maybe it will.

More seriously, when I was learning Haskell I got the impression that some topics were "more advanced" and should wait until later. Now, a few years in, I feel that understanding some of these topics earlier would have significantly sped up the learning process for me. If there are other people out there whose brains work somewhat like mine, then maybe they will be able to benefit from this tutorial. I can't say that everything I say here will be new, but I haven't seen these concepts organized in this way before.

This tutorial is not for absolute beginners. It assumes a basic knowledge of Haskell including the basics of data types, type signatures, and type classes. If you've been programming Haskell for a little bit, but are getting stuck on monads or monad transformers, then you might find some help here.

data Distance = Dist Double
data Mass = Mass Double

This code defines data types called Distance and Mass. The name on the left side of the equals sign is called a type constructor (or sometimes shortened to just type). The Haskell compiler automatically creates functions from the names just to the right of the equals sign. These functions are called data constructors because they construct the types Distance and Mass. Since they are functions, they are also first-class values, which means they have types as seen in the following ghci session.

$ ghci ltmt.hs
GHCi, version 7.0.3: http://www.haskell.org/ghc/  :? for helpghci> :t Dist
Dist :: Double -> Distance
ghci> :t Mass
Mass :: Double -> Mass
ghci> :t Distance
<interactive>:1:1: Not in scope: data constructor `Distance'

We see here that Dist and Mass are functions that return the types Distance and Mass respectively. Frequently you'll encounter code where the type and the constructor have the same name (as we have here with Mass). Distance, however, illustrates that these are really two separate entities. Distance is the type and Dist is the constructor. Types don't have types, so the ":t" command fails for Distance.

Now, we need to pause for a moment and think about the meaning of these things. What is the Distance type? Well, when we look at the constructor, we can see that a value of type Distance contains a single Double. The constructor function doesn't actually do anything to the Double value in the process of constructing the Distance value. All it does is create a new context for thinking about a Double, specifically the context of a Double that we intend to represent a distance quantity. (Well, that's not completely true, but for the purposes of this tutorial we'll ignore those details.) Let me repeat that. A type is just a context. This probably seems so obvious that you're starting to wonder about me. But I'm saying it because I think that keeping it in mind will help when talking about monads later.

Now let's look at another data declaration.

data Pair a = MkPair a a

This one is more interesting. The type constructor Pair takes an argument. The argument is some other type a and that is used in some part of the data constructor. When we look to the right side, we see that the data constructor is called MkPair and it constructs a value of type "Pair a" from two values of type "a".

MkPair :: a -> a -> Pair a

The same thing we said for the above types Distance and Mass applies here. The type constructor Pair represents a context. It's a context representing a pair of values. The type Pair Int represents a pair of Ints. The type Pair String represents a pair of strings. And on and on for whatever concrete type we use in the place of a.

Again, this is all very straightforward. But there is a significant distinction between the two type constructors Pair and Distance. Pair requires a type parameter, while Distance does not. This brings us to the topic of kinds. (Most people postpone this topic until later, but it's not hard to understand and I think it helps to clarify things later.) You know those analogy questions they use on standardized tests? Here's a completed one for you:

values : types    ::   types : kinds

Just as we categorize values by type, we categorize type constructors by kind. GHCi lets us look up a type constructor's kind with the ":k" command.

ghci> :k Distance
Distance :: *
ghci> :k Mass
Mass :: *
ghci> :k Dist
<interactive>:1:1: Not in scope: type constructor or class `Dist'
ghci> :k Pair
Pair :: * -> *
ghci> :k Pair Mass
Pair Mass :: *

In English we would say "Distance has kind *", and "Pair has kind * -> *". Kind signatures look similar to type signatures because they are. When we use Mass as Pair's first type argument, the result has kind *. The Haskell report defines kind signatures with the following two rules.

  • The symbol * represents the kind of all nullary type constructors (constructors that don't take any parameters).
  • If k1 and k2 are kinds, then k1->k2 is the kind of types that take one parameter that is a type of kind k1 and return a type of kind k2.

As an exercise, see if you can work out the kind signatures for the following type constructors. You can check your work with GHCi.

data Tuple a b = Tuple a b
data HardA a = HardA (a Int)
data HardB a b c = HardB (a b) (c a Int)

Before reading further, I suggest attempting to figure out the kind signatures for HardA and HardB because they involve a key pattern that will come up later.


Welcome back. The first example is just a simple extension of what we've already seen. The type constructor Tuple has two arguments, so it's kind signature is Tuple :: * -> * -> *. Also if you try :t you'll see that the data constructor's type signature is Tuple :: a -> b -> Tuple a b.

In the case of the last two types, it may be a little less obvious. But they build on each other in fairly small, manageable steps. For HardA, in the part to the left of the equals sign we see that there is one type parameter 'a'. From this, we can deduce that HardA's kind signature is something of the form ? -> *, but we don't know exactly what to put at the question mark. On the right side, all the individual arguments to the data constructor must have kind *. If (a Int) :: *, then the type 'a' must be a type constructor that takes one parameter. That is, it has kind * -> *, which is what we must substitute for the question mark. Therefore, we get the final kind signature HardA :: (* -> *) -> *.

HardB is a very contrived and much more complex case that exercises all the above simple principles. From HardB a b c we see that HardB has three type parameters, so it's kind signature has the form HardB :: a -> b -> c -> *. On the right side the (a b) tells us that b :: * and a :: * -> *. The second part (c a Int) means that c is a type constructor with two parameters where its first parameter is a, which has the type signature we described above. So this gives us c :: (* -> *) -> * -> *. Now, substituting all these in, we get HardB :: (* -> *) -> * -> ((* -> *) -> * -> *) -> *.

The point of all this is to show that when you see juxtaposition of type constructors (something of the form (a b) in a type signature), it is telling you that the context a is a non-nullary type constructor and b is its first parameter.

Continue on to Part 2 of the Less Travelled Monad Tutorial


by mightybyte (noreply@blogger.com) at April 17, 2012 10:49 PM

April 14, 2012

Alessandro Vermeulen

Software and Building Architecture Compared

People tend to only understand what they can see. For most people it is difficult to grasp more abstract matters without somehow visualizing them. Software is an example of such an abstract matter. Let us visit the process of developing software through a comparison with developing a building.

Everyone knows that in order to build something you will have to think about it first. Just putting some sticks together as you did when you were playing outside does not work very well for a skyscraper. For any construction you will need an architecture to design it first. After your architect has designed the project is has to go through some checks as to whether you will have enough sunlight in the house, enough ventilation and perhaps most importantly that the structure will not come down on itself! Ideally you only start building when both the architect, the contractor and yourself are satisfied that the building meets all requirements. Our knowledge on the construction of buildings has been growing since we first exited our caves and thus we generally can found our decisions during each step of the process on this knowledge.

Most people have not had the experience of just building some software that eventually crashes because they did not think it through, it is not something you do as a kid when playing outside. This makes things a bit more complicated to imagine, but the development of software is quite analogous to the development of some construction.

Coming up with a program is even more complicated than building a construction. It comes with all the issues mentioned above when creating a building. It first has to be designed, then it has to be passed along to someone leading the developers to check for practicality and ideally usability. After that it can be passed to the developers for the writing proper. The issue at hand being, however, is that in the world of software wishes from the client and the environment change continuously during the project. This causes an oscillation between the stages mentioned above creating long delays before the final product is finished. To make matters worse, the final product will not be the perfect fit to the requirements due to the delays.

Why does this happen, you wonder? There are several reasons. For starters the art of programming is still very young, learning from our mistakes. Even the Romans had some mishaps when building their famous aqueducts. This entails that we still lack the experience to found our design decisions on. Another reason is that demands and wishes change rapidly in our new hyper-dynamic society. When you set off on constructing a building you have a very good idea of why you want to do that, what it will take and what your usage of the building will be. Software projects are generally given less consideration.

A very contributing factor to the complexity of creating a software product is the fact that most cases the software has to integrate ‘perfectly’ in the physical process of the client, where most do not fully grasp their own process, and that it has to communicate with other software products. These other products might not be designed with the new scenario in mind and thus either need adjustment or another way around it has to be found.

So, this is why coming up with a software product is just as difficult as constructing a building, and is in most cases even harder. The next time someone asks why it takes so long to write (good) software you just ask how long it took to design his custom made house. :)

April 14, 2012 12:05 PM

April 11, 2012

Tom Moertel

The inner beauty of tree traversals (in Haskell)

This has been done a million times before, but if you haven’t seen it, it’s pretty neat. Let’s say you have a simple recursive data structure like a binary tree:

module Tree where

data Tree a = Empty
            | Node a (Tree a) (Tree a)
            deriving (Eq, Show)

-- some binary trees

t0 = Empty
t1 = Node 1 Empty Empty
t3 = Node 2 (Node 1 Empty Empty) (Node 3 Empty Empty)

Now say you want to traverse the nodes, in “preorder.” So you write a traversal function. It folds a binary operator f through the values in the tree, starting with an initial accumulator value of z. First, it visits the current node, then the left subtree, and finally the right subtree, combining each value it encounters with the current accumulator value to get the next accumulator value. The final accumulator value is the result.

Spelling out the steps, it looks like this:

preorder_traversal f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = let z'   = f v z     -- current node
                            z''  = go l z'   -- left subtree
                            z''' = go r z''  -- right subtree
                        in z'''

But if you wanted an “inorder” traversal instead, you’d write a slightly different function, visiting the left subtree before the current node:

inorder_traversal f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = let z'   = go l z    -- left subtree
                            z''  = f v z'    -- current node
                            z''' = go r z''  -- right subtree
                        in z'''

To see how the two traversal functions work, let’s use both to flatten the 3-node tree t3 we defined earlier.

flatten traversal = reverse . traversal (:) []

test0i = flatten inorder_traversal t3   -- [1,2,3]
test0p = flatten preorder_traversal t3  -- [2,1,3]

Great.

But now you start thinking about post-order traversal. Do you want to write a third traversal function that’s almost the same as the first two?

So you go back to the inorder traversal and start staring real hard at that let expression. Can you rewrite it? Yes:

inorder_traversal2 f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = go r . f v . go l $ z

And now you’ve got it. Just by reordering the applications of (go l), (f v), and (go r), you can control the traversal.

Which leads to a general traversal function that lets some other function control that ordering:

traverse step f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = step (f v) (go l) (go r) z

Doesn’t that look beautiful?

Now you can just spell out your desired traversals:

preorder   = traverse $ \n l r -> r . l . n
inorder    = traverse $ \n l r -> r . n . l
postorder  = traverse $ \n l r -> n . r . l

A test drive:

test1p = flatten preorder t3   -- [2,1,3]
test1i = flatten inorder t3    -- [1,2,3]
test1o = flatten postorder t3  -- [1,3,2]

And you’re happy.

But can you go further? What if your trees are binary search trees and you just want to find the minimum or maximum value? Can you just traverse to the left or right?

Yes:

leftorder  = traverse $ \n l r -> l . n
rightorder = traverse $ \n l r -> r . n

treemin = leftorder min maxBound
treemax = rightorder max minBound

test2l = treemin t3 :: Int  -- 1
test2r = treemax t3 :: Int  -- 3

Isn’t that neat?

by Tom Moertel at April 11, 2012 03:02 PM

Mark Jason Dominus

My Git Habits

Miles Gould asked his Twitter followers whether they used git-add -p or git-commit -a and how often. My reply was too long for Twitter, so here it is.

First the short version: I use git-add -p frequently, and git-commit -a almost never. The exception is when I'm working on the repo that holds my blog, where I rarely commit changes to more than one or two files at a time. Then I'll usually just git-commit -a -m ....

But I use git-add -p all the time. Typically what will happen is that I will be developing some fairly complicated feature. It will necessitate a bunch of changes and reshuffling elsewhere in the system. I'll make commits on the topic branch as I go along without worrying too much about whether the commits are neatly packaged.

Often I'll be in the middle of something, with a dirty work tree, when it's time to leave for the day. Then I'll just commit everything with the subject WIP ("work-in-progress"). First thing the next morning I'll git-reset HEAD^ and continue where I left off.

So the model is that the current head is usually a terrible mess, accumulating changes as it moves forward in time. When I'm done, I will merge the topic into master and run the tests.

If they pass, I am not finished. The merge I just created is only a draft merge. The topic branch is often full of all sorts of garbage, commits where I tried one approach, found it didn't work later on, and then tried a different approach, places where I committed debugging code, and so on. So it is now time to clean up the topic branch. Only the cleaned-up topic branch gets published.

Cleaning up messy topic branches

The core of the cleanup procedure is to reset the head back to the last place that look good, possibly all the way back to the merge-base if that is not too long ago. This brings all the topic changes into the working directory. Then:

  1. Compose the commits: Repeat until the working tree is clean:
    1. Eyeball the output of git-diff
    2. Think of an idea for an intelligible commit
    3. Use git-add -p to stage the planned commit
    4. Use git diff --cached to make sure it makes sense
    5. Commit it
  2. Order the commits: Use git-rebase --interactive
Notice that this separates the work of composing the commits from the work of ordering them. This is more important than it might appear. It would be extremely difficult to try to do these at the same time. I can't know the sensible order for the commits until I know what the commits are! But it's very hard to know what the commits are without actually making them.

By separating these tasks, I can proceed something like this: I eyeball the diff, and the first thing I see is something about the penguin feature. I can immediately say "Great, I'll make up a commit of all the stuff related to the penguin feature", and proceed to the git-add -p step without worrying that there might be other stuff that should precede the penguin feature in the commit sequence. I can focus on just getting the penguin commit right without needing to think about any of the other changes.

When the time comes to put the commits in order, I can do it well because by then I have abstracted away all the details, and reduced each group of changes to a single atomic unit with a one-line description.

For the most complicated cases, I will print out the diffs, read them over, and mark them up in six colors of highlighter: code to throw away gets marked in orange; code that I suspect is erroneous is pink. I make many notes in pen to remind me how I want to divide up the changes into commits. When a commit occurs to me I'll jot a numbered commit message, and then mark all the related parts of the diff with that number. Once I have the commits planned, I'll reset the topic ref and then run through the procedure above, using git-add -p repeatedly to construct the commits I planned on paper. Since I know ahead of time what they are I might do them in the right order, but more likely I'll just do them in the order I thought of them and then reorder them at the end, as usual.

For simple cases I'll just do a series of git-rebase --interactive passes, pausing at any leftover WIP commits to run the loop above, reordering the commits to squash related commits together, and so on.

The very simplest cases of all require no cleanup, of course.

For example, here's my current topic branch, called c-domain, with the oldest commits at the top:

        055a2f7 correction to bulk consumer template
        d9630bd DomainActivator half of Pobox Domain consumer
        ebebb4a Add HasDomain role to provide ->domain reader for domain consumers
        ade6ac6 stubbed domain test
        e170e77 start templates for Pobox domain consumers
        067ca81 stubbed Domain::ThumbTwiddler
        685a3ee cost calculations for DomainActivator
        ec8b1cc test fixes; trivial domain test passes now
        845b1f2 rename InvoiceCharge::CreateDomain to ..::RegisterDomain
(e)     6083a97 add durations to Domain consumers and charges
        c64fda0 tests for Domain::Activator consumer
        41e4292 repeat activator tests for 1-year and 3-year durations
        7d68065 tests for activator's replacement
(d)     87f3b09 move days_in_year to Moonpig::Util
        3cd9f3b WIP
        e5063d4 add test for sent invoice in domain.t
        c8dbf41 WIP
        9e6ffa4 add missing MakesReplacement stuff
        fc13059 bring in Net::OpenSRS module
(c)     52c18fb OpenSRS interface
        893f16f notes about why domain queries might fail
(b)     f64361f rename "croak" method to "fail" to avoid conflicts
        4e500ec Domain::Activator initial_invoice_charge_pairs
(a)     3c5cdd4 WIP
3c5cdd4 (a) was the end-of-day state for yesterday; I made it and pushed it just before I dashed out the door to go home. Such commits rarely survive beyond the following morning, but if I didn't make them, I wouldn't be able to continue work from home if the mood took me to do that.

f64361f (b) is a prime candidate for later squashing. 5c218fb (c) introduced a module with a "croak" method. This turned out to be a stupid idea, because this conflicted with the croak function from Perl's Carp module, which we use everywhere. I needed to rename it. By then, the intervening commit already existed. I probably should have squashed these right away, but I didn't think of it at the time. No problem! Git means never having to say "If only I'd realized sooner."

Similarly, 6083a97 (e) added a days_in_year function that I later decided at 87f3b09 (d) should be in a utility module in a different repository. 87f3b09 will eventually be squashed into 6083a97 so that days_in_year never appears in this code at all.

I don't know what is in the WIP commits c8dbf41 or 3cd9f3b, for which I didn't invent commit messages. I don't know why those are left in the tree, but I can figure it out later.

An example cleanup

Now I'm going to clean up this branch. First I git-checkout -b cleanup c-domain so that if something goes awry I can start over completely fresh by doing git-reset --hard c-domain. That's probably superfluous in this case because origin/c-domain is also pointing to the same place, and origin is my private repo, but hey, branches are cheap.

The first order of business is to get rid of those WIP commits. I'll git-reset HEAD^ to bring 3c5cdd4 into the working directory, then use git-status to see how many changes there are:

         M lib/Pobox/Moonpig/Consumer/Domain/Activator.pm
         M lib/Pobox/Moonpig/Role/HasDomain.pm
         M lib/Pobox/Moonpig/TemplateSet.pm
        ?? bin/register_domains
         M t/consumer/domain.t
        ?? t/lib/MockOpenSRS.pm
(This is the output from git-status --short, for which I have an alias, git s. I use this probably 99 times as often as plain git-status.)

Not too bad, probably no need for a printout. The new bin/register-domains program can go in right away by itself:

        % git add bin
        % git commit -m 'new register_domains utility program'
Next I'll deal with that new mock object class in t/lib/MockOpenSRS.pm. I'll add that, then use git-add -p to add the related changes from the other files:

        % git add t/lib
        % git add -p
        ...
        % git s
        MM lib/Pobox/Moonpig/Consumer/Domain/Activator.pm
         M lib/Pobox/Moonpig/Role/HasDomain.pm
         M lib/Pobox/Moonpig/TemplateSet.pm
        A  t/lib/MockOpenSRS.pm
        MM t/consumer/domain.t
        % git ix
        ...
The git ix command at the end there is an alias for git diff --cached: it displays what's staged in the index. The output looks good, so I'll commit it:

        % git commit -m 'mock OpenSRS object; add tests'
Now I want to see if those tests actually pass. Maybe I forgot something!
        % git stash
        % make test
        ...
        OK
        % git stash pop
The git-stash command hides the unrelated changes from the test suite so that I can see if the tests I just put into t/consumer/domain.t work properly. They do, so I bring back the stashed changes and continue. If they didn't, I'd probably amend the last commit with git commit --amend and try again.

Continuing:

        % git diff
        ...
        % git add -p lib/Pobox/Moonpig/Role/HasDomain.pm
        ...
        % git commit -m 'Domains do not have explicit start dates'
        % git diff
        ...
        % git add -p
        ...
        % git commit --fixup :/mock
That last bit should have been part of the "mock OpenSRS object" commit, but I forgot it. So I make a fixup commit, which I'll merge into the main commit later on. A fixup commit is one whose subject begins with fixup!. Did you know that you can name a commit by writing :/text, and it names the most recent commit whose message contains that text?

It goes on like that for a while:

        % git diff
        ...
        % git add -p ...
        ...
        % git commit -m 'Activator consumer can generate special charges'
        % git diff
        ...
        % git checkout lib/Pobox/Moonpig/Role/HasDomain.pm
The only uncommitted change left in HasDomain.pm was a superfluous line, so I just threw it away.

        % git diff
        ...
        % git add -u
        % git commit -m 'separate templates for domain-registering and domain-renewing consumers'
By this time all the remaining changes belong in the same commit, so I use git-add -u to add them all at once. The working tree is now clean. The history is as I showed above, except that in place of the final WIP commit, I have:

        a3c0b92 new register_domains utility program
        53d704d mock OpenSRS object; add tests
        a24acd8 Domains do not have explicit start dates
        17a915d fixup! mock OpenSRS object; add tests
        86e472b Activator consumer can generate special charges
        5b2ad2b separate templates for domain-registering and domain-renewing consumers
(Again the oldest commit is first.) Now I'll get rid of that fixup!:

        % git rebase -i --autosquash HEAD~6
Because of --autosquash, the git-rebase menu is reordered so that the fixup commit is put just after the commit it fixes up, and its default action is 'fixup' instead of 'pick'. So I don't need to edit the rebase instructions at all. But I might as well take the opportunity to put the commits in the right order. The result is:

        a3c0b92 new register_domains utility program
        ea8dacd Domains do not have explicit start dates
        297366a separate templates for domain-registering and domain-renewing consumers
        4ef0e28 mock OpenSRS object; add tests
        c3ab1eb Activator consumer can generate special charges
I have two tools for dealing with cleaned-up branches like this one. One is git-vee, which compares two branches. It's just a wrapper around the command git log --decorate --cherry-mark --oneline --graph --boundary A"..."B.

Here's a comparison the original c-domain branch and my new cleanup version:

        % git vee c-domain
        * c3ab1eb (HEAD, cleanup) Activator consumer can generate special charges
        * 4ef0e28 mock OpenSRS object; add tests
        * 297366a separate templates for domain-registering and domain-renewing consumer
        * ea8dacd Domains do not have explicit start dates
        * a3c0b92 new register_domains utility program
        | * 3c5cdd4 (origin/c-domain, c-domain) WIP
        |/  
        o 4e500ec Domain::Activator initial_invoice_charge_pairs
This clearly shows where the original and cleaned up branches diverge, and what the differences are. I also use git-vee to compare pre- and post-rebase versions of branches (with git-vee ORIG_HEAD) and local branches with their remote tracking branches after fetching (with git-vee remote or just plain git-vee).

A cleaned-up branch should usually have the same final tree as the tree at the end of the original branch. I have another tool, git-treehash, which compares trees. By default it compares HEAD with ORIG_HEAD, so after I use git-rebase to squash or to split commits, I sometimes run "git treehash" to make sure that the tree hasn't changed. In this example, I do:

        % git treehash c-domain HEAD
        d360408d1afa90e0176aaa73bf8d3cae641a0850 HEAD
        f0fd6ea0de7dbe60520e2a69fbec210260370d78 c-domain
which tells me that they are not the same. Most often this happens because I threw away all the debugging code that I put in earlier, but this time it was because of that line of superfluous code I eliminated from HasDomain.pm. When the treehashes differ, I'll use git-diff to make sure that the difference is innocuous:

        % git diff c-domain
        diff --git a/lib/Pobox/Moonpig/Role/HasDomain.pm b/lib/Pobox/Moonpig/Role/HasDomain.pm
        index 3d8bb8c..21cb752 100644
        --- a/lib/Pobox/Moonpig/Role/HasDomain.pm
        +++ b/lib/Pobox/Moonpig/Role/HasDomain.pm
        @@ -5,7 +5,6 @@ use Carp qw(croak confess);
         use ICG::Handy qw(is_domain);
         use Moonpig::Types qw(Factory Time);
         use Moose::Util::TypeConstraints qw(duck_type enum subtype);
        -use MooseX::SetOnce;

         with (
           'Moonpig::Role::StubBuild',

Okay then.

The next task is probably to deal with the older WIP commits. This time I'll omit all the details. But the enclosing procedure looks like this:

        % git checkout -b wip-cleanup c8dbf41
        % git reset HEAD^
        % ... (a lot of git-add -p as above) ...
        ...

        % git vee c8dbf41
        * 4c6ff45 (wip-cleanup) get rid of unused twiddler test
        * b328de5 test full payment cycle
        * 201a4f2 abstract out pay_invoice operation
        * 55ae45e add upper limit (default 30d) to wait_until utility
        | * c8dbf41 WIP
        |/  
        o e5063d4 add test for sent invoice in domain.t

        % git treehash c8dbf41 HEAD
        7f52ba68923e2ede8fda407ffa9c06c5c48338ae
        % git checkout cleanup
        % git rebase wip-cleanup
The output of git-treehash says that the tree at the end of the wip-cleanup branch is identical to the one in the WIP commit it is supposed to replace, so it's perfectly safe to rebase the rest of the cleanup branch onto it, replacing the one WIP commit with the four new commits in wip-cleanup. Now the cleaned up branch looks like this:

        % git vee c-domain
        * a425aa1 (HEAD, cleanup) Activator consumer can generate special charges
        * 2bb0932 mock OpenSRS object; add tests
        * a77bfcb separate templates for domain-registering and domain-renewing consumer
        * 4c44db2 Domains do not have explicit start dates
        * fab500f new register_domains utility program
        = 38018b6 Domain::Activator initial_invoice_charge_pairs
        = aebbae6 rename "croak" method to "fail" to avoid conflicts
        = 45a224d notes about why domain queries might fail
        = 80e4a90 OpenSRS interface
        = 27f4562 bring in Net::OpenSRS module
        = f5cb624 add missing MakesReplacement stuff
        * 4c6ff45 (wip-cleanup) get rid of unused twiddler test
        * b328de5 test full payment cycle
        * 201a4f2 abstract out pay_invoice operation
        * 55ae45e add upper limit (default 30d) to wait_until utility
        | * 3c5cdd4 (origin/c-domain, c-domain) WIP
        | = 4e500ec Domain::Activator initial_invoice_charge_pairs
        | = f64361f rename "croak" method to "fail" to avoid conflicts
        | = 893f16f notes about why domain queries might fail
        | = 52c18fb OpenSRS interface
        | = fc13059 bring in Net::OpenSRS module
        | = 9e6ffa4 add missing MakesReplacement stuff
        | * c8dbf41 WIP
        |/  
        o e5063d4 add test for sent invoice in domain.t
git-vee marks a commit with an equal sign instead of a star if it's equivalent to a commit in the other branch. The commits in the middle marked with equals signs are the ones that weren't changed. The upper WIP was replaced with five commits, and the lower one with four.

I've been planning for a long time to write a tool to help me with breaking up WIP commits like this, and with branch cleanup in general: It will write each changed hunk into a file, and then let me separate the hunk files into several subdirectories, each of which represents one commit, and then it will create the commits automatically from the directory contents. This is still only partly finished, but I think when it's done it will eliminate the six-color diff printouts.

[ Addendum 20110404: Further observation has revealed that I almost never use git-commit -a, even when it would be quicker to do so. Instead, I almost always use git-add -u and then git-commit the resulting index. This is just an observation, and not a claim that my practice is either better or worse than using git-commit -a. ]

by Mark Dominus (mjd@plover.com) at April 11, 2012 01:32 PM

April 09, 2012

Magnus Therning

Manual setup of Qt+Eclipse on Windows

Before the weekend I started looking at using Qt on Windows. More specifically I wanted to know whether this combination could be an option for a sub-project at work. We need to develop a program for the Windows desktop, and due to the overall context it would make sense to write it in C++ (that’s what we use for another part of the project). We already use both Eclipse and Visual Studio in the project, but I strongly prefer Eclipse, so I was hoping to be able to use it. However, it seems that the Qt developers strongly favour their own tool Qt Creator, though there are (outdated?) integrators for both Eclipse and Visual Studio. I’d rather avoid introducing a third IDE into a project—two is already one too many in my opinion. Anyway, I think I managed to find an acceptable configuration of Eclipse without using that old Qt integration plugin together with the MSVC (I was using the gratis version of MSVC for this).

Qt setup

I decided to install Qt into C:\QtSDK, and then I made the following permanent changes to the environment:

> set QTDIR=C:\QtSDK\Desktop\Qt\4.8.0\msvc2010
> set QMAKESPEC=%QTDIR%\mkspecs\win32-msvc2010
> set PATH=%PATH%;%QTDIR%\bin;C:\QtSDK\QtCreator\bin

Starting Eclipse so that it finds the compiler

It’s slightly disappointing that Eclipse happily lets one create MSVC project that isn’t buildable because it doesn’t know where the compiler is located. One easy way to remedy that seems to create a BAT file to create the proper environment for Eclipse:

@echo off
setlocal
call "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat"
start C:\Eclipse\Indigo\eclipse.exe
endlocal

Creating the project

Creating a “makefile” project in Eclipse is fairly straight forward; one needs a C/C++ project, of the makefile type, and make it empty too so that there isn’t any cruft in the way. Then add a single source file, e.g. main.cxx:

#include <iostream>
#include <Qt/QtGui>
 
int main(int argc, char **argv)
{
    std::cout << __FUNCTION__ << std::endl;
    QApplication app(argc, argv);
    return(app.exec());
}

And then a project file, e.g. Test.pro:

TEMPLATE = app
TARGET = 
DEPENDPATH += .
INCLUDEPATH += .
 
CONFIG += qt
 
HEADERS +=
SOURCES += main.cxx

After this use qmake to create the required makefile. I decided to use a subdirectory (_build) in the project, which qmake seems to have full support for:

> qmake ..\Test.pro

Setting up building from Eclipse

In the project properties modify the C/C++ Build settings for the Debug target. Instead of the default build command (which is make) one can use nmake, or even better jom:

  • Build command: C:/QtSDK/QTCreator/bin/jom -f Makefile.Debug
  • Build directory: ${workspace_loc:/Test}/_build

Then one can create a Release target, which differs only in that it builds using Makefile.Release.

Running qmake from inside Eclipse

It’s very convenient to be able to run qmake and re-generate the makefiles from inside Eclipse. One can set that up by adding an external tool:

  • Location: C:\QtSDK\Desktop\Qt\4.8.0\msvc2010\bin\qmake.exe
  • Working directory: ${workspace_loc:/Test}/_build
  • Arguments: ../Test.pro

In closing

I plan to also have a look at the Qt Visual Studio Add-in, though I suspect we might be using the latest version of VS, which might cause trouble.

Suggestions for further integration with Eclipse would be most welcome, e.g. for forms and translations.

Share/Bookmark

by Magnus at April 09, 2012 08:24 PM

Bryan O'Sullivan

(re)announcing statprof, a statistical profiler for Python

Back in 2005, Andy Wingo wrote a neat little statistical profiler named statprof that promptly disappeared into obscurity. It has since languished almost unknown, with a handful of people writing semi-private forks that themselves seem to be dead.

Statistical profiling (also known as sampling profiling) is simple and sweet: the profiler periodically wakes up and samples the stack, then when all is done, it prints a simple report of which lines showed up most often in the profile.

Why would this matter, though? Python already has two built-in profilers: lsprof and the long-deprecated hotshot. The trouble with lsprof is that it only tracks function calls. If you have a few hot loops within a function, lsprof is nearly worthless for figuring out which ones are actually important.

A few days ago, I found myself in exactly the situation in which lsprof fails: it was telling me that I had a hot function, but the function was unfamiliar to me, and long enough that it wasn’t immediately obvious where the problem was.

After a bit of begging on Twitter and Google+, someone pointed me at statprof. But there was a problem: although it was doing statistical sampling (yay!), it was only tracking the first line of a function when sampling (wtf!?). So I fixed that, spiffed up the documentation, and now it’s both usable and not misleading. Here’s an example of its output, locating the offending line in that hot function more accurately:

  %   cumulative      self          
 time    seconds   seconds  name    
 68.75      0.14      0.14  scmutil.py:546:revrange
  6.25      0.01      0.01  cmdutil.py:1006:walkchangerevs
  6.25      0.01      0.01  revlog.py:241:__init__
  [...blah blah blah...]
  0.00      0.01      0.00  util.py:237:__get__
---
Sample count: 16
Total time: 0.200000 seconds

I have uploaded statprof to the Python package index, so it’s almost trivial to install: “easy_install statprof” and you’re up and running.

Since the code is up on github, please feel welcome to contribute bug reports and improvements. Enjoy!

by Bryan O'Sullivan at April 09, 2012 06:51 PM

Luke Palmer

Heart

It is part of growing up, I keep telling myself — doing what I know — for some definition of know — is right, despite the advice of my family and almost everyone (but my best friend who is my only beacon in this whole mess). I have a good family — supportive, have my best interest in mind, certainly not the image of the disapproving father so pervasive — and partially I haven’t been completely honest with them, because it’s scary. Nonetheless, I feel a lot of pressure from their attempting-to-be-neutral positions, and I know what I want — what I need to do, but when the time comes to say it I can’t, condemning myself to this purgatory.

I’m not going to finish college. I am very close, only a few credits away, but it is not going to happen at the end of this semester, and everyone is like “but it’s just one more and it’s important for the future” — not so different from my reasoning for returning to college in the first place — I have been at this decision point before, and did convince myself with the assistance of my family that it was the right thing to do. Maybe it was once, and although I did not achieve the goals I set for it, it isn’t right anymore.

Here’s the really hard part, and I have to speak this with less certainty than the other, because different parts of my mind and body are fighting over it. I don’t think I’m going to finish this semester. Try as I might (whatever that means) I cannot commit myself to something that I don’t truly believe is serving me, and right now that is school. I don’t have that kind of control over myself. My grades are really slipping; each moment here feels like trying to run in a dream, suspended in the air. I know, what’s another month? It really doesn’t matter either way. It would matter if I wanted to go to grad school, but years of getting to know myself and being friends with grad students, I don’t think it is the place for me. I am too disorganized, my intellectual exploration is founded in too much curiosity and not enough desire to contribute. Suddenly a pursuit will become uninteresting and another will take me by surprise, but you can’t just switch like that in school.

But you can just switch like that in life. Why would I arbitrarily obligate myself to someone else when I am exploring what I love? Out there in the cruel, forgiving, free world, I can pursue whatever I like whenever and however I like to. Yes, I need to make money, but that’s not such a huge deal. I don’t really get why people make their way of earning money the centerpiece of their lives. Insert canonical white-picket-fence rant.

I don’t have a good phrase to describe who I want to be or what I’m going for. I think of such phrases as potentially guiding, locally, but ultimately limiting. To define myself with words is to forget every moment the words do not account for — when would someone include the Pepsi they had for breakfast in their self-definition? — but that bottle of salt and sugar is part of me, negative or however you want to judge it. Of course, not having such a phrase makes it difficult to assess the value of a difficult easy decision like this, and without a mechanism for assessing value I have no choice but to be human and follow my hearts — there’s nothing else that I can say with my vocabulary that doesn’t sound like a waste of my life.

I have long valued every moment of my time. A year of my life spent unhappy in order to support the remainder of my life never seemed worthwhile to me — I know that sounds irrational — but that seems to be the way I relate to time. A month spent in school, a month not making my living by sharing my musical heart, a month depressed and careless, a month of missed opportunity.

And yet, it is only a month. But why would I stay? I can’t articulate any convincing reason. It will make it less work should I ever decide to come back and finish — but that is actually false. One class is just as many as four, if not more.

I have had my struggles, but at important times I have always listened to the guidance of my family, I think I have always made what they saw as the best choice. This time, I think, their poor choice is the right one — if only symbolically, if only to remind myself whose life I am living.

Flattr this


by Luke at April 09, 2012 05:16 PM

Edwin Brady

New draft: Implementation of a General Purpose Programming Language with Dependent Types

I’ve written a draft paper describing the implementation of Idris,
Implementation of a General Purpose Programming Language with Dependent Types, abstract as follows:

Many components of a dependently typed programming language are by now well understood, for example the underlying type theory, type checking, unification and evaluation. How to combine these components into a realistic and usable high level language is, however, folklore, discovered anew by succesive language implementations. In this paper, I describe the implementation of a new dependently typed functional programming language, Idris. Idris is intended to be a general purpose programming language and as such provides high level concepts such as implicit syntax, type classes and do notation. I describe the high level language and the underlying type theory, and present a method for elaborating concrete high level syntax with implicit arguments and type classes into a fully explicit type theory. Furthermore, I show how this method, based on a domain specific language embedded in Haskell, facilitates the implementation of new high level language constructs.

There’s still some polishing to do, but comments on the content, form, and so on, would be most welcome. Enjoy!


by edwinb at April 09, 2012 11:11 AM