Planet Haskell

January 27, 2012

Tom Moertel

The inner beauty of tree traversals

This has been done a million times before, but if you haven’t seen it, it’s pretty neat. Let’s say you have a simple recursive data structure like a binary tree:

module Tree where

data Tree a = Empty
            | Node a (Tree a) (Tree a)
            deriving (Eq, Show)

-- some binary trees

t0 = Empty
t1 = Node 1 Empty Empty
t3 = Node 2 (Node 1 Empty Empty) (Node 3 Empty Empty)

Now say you want to traverse the nodes, in “preorder.” So you write a traversal function. It folds a binary operator f through the values in the tree, starting with an initial accumulator value of z. First, it visits the current node, then the left subtree, and finally the right subtree, combining each value it encounters with the current accumulator value to get the next accumulator value. The final accumulator value is the result.

Spelling out the steps, it looks like this:

preorder_traversal f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = let z'   = f v z     -- current node
                            z''  = go l z'   -- left subtree
                            z''' = go r z''  -- right subtree
                        in z'''

But if you wanted an “inorder” traversal instead, you’d write a slightly different function, visiting the left subtree before the current node:

inorder_traversal f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = let z'   = go l z    -- left subtree
                            z''  = f v z'    -- current node
                            z''' = go r z''  -- right subtree
                        in z'''

To see how the two traversal functions work, let’s use both to flatten the 3-node tree t3 we defined earlier.

flatten traversal = reverse . traversal (:) []

test0i = flatten inorder_traversal t3   -- [1,2,3]
test0p = flatten preorder_traversal t3  -- [2,1,3]

Great.

But now you start thinking about post-order traversal. Do you want to write a third traversal function that’s almost the same as the first two?

So you start staring real hard at that let statement. Can you rewrite it? Yes:

inorder_traversal2 f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = go r . f v . go l $ z

And now you’ve got it. Just by reordering the applications of (go l), (f v), and (go r), you can control the traversal.

Which leads to a general traversal function that lets some other function control that ordering:

traverse step f z tree = go tree z
  where
    go Empty        z = z
    go (Node v l r) z = step (f v) (go l) (go r) z

Doesn’t that look beautiful?

Now you can just spell out your desired traversals:

preorder   = traverse $ \n l r -> r . l . n
inorder    = traverse $ \n l r -> r . n . l
postorder  = traverse $ \n l r -> n . r . l

A test drive:

test1p = flatten preorder t3   -- [2,1,3]
test1i = flatten inorder t3    -- [1,2,3]
test1o = flatten postorder t3  -- [1,3,2]

And you’re happy.

But can you go further? What if your trees are binary search trees and you just want to find the minimum or maximum value? Can you just traverse to the left or right?

Yes:

leftorder  = traverse $ \n l r -> l . n
rightorder = traverse $ \n l r -> r . n

treemin = leftorder min maxBound
treemax = rightorder max minBound

test2l = treemin t3 :: Int  -- 1
test2r = treemax t3 :: Int  -- 3

Isn’t that neat?

by Tom Moertel at January 27, 2012 04:48 AM

January 26, 2012

Alessandro Vermeulen

Getting rid of programming JavaScript with Haskell

For my Experimentation Project at Utrecht University I ported the “JCU” application to Haskell. The JCU application is used to give Dutch High school students the opportunity to taste Prolog.

The project uses the Utrecht Haskell Compiler and its JavaScript backend. The UHC translates Haskell to Core and then translates this Core language to JavaScript. For more information on this see the blog of the creator of the UHC JavaScript backend.

Please read my report on this project. The project is hosted on GitHub in the following repositories:

Incomplete proof in the HS JCU App Complete proof in the HS JCU App

January 26, 2012 07:29 PM

Manuel M T Chakravarty

Free, nicely presented textbooks with good distribution

Free, nicely presented textbooks with good distribution have got quite an appeal.

Posted via email from Just Testing | Comment »

January 26, 2012 01:42 PM

Luke Palmer

What was the probability of that?

A group of college students bask in their crude weekly haven of Texas Hold’em poker. Amid the usual banter, they watch as one deals out the flop: a three of spades, a three of diamonds, and a three of clubs. The room falls silent and tensions are suspended as if the ground had fallen out from beneath the house. The highest card may win this hand, a pocket pair is almost golden, and everyone wonders if someone has the single remaining three. A round of timid betting ensues — a wrong step here could be very expensive. After the round completes, the fourth card is dealt onto the table: the three of hearts. The tensions are dropped as everyone laughs incredulously, and out of the laughter emerges “what’s the probability of that?”

One of the more mathematically adept players pulls out his phone and does a quick calculation: \frac{4}{52}\frac{3}{51}\frac{2}{50}\frac{1}{49} — about 1 in 270,000. Everyone is wowed by the rarity of the event they just witnessed.

That is indeed the correct probability to get four consecutive threes from a deck of cards. But is that really the question here? Surely nearly the same response would have occurred if it had been four fours, or four nines. If it were four aces people would be even more astonished. The same response would have occurred if the cards had been four to a straight flush; e.g. the five through eight of spades. There are many such situations. “Four threes” is the most immediate pattern that we recognize as anomalous, but it is not the only anomalous pattern.

So what event is really being referred to by that? Those specific four cards had a 1 in 6.5 million chance of coming up in the order they did from a player’s perspective before the hand, and a 100% chance of coming up in the order they did from a perspective after the hand [some will note that I am using a Bayesian interpretation of probability at this point]. The probability of the specific real world event that occurred (including the orientations and temperatures of the cards, the reactions of the players, and the taste of Jake Danser’s bite of Chinese dinner 500 miles away), from the perspective of any of the players, is unbelievably small.

In situations where this question is asked, I always jokingly answer the question with “1″. Most of the time people laugh and let the question rest, but sometimes the conversation turns more serious. In this case I try (in all cases so far, in vain) to explain the plurality of perspective at play here. The “serious” answer I have for this question, doing my best to interpret the intended meaning of the statement while incorporating these issues, is a number I cannot calculate but that I can describe: it is the probability that the speaker would have been astonished by the event that occurred; essentially the probability that he would remark “what’s the probability of that?”. I think that is quite a bit higher than one in 270,000, so the event we witnessed was not as rare as the simple calculation would have us believe.

The dissonance of such situations points to a common simplistic view of probability: that events (in the colloquial sense) have probabilities. A distinction that is commonly understood (and commonly misunderstood) is that between frequentism, which talks about running an experiment a bunch of times and calculating the proportion of experiments that satisfied some criterion, and Bayesianism, which talks about the confidence one has in some property being true in terms of a subjective state of knowledge. This is a fascinatingly subtle distinction: they coincide on most but not all questions, and there is much argument about which one is “right” (I think that is a rather silly argument, as if a probability were something bestowed to us from nature). However, both of these views are calculating the probability of a property (a set of events) being true (happening), and both of these views rely on an assumption of prior information: the frequentists on the set-up of the experiment and the Bayesians on their prior state of knowledge about the world. The idea that an event has a single, objective probability says something very strong about the universe: that there is some essential, natural source of randomness (to dispel any quantum objections, I point to E.T. Jaynes’s critique of the view that the randomness in quantum theory is an essential property of nature). But even if there were some essential source of randomness, surely the universe is more determined than our probabilistic calculations assume: from the view of this universe, the deck cannot be uniformly shuffled after only seven shuffles because it knows how your hands and your brain work. So we were never talking about an essential, natural randomness.

In our normal use of probabilities, we don’t run into this problem because we are predicting the future: e.g. “what is the probability that Obama will be re-elected?”. We conceive of this as a single event, but it is actually a wide collection of events. Given some prior knowledge, this set of events has a definite probability. But we are inconsistent about how we treat future events and past events: past events are not collections — they happened, and they happened in exactly one way. From the perspective of now, the probability that it would happen any other way is zero. From the perspective of before the event, we are asking whether “it” would happen or not, but we are entirely unclear about what measurable property we mean by “it”. We mean something different when we refer to events in the future and events in the past.

In summary, probabilities are functions of (1) prior information and (2) a criterion for judging when it is to be considered true. Probability is meaningless to apply to a single event.

N.B. in probability theory, the word event is already defined to mean a set of outcomes. If you read this article with that definition in mind, you will have been very confused :-).

If this post helped you think more clearly about probability, show your appreciation and Flattr this.


by Luke at January 26, 2012 05:42 AM

Computably Uncountable

We are all familiar with Cantor’s diagonal argument that proves there exist infinite sets which are “larger” than the set of natural numbers. In this post I will show that we can express this argument in the form of a program, thus showing that there are countable sets which are “computably uncountable”.

I begin with the program itself:

type Cantor = Nat -> Bool

diagonal :: (Nat -> Cantor) -> Cantor
diagonal cs n = not (cs n n)

Cantor is “the cantor space”, the type of infinite sequences of booleans. We will call such an infinite sequence “a Cantor“. There are clearly infinitely many Cantors; e.g. take the range of this function which gives False at every position except the one specified:

unit :: Nat -> Cantor
unit m n = m == n

diagonal is (Georg) Cantor’s diagonal argument written as a program — it takes an alleged sequence of all Cantors, and returns a Cantor which does not occur in the sequence, by construction. This function shows by contradiction that we cannot put Cantors in 1 to 1 correspondence with naturals, and thus that there are more Cantors than there are naturals.

So how many Cantors are there? Since Nat -> Bool is a Haskell type — the type of computable functions from Nat to BoolCantors must be representable by programs. We can encode programs as numbers by treating their source code as base-128 numbers. Hence, there are no more Cantors than naturals, and so Cantors can be put into 1 to 1 correspondence with naturals.

Wait — what? There are more Cantors than Nats, but they both have the same size? Something is wrong. Indeed, in the process of this argument we have asserted both

  1. “We cannot put Cantors in 1 to 1 correspondence with naturals”
  2. Cantors can be put into 1 to 1 correspondence with naturals”

We clearly can’t have both.

I

The erroneous statement is (2). It is undecidable whether a given program represents a Cantor. If the nth Cantor is ⊥ at n, then diagonal will fail: diagonal cs n = not (cs n n) = not ⊥ = ⊥. Because ⊥ is a fixed point of not, diagonal cannot return an element different from the one it was given. Thus for diagonal to work, we must require that Cantors be fully-defined — no infinite loops!

With this requirement, we can no longer put Cantors in 1 to 1 correspondence with the naturals, because we would have to solve the halting problem. It is not enough that the type of the term is a Cantor, it now must be fully defined for all inputs, and determining that given arbitrary source code is an undecidable problem.

II

The erroneous statement is (1). Cantors are computable functions, so as we have argued, they have the same cardinality as the naturals. There are no more programs than numbers, so by the definition of equal cardinality we can put them in 1 to 1 correspondence with a function.

The problem with (1) occurs because diagonal takes as its first argument not an arbitrary sequence of Cantors, but a computable sequence of Cantors. If cs is not computable, then neither is diagonal cs (for we no longer have cs‘s source code with which to construct it), and Cantors are defined to be computable sequences. So diagonal fails to contradict our bijection.

III

The erroneous statement is (2). Section II claims to put Cantors and naturals in 1 to 1 correspondence, but it is lying. Section II is formulated with respect to some axiom system A, which is something like ZF. If it were “telling the truth”, we would expect there to be some term f in the language of A such that for every fully defined Cantor program c, there is some natural number n such that we have A \vdash f(\bar{n}) = \bar{c} (i.e. it is a theorem of A that f(1 + 1 + … + 1) = (source code of c)).

Let’s suppose we have written down the axioms of A into a Haskell program, and we have a (partial) function proofSearch :: Nat -> Cantor, which, given a number n, searches for theorems of the form f(\bar{n}) = \bar{c} and compiles and returns the first such c it finds. In the case that there is no such statement, it just runs forever; similarly for the case that c fails to compile. Although cumbersome to write, I’m sure we agree that this is possible to write. If section II is not lying, then we expect that for every natural n, proofSearch n does in fact return a valid Cantor.

Now, let us return to familiar lands with a new program:

evidence :: Cantor
evidence = diagonal proofSearch

Oh my! If section II is the truth, then proofSearch is a total, computable function of type Nat -> Cantor, which we can pass to diagonal to find a Cantor that it missed! So it must have been lying, either (1) about its function f finding every possible Cantor or (2) about it actually possessing such a function (i.e. it “proved” that there is such a function, but it couldn’t actually represent it). In either case, it did not actually create a 1 to 1 correspondence between the naturals and Cantors.

IV

Left as an exercise for the reader.


Which one is it really?

Flattr this


by Luke at January 26, 2012 05:33 AM

January 25, 2012

Bjorn Buckwalter

Type level integers with Type Families (numtype-tf 0.1)

Pretty much since they were introduced into GHC I’ve been meaning to try to port the dimensional library to use Type Families instead of Functional Dependencies. However, functional dependencies have served me quite well and I’ve been lacking the inspiration to dig into type families. That is, until I read the Leonidas’ Statically Typed Vector Algebra Using Type Families. Leonidas’ blog post resonated with me as the problem it addresses is related to one I myself have been interested in for some time (linear algebra with static guarantees). It also gave me a clear vision of how to apply type families to the fundamendal problem of dimensional – type level integers.

I’ve implemented the numtype-tf library which is in essence numtype using type families (hence the “-tf”) instead of functional dependencies. Some of my design decisions are slightly different (beside the choice of type system extension), e.g. the representation of negative integers is different and the PosType and NegType classes have been dropped in numtype-tf. The haddocks have also been improved a little.

A feature of the functionally dependent numtype that I was unable to reproduce with type families is ”mutual dependencies”, used in the Sum and Div type classes of numtype. Here is an example of type signatures (instances omitted) and ghci usage:

> class Sum a b c | a b -> c, a c -> b, b c -> a
> (+) :: (Sum a b c) => a -> b -> c
> (-) :: (Sum a b c) => c -> b -> a
\*Numeric.NumType> pos3 - pos5
NumType -2

To the best of my knowledge corresponding code with type families would be along the lines of:

> type family Sum a b
> (+) :: a -> b -> Sum a b
> (-) :: Sum a b -> b -> a
\*Numeric.NumType.TF> pos3 - pos5

<interactive>:1:1:
    Couldn't match type `Sum a0 Pos5' with `S Pos2'
    …

In order for the latter to compute we must assist the type checker by being explicit about the type of a0, i.e. pos3 - pos5 :: Neg2, which largely defeats the purpose of type level arithmetic. Instead two type families are used in numtype-tf:

> type family Add a b  -- a + b
> (+) :: a -> b -> Add a b
> type family Sub a b -- a - b.
> (-) :: a -> b -> Sub a b

This works OK but the lack och mutual dependencies in type families restricts what I will losely refer to as “type inference back tracking”. This may or may not be a significant drawback compared to fundep numtype depending on the use case.

If you are interested in comparing code written with type families to code written with functional dependencies take a look at the respective modules on Github: Numeric.NumType.TF vs. Numeric.NumType.

The next step is to reimplement dimensional as dimensional-tf in order to exercise numtype-tf and better understand the impact of the differences with respect to numtype. I am also looking forward to playing with GHC.TypeNats in GHC 7.4.

Flattr this

by Björn Buckwalter (noreply@blogger.com) at January 25, 2012 01:44 PM

Yesod Web Framework

Persistent 0.7, Yesod 0.10 beta

I'm very happy to announce the newest release of Persistent. Version 0.7 is now on Hackage, and supports three major changes:

  • The typeclass hierarchy has been split up into PersistStore and PersistQuery. This means that it should be much easier to add backends for data stores like Redis. (Thanks to Greg Weber for this.)
  • There's a much more convenient raw SQL interface with automated data marshaling. (Thanks to Felipe Lessa for this.)
  • The EntityDef and Template Haskell code has been drastically cleaned up and simplified. This makes it easier to add new features as separate TH components, and fixes a number of bugs related to automated migrations with field renames. (I'll take credit here.)

Persistent 0.7 will be used in the upcoming Yesod 0.10... Speaking of which, Yesod 0.10 is now officially in beta! I strongly encourage all users to give this a test drive. Greg has put together a 0.10 upgrade guide, and we'll likely have a blog post giving the typical "here's how I upgraded Haskellers.com."

We'll go into the exciting new features of Yesod 0.10 in the actual release announcement, but for testing purposes, there hasn't been a significant number of changes. This release was mostly focused on API cleanup. Remember: the goal is to turn Yesod 0.10 into 1.0!

The only feature we still plan to add before the official release is merging in Nubis's test branch. This is a really nice set of features, not only adding easy, high-level testing to the scaffolding, but in general making the scaffolding a more user-friendly site. Thank you Nubis!

In order to install the beta, you can either install from Github or from the Yesod Yackage site. To install from Github:

  1. cabal update
  2. git clone https://github.com/yesodweb/yesod
  3. cd yesod
  4. git submodule update --init
  5. ./scripts/install

To install from Yackage, add the following line to your ~/.cabal/config file:

remote-repo: yesodweb-yackage:http://yackage.yesodweb.com/

And then run cabal update && cabal install yesod.

January 25, 2012 07:55 AM

January 24, 2012

Edward Z. Yang

Modelling IO: MonadIO and beyond

The MonadIO problem is, at the surface, a simple one: we would like to take some function signature that contains IO, and replace all instances of IO with some other IO-backed monad m. The MonadIO typeclass itself allows us to transform a value of form IO a to m a (and, by composition, any function with an IO a as the result). This interface is uncontroversial and quite flexible; it’s been in the bootstrap libraries ever since it was created in 2001 (originally in base, though it migrated to transformers later). However, it was soon discovered that when there were many functions with forms like IO a -> IO a, which we wanted to convert into m a -> m a; MonadIO had no provision for handling arguments in the negative position of functions. This was particularly troublesome in the case of exception handling, where these higher-order functions were primitive. Thus, the community began searching for a new type class which captured more of IO.

While the semantics of lift were well understood (by the transformer laws), it wasn’t clear what a more powerful mechanism looked like. So, early attacks at the problem took the approach of picking a few distinguished functions which we wanted, placing them in a typeclass, and manually implementing lifted versions of them. This lead to the development of the already existing MonadError class into a more specialized MonadCatchIO class. However, Anders Kaseorg realized that there was a common pattern to the implementation of the lifted versions of these functions, which he factored out into the MonadMorphIO class. This approach was refined into the MonadPeelIO and MonadTransControlIO typeclasses. However, only MonadError was in the core, and it had failed to achieve widespread acceptance due to some fundamental problems.

I believe it is important and desirable for the community of library writers to converge on one of these type classes, for the primary reason that it is important for them to implement exception handling properly, a task which is impossible to do if you want to export an interface that requires only MonadIO. I fully expected monad-control to be the “winner”, being the end at a long lineage of type classes. However, I think it would be more accurate to describe MonadError and MonadCatchIO as one school of thought, and MonadMorphIO, MOnadPeelIO and MonadTransControlIO as another.

In this blog post, I’d like to examine and contrast these two schools of thought. A type class is an interface: it defines operations that some object supports, as well as laws that this object abides by. The utility in a type class is both in its generality (the ability to support multiple implementations with one interface) as well as its precision (the restriction on permissible implementations by laws, making it easier to reason about code that uses an interface). This is the essential tension: and these two schools have very different conclusions about how it should be resolved.

Modelling exceptions

This general technique can be described as picking a few functions to generalize in a type class. Since a type class with less functions is preferable to one with more (for generality reasons), MonadError and MonadCatchIO have a very particular emphasis on exceptions:

class (Monad m) => MonadError e m | m -> e where
  throwError :: e -> m a
  catchError :: m a -> (e -> m a) -> m a

class MonadIO m => MonadCatchIO m where
  catch   :: Exception e => m a -> (e -> m a) -> m a
  block   :: m a -> m a
  unblock :: m a -> m a

Unfortunately, these functions are marred by some problems:

  • MonadError encapsulates an abstract notion of errors which does not necessarily include asynchronous exceptions. That is to say, catchError undefined h will not necessarily run the exception handler h.
  • MonadError is inadequate for robust handling of asynchronous exceptions, because it does not contain an interface for mask; this makes it difficult to write bracketing functions robustly.
  • MonadCatchIO explicitly only handles asynchronous exceptions, which means any pure error handling is not handled by it. This is the “finalizers are sometimes skipped” problem.
  • MonadCatchIO, via the MonadIO constraint, requires the API to support lifting arbitrary IO actions to the monad (whereas a monad designer may create a restricted IO backed monad, limiting what IO actions the user has access to.)
  • MonadCatchIO exports the outdated block and unblock function, while modern code should use mask instead.
  • MonadCatchIO exports an instance for the ContT transformer. However, continuations and exceptions are known to have nontrivial interactions which require extra care to handle properly.

In some sense, MonadError is a non-sequitur, because it isn’t tied to IO at all; perfectly valid instances of it exist for non-IO backed monads as well. MonadCatchIO is closer; the latter three points are not fatal ones could be easily accounted for:

class MonadException m where
  throwM  :: Exception e => e -> m a
  catch   :: Exception e => m a -> (e -> m a) -> m a
  mask    :: ((forall a. m a -> m a) -> m b) -> m b

(With a removal of the ContT instance.) However, the “finalizers are sometimes skipped” problem is a bit more problematic. In effect, it is the fact that there may exist zeros which a given instance of MonadCatchIO may not know about. It has been argued that these zeros are none of MonadCatchIO’s business; one inference you might draw from this is that if you have short-circuiting which you would like to respect finalizers installed using MonadException, it should be implemented using asynchronous exceptions. In other words, ErrorT is a bad idea.

However, there is another perspective you can take: MonadException is not tied just to asynchronous exceptions, but any zero-like value which obeys the same laws that exceptions obey. The semantics of these exceptions are described in the paper Asynchronous Exceptions in Haskell. They specify exactly the interaction of masking, throw and catch, as well as how interrupts can be introduced by other threads. In this view, whether or not this behavior is prescribed by the RTS or by passing pure values around is an implementation detail: as long as an instance is written properly, zeros will be properly handled. This also means that it is no longer acceptable to provide a MonadException instance for ErrorT e, unless we also have an underlying MonadException for the inner monad: we can’t forget about exceptions on the lower layers!

There is one last problem with this approach: once the primitives have been selected, huge swaths of the standard library have to be redefined by “copy pasting” their definitions but having them refer to the generalized versions. This is a significant practical hurdle for implementing a library based on this principle: it’s simply not enough to tack a liftIO to the beginning of a function!

I think an emphasis on the semantics of the defined type class will be critical for the future of this lineage of typeclasses; this is an emphasis that hasn’t really existed in the past. From this perspective, we define with our typeclass not only a way to access otherwise inaccessible functions in IO, but also how these functions should behave. We are, in effect, modeling a subset of IO. I think Conal Elliott would be proud.

There is a lively debate going on about extensions to the original semantics of asynchronous exceptions, allowing for the notion of “recoverable” and “unrecoverable” errors. (It’s nearer to the end of the thread.)

Threading pure effects

This technique can be described as generalizing the a common implementation technique which was used to implement many of the original functions in MonadCatchIO. These are a rather odd set of signatures:

class Monad m => MonadMorphIO m where
  morphIO :: (forall b. (m a -> IO b) -> IO b) -> m a

class MonadIO m => MonadPeelIO m where
  peelIO :: m (m a -> IO (m a))

class MonadBase b m => MonadBaseControl b m | m -> b where
  data StM m :: * -> *
  liftBaseWith :: (RunInBase m b -> b a) -> m a
  restoreM :: StM m a → m a
type RunInBase m b = forall a. m a -> b (StM m a)

The key intuition behind these typeclasses is that they utilize polymorphism in the IO function that is being lifted in order to thread the pure effects of the monad stack on top of IO. You can see this as the universal quantification in morphIO, the return type of peelIO (which is IO (m a), not IO a), and the StM associated type in MonadBaseControl. For example, Int -> StateT s IO a, is equivalent to the type Int -> s -> IO (s, a). We can partially apply this function with the current state to get Int -> IO (s, a); it should be clear then that as long as the IO function we’re lifting lets us smuggle out arbitrary values, we can smuggle out our updated state and reincorporate it when the lifted function finishes. The set of functions which are amenable to this technique are precisely the ones for which this threaded is possible.

As I described in this post, this means that you won’t be able to get any transformer stack effects if they aren’t returned by the function. So perhaps a better word for MonadBaseControl is not that it is unsound (although it does admit strange behavior) but that it is incomplete: it cannot lift all IO functions to a form where the base monad effects and the transformer effects always occur in lockstep.

This has some interesting implications. For example, this forgetfulness is in fact precisely the reason why a lifted bracketing function will always run no matter if there are other zeros: finally by definition is only aware of asynchronous exceptions. This makes monad-control lifted functions very explicitly only handling asynchronous exceptions: a lifted catch function will not catch an ErrorT zero. However, if you manually implement finally using lifted versions of the more primitive functions, finalizers may be dropped.

It also suggests an alternate implementation strategy for monad-control: rather than thread the state through the return type of a function, it could instead be embedded in a hidden IORef, and read out at the end of the computation. In effect, we would like to embed the semantics of the pure monad transformer stack inside IO. Some care must be taken in the forkIO case, however: the IORefs need to also be duplicated appropriately, in order to maintain thread locality, or MVars used instead, in order to allow coherent non-local communication.

It is well known that MonadBaseControl does not admit a reasonable instance for ContT. Mikhail Vorozhtsov has argued that this is too restrictive. The difficulty is that while unbounded continuations do not play nice with exceptions, limited use of continuation passing style can be combined with exceptions in a sensible way. Unfortunately, monad-control makes no provision for this case: the function it asks a user to implement is too powerful. It seems the typeclasses explicitly modeling a subset of IO are, in some sense, more general! It also highlights the fact that these type classes are first and foremost driven by an abstraction of a common implementation pattern, rather than any sort of semantics.

Conclusion

I hope this essay has made clear why I think of MonadBaseControl as an implementation strategy, and not as a reasonable interface to program against. MonadException is a more reasonable interface, which has a semantics, but faces significant implementation hurdles.

by Edward Z. Yang at January 24, 2012 06:31 PM

Yesod Web Framework

Wiki: markdown, chat subsite, event source

This example will tie together a few different ideas. We'll start with a chat subsite, which allows us to embed a chat widget on any page. We'll use the HTML 5 event source API to handle sending events from the server to the client.

-- @Chat.hs
{-# LANGUAGE OverloadedStrings, TypeFamilies, QuasiQuotes,
             TemplateHaskell, FlexibleInstances, MultiParamTypeClasses,
             FlexibleContexts
  #-}
-- | This modules defines a subsite that allows you to insert a chat box on
-- any page of your site. It uses eventsource for sending the messages from
-- the server to the browser.
module Chat where

import Yesod
import Control.Concurrent.Chan (Chan, dupChan, writeChan)
import Data.Text (Text)
import Network.Wai.EventSource (ServerEvent (..), eventSourceApp)
import Language.Haskell.TH.Syntax (Type (VarT), Pred (ClassP), mkName)
import Blaze.ByteString.Builder.Char.Utf8 (fromText)
import Data.Monoid (mappend)

-- | Our subsite foundation. We keep a channel of events that all connections
-- will share.
data Chat = Chat (Chan ServerEvent)

-- | We need to know how to check if a user is logged in and how to get
-- his/her username (for printing messages).
class (Yesod master, RenderMessage master FormMessage)
        => YesodChat master where
    getUserName :: GHandler sub master Text
    isLoggedIn :: GHandler sub master Bool

-- Now we set up our subsite. The first argument is the subsite, very similar
-- to how we've used mkYesod in the past. The second argument is specific to
-- subsites. What it means here is "the master site must be an instance of
-- YesodChat".
--
-- We define two routes: a route for sending messages from the client to the
-- server, and one for opening up the event stream to receive messages from
-- the server.
mkYesodSub "Chat"
    [ ClassP ''YesodChat [VarT $ mkName "master"]
    ] [parseRoutes|
/send SendR POST
/recv ReceiveR GET
|]

-- | Get a message from the user and send it to all listeners.
postSendR :: YesodChat master => GHandler Chat master ()
postSendR = do
    from <- getUserName

    -- Note that we're using GET parameters for simplicity of the Ajax code.
    -- This could easily be switched to POST. Nonetheless, our overall
    -- approach is still RESTful since this route can only be accessed via a
    -- POST request.
    body <- runInputGet $ ireq textField "message"

    -- Get the channel
    Chat chan <- getYesodSub

    -- Send an event to all listeners with the user's name and message.
    liftIO $ writeChan chan $ ServerEvent Nothing Nothing $ return $
        fromText from `mappend` fromText ": " `mappend` fromText body

-- | Send an eventstream response with all messages streamed in.
getReceiveR :: GHandler Chat master ()
getReceiveR = do
    -- First we get the main channel
    Chat chan0 <- getYesodSub

    -- We duplicated the channel, which allows us to create broadcast
    -- channels.
    chan <- liftIO $ dupChan chan0

    -- Now we use the event source API. eventSourceApp takes two parameters:
    -- the channel of events to read from, and the WAI request. It returns a
    -- WAI response, which we can return with sendWaiResponse.
    req <- waiRequest
    res <- lift $ eventSourceApp chan req
    sendWaiResponse res

-- | Provide a widget that the master site can embed on any page.
chatWidget :: YesodChat master
           => (Route Chat -> Route master)
           -> GWidget sub master ()
-- This toMaster argument tells us how to convert a Route Chat into a master
-- route. You might think this is redundant information, but taking this
-- approach means we can have multiple chat subsites in a single site.
chatWidget toMaster = do
    -- Get some unique identifiers to help in creating our HTML/CSS. Remember,
    -- we have no idea what the master site's HTML will look like, so we
    -- should not assume we can make up identifiers that won't be reused.
    -- Also, it's possible that multiple chatWidgets could be embedded in the
    -- same page.
    chat <- lift newIdent   -- the containing div
    output <- lift newIdent -- the box containing the messages
    input <- lift newIdent  -- input field from the user

    ili <- lift isLoggedIn  -- check if we're already logged in
    if ili
        then do
            -- Logged in: show the widget
            [whamlet|
<div ##{chat}>
    <h2>Chat
    <div ##{output}>
    <input ##{input} type=text placeholder="Enter Message">
|]
            -- Just some CSS
            toWidget [lucius|
##{chat} {
    position: absolute;
    top: 2em;
    right: 2em;
}
##{output} {
    width: 200px;
    height: 300px;
    border: 1px solid #999;
    overflow: auto;
}
|]
            -- And now that Javascript
            toWidgetBody [julius|
// Set up the receiving end
var output = document.getElementById("#{output}");
var src = new EventSource("@{toMaster ReceiveR}");
src.onmessage = function(msg) {
    // This function will be called for each new message.
    var p = document.createElement("p");
    p.appendChild(document.createTextNode(msg.data));
    output.appendChild(p);

    // And now scroll down within the output div so the most recent message
    // is displayed.
    output.scrollTop = output.scrollHeight;
};

// Set up the sending end: send a message via Ajax whenever the user hits
// enter.
var input = document.getElementById("#{input}");
input.onkeyup = function(event) {
    var keycode = (event.keyCode ? event.keyCode : event.which);
    if (keycode == '13') {
        var xhr = new XMLHttpRequest();
        var val = input.value;
        input.value = "";
        var params = "?message=" + encodeURI(val);
        xhr.open("POST", "@{toMaster SendR}" + params);
        xhr.send(null);
    }
}
|]
        else do
            -- User isn't logged in, give a not-logged-in message.
            master <- lift getYesod
            [whamlet|
<p>
    You must be #
    $maybe ar <- authRoute master
        <a href=@{ar}>logged in
    $nothing
        logged in
    \ to chat.
|]

This module stands on its own, and can be used in any application. Next we'll provide such a driver application: a wiki. Our wiki will have a hard-coded homepage, and then a wiki section of the site. We'll be using multiple dynamic pieces to allow an arbitrary hierarchy of pages within the Wiki.

For storage, we'll just use a mutable reference to a Map. For a production application, this should be replaced with a proper database. The content will be stored and served as Markdown. yesod-auth's dummy plugin will provide us with (fake) authentication.

{-# LANGUAGE OverloadedStrings, TypeFamilies, QuasiQuotes,
             TemplateHaskell, FlexibleInstances, MultiParamTypeClasses,
             FlexibleContexts
  #-}
import Yesod
import Yesod.Auth
import Yesod.Auth.Dummy (authDummy)
import Chat
import Control.Concurrent.Chan (Chan, newChan)
import Network.Wai.Handler.Warp (run)
import Data.Text (Text)
import qualified Data.Text.Lazy as TL
import qualified Data.IORef as I
import qualified Data.Map as Map
import Text.Markdown (markdown, def)

-- | Our foundation type has both the chat subsite and a mutable reference to
-- a map of all our wiki contents. Note that the key is a list of Texts, since
-- a wiki can have an arbitrary hierarchy.
--
-- In a real application, we would want to store this information in a
-- database of some sort.
data Wiki = Wiki
    { getChat :: Chat
    , wikiContent :: I.IORef (Map.Map [Text] Text)
    }

-- Set up our routes as usual.
mkYesod "Wiki" [parseRoutes|
/ RootR GET                 -- the homepage
/wiki/*Texts WikiR GET POST -- note the multipiece for the wiki hierarchy
/chat ChatR Chat getChat    -- the chat subsite
/auth AuthR Auth getAuth    -- the auth subsite
|]

instance Yesod Wiki where
    approot _ = ""
    authRoute _ = Just $ AuthR LoginR -- get a working login link

    -- Our custom defaultLayout will add the chat widget to every page.
    -- We'll also add login and logout links to the top.
    defaultLayout widget = do
        pc <- widgetToPageContent $ widget >> chatWidget ChatR
        mmsg <- getMessage
        hamletToRepHtml [hamlet|
$doctype 5
<html>
    <head>
        <title>#{pageTitle pc}
        ^{pageHead pc}
    <body>
        $maybe msg <- mmsg
            <div .message>#{msg}
        <nav>
            <a href=@{AuthR LoginR}>Login
            \ | #
            <a href=@{AuthR LogoutR}>Logout
        ^{pageBody pc}
|]

-- Fairly standard YesodAuth instance. We'll use the dummy plugin so that you
-- can create any name you want, and store the login name as the AuthId.
instance YesodAuth Wiki where
    type AuthId Wiki = Text
    authPlugins _ = [authDummy]
    loginDest _ = RootR
    logoutDest _ = RootR
    getAuthId = return . Just . credsIdent
    authHttpManager = error "authHttpManager" -- not used by authDummy

-- Just implement authentication based on our yesod-auth usage.
instance YesodChat Wiki where
    getUserName = requireAuthId
    isLoggedIn = do
        ma <- maybeAuthId
        return $ maybe False (const True) ma

instance RenderMessage Wiki FormMessage where
    renderMessage _ _ = defaultFormMessage

-- Nothing special here, just giving a link to the root of the wiki.
getRootR :: Handler RepHtml
getRootR = defaultLayout [whamlet|
<p>Welcome to the Wiki!
<p>
    <a href=@{wikiRoot}>Wiki root
|]
  where
    wikiRoot = WikiR []

-- A form for getting wiki content
wikiForm mtext = renderDivs $ areq textareaField "Page body" mtext

-- Show a wiki page and an edit form
getWikiR :: [Text] -> Handler RepHtml
getWikiR page = do
    -- Get the reference to the contents map
    icontent <- fmap wikiContent getYesod

    -- And read the map from inside the reference
    content <- liftIO $ I.readIORef icontent

    -- Lookup the contents of the current page, if available
    let mtext = Map.lookup page content

    -- Generate a form with the current contents as the default value.
    -- Note that we use the Textarea wrapper to get a <textarea>.
    ((_, form), _) <- generateFormPost $ wikiForm $ fmap Textarea mtext
    defaultLayout $ do
        case mtext of
            -- We're treating the input as markdown. The markdown package
            -- automatically handles XSS protection for us.
            Just text -> toWidget $ markdown def $ TL.fromStrict text
            Nothing -> [whamlet|<p>Page does not yet exist|]
        [whamlet|
<h2>Edit page
<form method=post>
    ^{form}
    <div>
        <input type=submit>
|]

-- Get a submitted wiki page and updated the contents.
postWikiR :: [Text] -> Handler RepHtml
postWikiR page = do
    icontent <- fmap wikiContent getYesod
    content <- liftIO $ I.readIORef icontent
    let mtext = Map.lookup page content
    ((res, form), _) <- runFormPost $ wikiForm $ fmap Textarea mtext
    case res of
        FormSuccess (Textarea t) -> do
            liftIO $ I.atomicModifyIORef icontent $
                \m -> (Map.insert page t m, ())
            setMessage "Page updated"
            redirect $ WikiR page
        _ -> defaultLayout [whamlet|
<form method=post>
    ^{form}
    <div>
        <input type=submit>
|]

main :: IO ()
main = do
    -- Create our server event channel
    chan <- newChan

    -- Initially have a blank database of wiki pages
    icontent <- I.newIORef Map.empty

    -- Run our app
    warpDebug 3000 $ Wiki (Chat chan) icontent

January 24, 2012 10:12 AM

Erik de Castro Lopo

Benchmarking and QuickChecking readInt.

I'm currently working on converting my http-proxy library from using the Data.Enumerator package to Data.Conduit (explanation of why in my last blog post).

During this conversion, I have been studying the sources of the Warp web server because my http-proxy was originally derived from the Enumerator version of Warp. While digging through the Warp code I found the following code (and comment) which is used to parse the number provided in the Content-Length field of a HTTP header:

  -- Note: This function produces garbage on invalid input. But serving an
  -- invalid content-length is a bad idea, mkay?
  readInt :: S.ByteString -> Integer
  readInt = S.foldl' (\x w -> x * 10 + fromIntegral w - 48) 0

The comment clearly states that that this function can produce garbage, specifically if the string contains anything other than ASCII digits. The comment is also correct that an invalid Content-Length is a bad idea. However, on seeing the above code, and remembering something I had seen recently in the standard library, I naively sent the Yesod project a patch replacing the above code with a version that uses the readDec function from the Numeric module:

  import Data.ByteString (ByteString)
  import qualified Data.ByteString.Char8 as B
  import qualified Numeric as N

  readInt :: ByteString -> Integer
  readInt s =
      case N.readDec (B.unpack s) of
          [] -> 0
          (x, _):_ -> x

About 3-4 hours after I submitted the patch I got an email from Michael Snoyman saying that parsing the Content-Length field is a hot spot for the performance of Warp and that I should benchmark it against the code I'm replacing to make sure there is no unacceptable performance penalty.

That's when I decided it was time to check out Bryan O'Sullivan's Criterion bench-marking library. A quick read of the docs and bit of messing around and I was able to prove to myself that using readDec was indeed much slower than the code I wanted to replace.

The initial disappointment of finding that a more correct implementation was significantly slower than the less correct version quickly turned to joy as I experimented with a couple of other implementations and eventually settled on this:

  import Data.ByteString (ByteString)
  import qualified Data.ByteString.Char8 as B
  import qualified Data.Char as C

  readIntTC :: Integral a => ByteString -> a
  readIntTC bs = fromIntegral
          $ B.foldl' (\i c -> i * 10 + C.digitToInt c) 0
          $ B.takeWhile C.isDigit bs

By using the Integral type class, this function converts the given ByteString to any integer type (ie any type belonging to the Integral type class). When used, this function will be specialized by the Haskell compiler at the call site to to produce code to read string values into Ints, Int64s or anything else that is a member of the Integral type class.

For a final sanity check I decided to use QuickCheck to make sure that the various versions of the generic function were correct for values of the type they returned. To do that I wrote a very simple QuickCheck property as follows:

  prop_read_show_idempotent :: Integral a => (ByteString -> a) -> a -> Bool
  prop_read_show_idempotent freader x =
      let posx = abs x
      in posx == freader (B.pack $ show posx)

This QuickCheck property takes the function under test freader and QuickCheck will then provide it values of the correct type. Since the function under test is designed to read Content-Length values which are always positive, we only test using the absolute value of the value randomly generated by QuickCheck.

The complete test program can be found on Github in this Gist and can be compiled and run as:

  ghc -Wall -O3 --make readInt.hs -o readInt && ./readInt

When run, the output of the program looks like this:

  Quickcheck tests.
  +++ OK, passed 100 tests.
  +++ OK, passed 100 tests.
  +++ OK, passed 100 tests.
  Criterion tests.
  warming up
  estimating clock resolution...
  mean is 3.109095 us (320001 iterations)
  found 27331 outliers among 319999 samples (8.5%)
    4477 (1.4%) low severe
    22854 (7.1%) high severe
  estimating cost of a clock call...
  mean is 719.4627 ns (22 iterations)

  benchmarking readIntOrig
  mean: 4.653041 us, lb 4.645949 us, ub 4.663823 us, ci 0.950
  std dev: 43.94805 ns, lb 31.52653 ns, ub 73.82125 ns, ci 0.950

  benchmarking readDec
  mean: 13.12692 us, lb 13.10881 us, ub 13.14411 us, ci 0.950
  std dev: 90.63362 ns, lb 77.52619 ns, ub 112.4304 ns, ci 0.950

  benchmarking readRaw
  mean: 591.8697 ns, lb 590.9466 ns, ub 594.1634 ns, ci 0.950
  std dev: 6.995869 ns, lb 3.557109 ns, ub 14.54708 ns, ci 0.950

  benchmarking readInt
  mean: 388.3835 ns, lb 387.9500 ns, ub 388.8342 ns, ci 0.950
  std dev: 2.261711 ns, lb 2.003214 ns, ub 2.585137 ns, ci 0.950

  benchmarking readInt64
  mean: 389.4380 ns, lb 388.9864 ns, ub 389.9312 ns, ci 0.950
  std dev: 2.399116 ns, lb 2.090363 ns, ub 2.865227 ns, ci 0.950

  benchmarking readInteger
  mean: 389.3450 ns, lb 388.8463 ns, ub 389.8626 ns, ci 0.950
  std dev: 2.599062 ns, lb 2.302428 ns, ub 2.963600 ns, ci 0.950

At the top of the output is proof that all three specializations of the generic function readIntTC satisfy the QuickCheck property. From the Criterion output its pretty obvious that the Numeric.readDec version is about 3 times slower that the original function. More importantly, all three version of this generic function are an order of magnitude faster than the original.

That's a win! I will be submitting my new function for inclusion in Warp.

Update : 14:13

At around the same time I submitted my latest version for readInt Vincent Hanquez posted a comment on the Github issue suggesting I look at the GHC MagicHash extension and pointed me to an example.

Sure enough, using the MagicHash technique resulted in something significantly faster again.

January 24, 2012 12:52 AM

January 23, 2012

Noam Lewis

When to scroll? The problem of infinite-sized UI elements inside a scrollviewer

While refactoring a WPF application, I’ve stumbled into a general problem in UI layout.

WPF has an element called ScrollViewer, which is basically a panel that contains elements and shows scroll bars when the content inside is too big to fit in the size of the ScrollViewer’s visible area. Consider the following three cases.

Case 1: Finite-sized UI content within a ScrollViewer

In this case, obviously the scroll bars should only appear if the visible area of the ScrollViewer is too small to show the content. If the window is large enough, there entire finite-sized content will fit in the screen and no scroll bars are necessary (although we may want to still show them, disabled, for stylistic reasons).

Case 2: Infinite-sized UI content within a limited container (NOT a ScrollViewer)

Sometimes we have content that can use any size assigned to it. For example, a grid (in my case, DevExpress’ WPF GridControl) of data rows. Assuming a huge (or even infinite) data source, the grid’s size on the screen has no upper bound – it can always grow more and always show more content. The grid’s size in the UI must be limited somehow. There are two ways to limit the size of such a UI control:

  1. Set a maximum size on the control, or
  2. Place the infinitely-sized control in a container that has a maximum size defined.

In other words, the infinite control must lie in a tree of containers where at least one container up the ancestry must have defined a maximum size. If no parent container specifies a maximum size, eventually the top-level container – the window – DOES have a maximum size and the infinite-sized control will fill the whole window (and nothing more).

Case 3: Infinite-sized UI content within a ScrollViewer

Now, consider that the container of the infinite UI control (such as a grid with infinite rows in the data source), is actually container within a ScrollViewer. In this case, the ScrollViewer, being what it is, does NOT limit the size of its contents, so the infinitely-sized control will “explode” and will, by our algorithm, try to occupy an infinite size. Specifically in the case of a DevExpress WPF GridControl, the authors of that control know to detect that situation – and an exception is thrown, stating:

DevExpress.Wpf.Grid.InfiniteGridSizeException was unhandled
  Message=By default, an infinite grid height is not allowed since all grid rows will be rendered and hence the grid will work very slowly. To fix this issue, you should place the grid into a container that will give a finite height to the grid, or you should manually specify the grid’s Height or MaxHeight. Note that you can also avoid this exception by setting the GridControl.AllowInfiniteGridSize static property to True, but in that case the grid will run slowly.”

Problem is, sometimes we want to put that grid in a ScrollViewer so that when the screen is too small, the grid will assume some minimal size, and a scroll bar will be shown if the screen (or window) is smaller than the minimum. If the window is huge, what we want is to expand the grid to fill the available space in the window – as big as the window can be, with no limit. If someone is using the application on a 5000-inch screen, we want to use all that space. If someone is using a 1-inch screen, we want the grid to be 3 inches and show a scroll bar.

So, the solution seems simple enough: we can tell the ScrollViewer to have dual behavior:

  1. If the available screen size is smaller than some minimum, allow scrolling, and set the size available for the content to be that minimum size.
  2. If the available screen size is larger than the minimum, behave like a regular container that gives its children only the space it has on the screen.

For example, if we decide the content requires at least 100 pixels, if the ScrollViewer has 80 pixels available – make the content within the scrollable area exactly 100 pixels, and show scroll bars. If the size available for the ScrollViewer is 200 pixels (more than the minimum 100 pixels) – don’t allow scrolling, and let the contained UI content use up to 200 pixels. Here’s a WPF behavior for ScrollViewer that does exactly that:

    public class ScrollViewerMaxSizeBehavior : Behavior<ScrollViewer>
    {
        public static readonly DependencyProperty MinContentHeightProperty = DependencyProperty.Register("MinContentHeight", typeof(int),
            typeof(ScrollViewerMaxSizeBehavior), new UIPropertyMetadata() { PropertyChangedCallback = MinSizeChanged } );
        public int MinContentHeight
        {
            get { return (int)GetValue(MinContentHeightProperty); }
            set { SetValue(MinContentHeightProperty, value); }
        }

        public static readonly DependencyProperty MinContentWidthProperty = DependencyProperty.Register("MinContentWidth", typeof(int),
            typeof(ScrollViewerMaxSizeBehavior), new UIPropertyMetadata() { PropertyChangedCallback = MinSizeChanged });
        public int MinContentWidth
        {
            get { return (int)GetValue(MinContentWidthProperty); }
            set { SetValue(MinContentWidthProperty, value); }
        }

        protected static void MinSizeChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
        {
            var self = d as ScrollViewerMaxSizeBehavior;
            if (null == self)
            {
                return;
            }
            self.Update();
        }

        protected override void OnAttached()
        {
            base.OnAttached();
            this.AssociatedObject.SizeChanged += this.ParentSizeChanged;
            this.Update();
        }

        protected override void OnDetaching()
        {
            this.AssociatedObject.SizeChanged -= this.ParentSizeChanged;
            base.OnDetaching();
        }

        protected void ParentSizeChanged(Object sender, SizeChangedEventArgs e)
        {
            this.Update();
        }

        private void Update()
        {
            if (null == this.AssociatedObject)
            {
                return;
            }
            var content = this.AssociatedObject.Content as FrameworkElement;

            if ((0 >= this.AssociatedObject.ActualHeight)
                || (0 >= this.AssociatedObject.ActualWidth))
            {
                // The attached ScrollViewer was probably not laid out yet, or has zero size.
                this.AssociatedObject.VerticalScrollBarVisibility = ScrollBarVisibility.Disabled;
                this.AssociatedObject.HorizontalScrollBarVisibility = ScrollBarVisibility.Disabled;
                return;
            }

            int minHeight = this.MinContentHeight;
            int minWidth = this.MinContentWidth;

            if ((minHeight <= 0) || (minWidth <= 0))
            {
                // Probably our attached properties were not initialized. By default we disable the scrolling completely,
                // to prevent exceptions from infinitely-sized objects within us.
                this.AssociatedObject.VerticalScrollBarVisibility = ScrollBarVisibility.Disabled;
                this.AssociatedObject.HorizontalScrollBarVisibility = ScrollBarVisibility.Disabled;
                return;
            }

            this.AssociatedObject.SizeChanged -= this.ParentSizeChanged;

            if (this.AssociatedObject.ActualHeight < minHeight)
            {
                this.AssociatedObject.VerticalScrollBarVisibility = ScrollBarVisibility.Auto;
                if (null != content)
                {
                    content.MaxHeight = minHeight - (content.Margin.Bottom + content.Margin.Top);
                }
            }
            else
            {
                this.AssociatedObject.VerticalScrollBarVisibility = ScrollBarVisibility.Disabled;
                if (null != content)
                {
                    content.MaxHeight = Double.PositiveInfinity;
                }
            }

            if (this.AssociatedObject.ActualWidth < minWidth)
            {
                this.AssociatedObject.HorizontalScrollBarVisibility = ScrollBarVisibility.Auto;
                if (null != content)
                {
                    content.MaxWidth = minWidth - (content.Margin.Left + content.Margin.Right);
                }
            }
            else
            {
                this.AssociatedObject.HorizontalScrollBarVisibility = ScrollBarVisibility.Disabled;
                if (null != content)
                {
                    content.MaxWidth = Double.PositiveInfinity;
                }
            }

            this.AssociatedObject.SizeChanged += this.ParentSizeChanged;
        }
    }

An here's how to use it in a XAML file (assuming the above class was defined in a namespace known as "custom" within the XML namespace):
Somewhere at the top: xmlns:Interactivity="clr-namespace:System.Windows.Interactivity;assembly=System.Windows.Interactivity"
Then:

    <ScrollViewer Style="{StaticResource AppHost_ScrollViewer}">
        <Interactivity:Interaction.Behaviors>
            <custom:ScrollViewerMaxSizeBehavior MinContentWidth="600"
                                                MinContentHeight="500"/>
        </Interactivity:Interaction.Behaviors>

        <!-- content -->

    </ScrollViewer>

Almost, but not quite

Continuing the 100-pixel example – what happens if we have some statically sized content (no infinite sizes) that requires more than what we defined as a minimum? For example, what if instead of a dynamically-sizable grid we have content that requires 200 pixels, and the window size is 100 pixels? In this case, the previous solution is bad: it will not allow scrolling to see the full 200 pixels. So our dual-behavior needs to know, somehow, if the content within it can expand at all an infinite size (and therefore requires the dual behavior defined above to prevent explosion to infinite size). Because if the content has a finite size, we simply want the ScrollViewer to behave as usual and allow scrolling to see that maximum size.

Conclusion – what’s missing in WPF (and possibly other UI frameworks)

What I’d expect is that there would be some property on UI controls specifying whether or not this element can expand infinitely (in height, width, or both). If any node in the UI’s element containment tree has this “infinite size” property set, then the ScrollViewer that contains this tree must act using the dual behavior and must have a minimum size defined (smaller size means we allow scrolling and set the content exactly to the minimum size; bigger-than-minimum size means we don’t allow scrolling and give the content whatever space we have and no more). If the “infinite size” property, propagated to the ScrollViewer from the contained tree is not set, the ScrollViewer acts like a regular ScrollViewer – allowing the content to grow to whatever size it needs, and showing scroll bars if needed.

For now, since there is no such feature in WPF, I’ll be using the aforementioned ScrollViewer behavior with appropriately defined minimum sizes for those screens that need them – hard coded, ugly, but works.


by sinelaw at January 23, 2012 05:48 PM

Edward Z. Yang

monad-control is tricky

Editor's note. I've toned down some of the rhetoric in this post. The original title was "monad-control is unsound".

MonadBaseControl and MonadTransControl, from the monad-control package, specify an appealing way to automatically lift functions in IO that take "callbacks" to arbitrary monad stacks based on IO. Their appeal comes from the fact that they seem to offer a more general mechanism than the alternative: picking some functions, lifting them, and then manually reimplementing generic versions of all the functions built on top of them.

Unfortunately, monad-control has rather surprising behavior for many functions you might lift.

For example, it doesn't work on functions which invoke the callback multiple times:

{-# LANGUAGE FlexibleContexts #-}

import Control.Monad.Trans.Control
import Control.Monad.State

double :: IO a -> IO a
double m = m >> m

doubleG :: MonadBaseControl IO m => m a -> m a
doubleG = liftBaseOp_ double

incState :: MonadState Int m => m ()
incState = get >>= \x -> put (x + 1)

main = execStateT (doubleG (incState)) 0 >>= print

The result is 1, rather than 2 that we would expect. If you are unconvinced, suppose that the signature of double was Identity a -> Identity a, e.g. a -> a. There is only one possible implementation of this signature: id. It should be obvious what happens, in this case.

If you look closely at the types involved in MonadBaseControl, the reason behind this should become obvious: we rely on the polymorphism of a function we would like to lift in order to pass StM m around, which is the encapsulated “state” of the monad transformers. If this return value is discarded by IO, as it is in our function double, there is no way to recover that state. (This is even alluded to in the liftBaseDiscard function!)

My conclusion is that, while monad-control may be a convenient implementation mechanism for lifted versions of functions, the functions it exports suffer from serious semantic incoherency. End-users, take heed!

Postscript. A similar injunction holds for the previous versions of MonadBaseControl/MonadTransControl, which went by the names MonadPeel and MonadMorphIO.

by Edward Z. Yang at January 23, 2012 05:39 PM

Disciple/DDC

The new Disciple Core language

It's been a while since updates. The last one was in August, and after that I got distracted with ICFP. When I got back I spent more time doing Coq proofs, finishing with Progress and Preservation for a version of SystemF2 with mutable algebraic data. The proofs are here. All data is allocated into the store (heap), and there are primitive functions to update it. In this language you can, say, destructively update a list so that the tail points to different physical data. This language doesn't have effect typing, but after this proof I felt like I had crossed the line from "Coq-shy" to "Coq-sure".

In early November I read about ciu-equivalence, and how to prove contextual equivalence of program transformations. Thinking ahead, it felt like time to cleanup the sponginess in the existing DDC core language, because I'd need to do this before trying to formalise it. Although the plain calculus has a proper semantics and a hand-done soundness proof, the actual core language as used in the compiler also has some half-baked hacks. The reviewers of a previous paper had made suggestions about how to cleanup other aspects of the core calculus, and having Coqified all those languages, I now understand why it was good advice.

Cutting to the chase, I've redesigned the DDC core language and there is an interpreter that works well enough to evaluate simple recursive functions. Here is an example:
  letrec {
ack [r1 r2 r3: %]
(m : Int r1) {!0 | Use r1 + Use r2 + Use r3}
(n : Int r2) { Read r1 + Read r2 + Alloc r3
| Use r1 + Use r2 + Use r3}
: Int r3
= letregion r4 in
let zero = 0 [r4] () in
let one = 1 [r4] () in
case eqInt [:r1 r4 r4:] m zero of {
1 -> addInt [:r2 r4 r3:] n one;
_ -> case eqInt [:r2 r4 r4:] n zero of {
1 -> ack [:r4 r4 r3:]
(subInt [:r1 r4 r4:] m one)
(1 [r4] ());

_ -> ack [:r4 r4 r3:]
(subInt [:r1 r4 r4:] m one)
(ack [:r1 r4 r4:] m (subInt [:r2 r4 r4:] n one));
}
}
} in ack [:R0# R0# R0#:] (2 [R0#] ()) (3 [R0#] ());;
Here is another example that does destructive update of an integer:
  letregion r1 with {w1 : Mutable r1} in
let x : Int r1 = 0 [r1] () in
let _ : Unit = updateInt [:r1 r1:] < w1 > x (2 [r1] ()) in
addInt [:r1 r1 R2#:] x (3 [r1] ());;
and one that suspends the allocation of an integer with lazy evaluation:
  letregion r1 with { w1 : Const r1; w2 : Lazy r1; w3 : Global r1 } in
let x : Int r1 lazy < w2 >
= purify < alloc [r1] w1 > in
forget < use [r1] w3 w1 > in
2 [r1] ()
in addInt [:r1 R0# R0#:] x (3 [R0#] ());;

Note that this is an intermediate representation for a compiler, and has vastly more type information than a real user would want to write. The compiler will perform type inference on the source language, and automatically translate the user program to this lower level language.

The new DDC core language is described on the wiki and so far I've been reasonably good at keeping the wiki up to date with what's implemented.

The main changes are:
  • Witnesses now exist in their own universe, at level 0 next to values. Both values and witnesses are classified by types, and types classified by kinds. This removes the dependent-kinding of the old language. The more I thought about it, the less fun formalising a dependently kinded language seemed to be.

  • I've removed the more-than constraints on effect and closure variables. The type inference algorithm I am using returns constraints on effect and closure variables, so I had added similar constraints to the core language. This was a bad idea because managing the additional subsumption judgements during core type checking is a headache. The new core language has no such constraints, and I think I know to process the output of the inferencer so I won't need them.

  • Introducing lazy evaluation is now done with the let .. lazy binding form instead of a primitive suspend operator. This goes hand-in-hand with the purify and forget keywords that mask the effect and closure of their body respectively. These two keywords are akin to the type casts used by SystemFc and the GHC core language to support GADTs. I think it makes more sense to mask out the effect of an expression before suspending it, than to pass a witness as to why it's pure. The former style will still be used in the source program because that's how the type inferencer works, but the latter should be easier to work with in the core language.
Anyway, if you darcs get the DDC repo you can make bin/ddci-core to build the interpreter. Run the examples like bin/ddci-core test/ddci-core/30-Eval/30-Letrec/Test.dcx. It's not completely finished, but all the examples under the test/ddci-core directory run fine.

The interpreter should be done in another couple of weeks. After that it'll be time to split off the LLVM backend from the existing compiler so that we can compile core programs directly.

by Ben Lippmeier (noreply@blogger.com) at January 23, 2012 06:20 AM

Jeremy Shaw

ANN: happstack-server 6.5.1

I am pleased to announce the release of happstack-server 6.5.1. Changes include:


Discussion here: 

by Jeremy Shaw (noreply@blogger.com) at January 23, 2012 01:01 AM

January 22, 2012

Dan Piponi (sigfpe)

Some parallels between classical and quantum mechanics

Introduction
This isn't really a blog post. More of something I wanted to interject in a discussion on Google plus but wouldn't fit in the text box.

I've always had trouble with the way the Legendre transform is introduced in classical mechanics. I know I'm not the only one. Many mathematicians and physicists have recognised that it seems to be plucked out of a hat like a rabbit and have even written papers to address this issue. But however much an author attempts to make it seem natural, it still looks like a rabbit to me.

So I have to ask myself, what would make me feel comfortable with the Legendre transform?

The Legendre transform is an analogue of the Fourier transform that uses a different semiring to the usual. I wrote briefly about this many years ago. So if we could write classical mechanics in a form that is analogous to another problem where I'd use a Fourier transform, I'd be happier. This is my attempt to do that.

When I wrote about Fourier transforms a little while back the intention was to immediately follow it with an analogous article about Legendre transforms. Unfortunately that's been postponed so I'm going to just assume you know that Legendre transforms can be used to compute inf-convolutions. I'll state clearly what that means below, but I won't show any detail on the analogy with Fourier transforms.

Free classical particles
Let's work in one dimension with a particle of mass whose position at time is . The kinetic energy of this particle is given by . Its Lagrangian is therefore .

The action of our particle for the time from to is therefore


The particle motion is that which minimises the action.

Suppose the position of the particle at time is and the position at time is . Then write for the action minimising path from to . So

where we're minimising over all paths such that .

Now suppose our system evolves from time to . We can consider this to be two stages, one from to followed by one from to . Let be the minimised action analogous to for the period to . The action from to is the sum of the actions for the two subperiods. So the minimum total action for the period to is given by


Let me simply that a little. I'll use where I previously used and for . So that last equation becomes:


Now suppose is translation-independent in the sense that . So we can write . Then the minimum total action is given by


Infimal convolution is defined by

so the minimum we seek is


So now it's natural to use the Legendre transform. We have the inf-convolution theorem:

where is the Legendre transform of given by

and so (where we use to represent Legendre transform with respect to the spatial variable).

Let's consider the case where from onwards the particle motion is free, so . In this case we clearly have translation-invariance and so the time evolution is given by repeated inf-convolution with and in the "Legendre domain" this is nothing other than repeated addition of .

Let's take a look at . We know that if a particle travels freely from to over the period from to then it must have followed the minimum action path and we know, from basic mechanics, this is the path with constant velocity. So

and hence the action is given by

So the time evolution of is given by repeated inf-convolution with a quadratic function. The time evolution of is therefore given by repeated addition of the Legendre transform of a quadratic function. It's not hard to prove that the Legendre transform of a quadratic function is also quadratic. In fact:

Addition is easier to work with than inf-convolution so if we wish to understand the time evolution of the action function it's natural to work with this Legendre transformed function.

So that's it for classical mechanics in this post. I've tried to look at the evolution of a classical system in a way that makes the Legendre transform natural.

Free quantum particles
Now I want to take a look at the evolution of a free quantum particle to show how similar it is to what I wrote above. In this case we have the Schrödinger equation

Let's suppose that from time onwards the particle is free so . Then we have

Now let's take the Fourier transform in the spatial variable. We get:

So

We can write this as

where

So the time evolution of the free quantum particle is given by repeated convolution with a Gaussian function which in the Fourier domain is repeated multiplication by a Gaussian. The classical section above is nothing but a tropical version of this section.

Conclusion
I doubt I've said anything original here. Classical mechanics is well known to be the limit of quantum mechanics as and it's well known that in this limit we find that occurrences of the semiring are replaced by the semiring . But I've never seen an article that attempts to describe classical mechanics in terms of repeated inf-convolution even though this is close to Hamilton's formulation and I've never seen an article that shows the parallel with the Schrödinger equation in this way. I'm hoping someone will now be able to say to me "I've seen that before" and post a relevant link below.

Note
I'm not sure how the above applies for a non-trivial potential . I wrote this little Schrödinger equation solver a while back. As might be expected, it's inconvenient to use the Fourier domain to deal with the part of the evolution due to . In order to simulate a time step of the code simulates in the Fourier domain assuming the particle is free and then spends solving for the -dependent part in the spatial domain. So even in the presence of non-trivial it can still be useful to work with a Fourier transform. Almost the same iteration could be used to numerically compute the action for the classical case.

by sigfpe (noreply@blogger.com) at January 22, 2012 12:34 AM

Christopher Done

Upgrading GHC

So this weekend I finally upgraded two large work projects from GHC 6.12.3! And I didn’t upgrade one up in some gradual nonsense, I jumped straight to GHC 7.2.2, the latest release, like a boss.

Why did I still have projects running on 6.12.3? GHC 6.12.3 is a two year old release, and the codebases I upgraded are also two years old. I have about 46 dependencies with specific constraints and some patched dependencies, which made the prospect of updating quite daunting. I didn’t really need to upgrade, the software I was writing worked.

Did I even try to uprade? Yes, I tried several times off-hand over the past months to upgrade my GHC but was met immediately with hard to grok Cabal complaints about version numbers. Not to mention some dependencies that just didn’t support the latest GHC. For a while I had genuinely thought it would be hopeless to upgrade these projects without tears.

Today I came back to it and decided that I’d start with an empty Cabal file and add dependencies incrementally until every one was satisfied, and then the project built. As far as GHC itself goes, separate from Cabal, I only had to add BangPatterns and FlexibleInstances to some files and it was happy.

Does this say anything about the stability of Haskell as a platform? I think so, a little. In the end it’s certainly easier for maintenance to keep up to date with the latest packages, and I think I will introduce this factor into the continuous integration system for my projects.

It also says that despite all the new stuff happening in Haskell, code is at least stable a few versions old. There were only a couple packages that I couldn’t use in GHC 6.12.3. It also says that GHC 6.12.3 was a very decent and stable release, having been using it in production for two years with no problems.

Now I’m quite excited about all the GHC features I’ve been missing out on that I’ll be able to start using. I have some reading to do, so I’ve opened up the release notes of 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.2.1, and 7.2.2, and the release notes of the upcoming 7.4. I should also be able to try out the Scion package now, which is rather awesome.

January 22, 2012 12:00 AM

January 21, 2012

Joyride Laboratories

On Selling Nikki and the Robots’ Story Mode

Episode 1 of the story mode is ready to be released and we are very excited! This will also mark the first public beta version.

We will start selling the story mode using a pay-what-you-want model, which is strongly inspired by the Humble Indie Bundle (which on the other hand was inspired by a World of Goo sale). After one week we will then set a fixed minimum price.

When you buy the game, you pay for the game as it is, but will also get all further episodes and updates for free. With each further episode, however, the minimum price will increase for new buyers of the game. The sooner you buy, the less you pay, hooray! This is similar to the Alpha-Beta-1.0 price increments of Minecraft.

Joyride Labs Preorder Page Concept

Right now we are working hard on the presale website and we hope to be able to release Nikki and the Robots' story mode until mid February.

In the next post we will tell you some more details about episode 1, in which Nikki's great adventure begins to unfold.

by Iwan at January 21, 2012 07:55 PM

January 20, 2012

Simon Meier

Talk: A guided tour through the bytestring library

Yesterday, I held a talk about the bytestring library at the Zurich HaskellerZ meetup group. The goal of the talk was to enable the audience to write efficient code based on the bytestring library; i.e., explain enough about internals of the library and GHC such that they can judge the cost of the operations on bytestrings both in terms of time and space. The talk also covers the new bytestring builder (based on blaze-builder), which is currently under review for inclusion in the next release of the bytestring library.

Here are the slides and their corresponding handout version for interested readers.

by Simon Meier (noreply@blogger.com) at January 20, 2012 11:28 PM

João Paulo Pizani Flor

Haskell Kata – Conway’s Game of Life

Hi everybody! Today I am going to write about another Code Kata experience I had using Haskell. This time the code didn’t come directly from a Coding Dojo (live meeting with friends), but rather from an “enhancement” over the solution we wrote at a meeting.

In that Coding Dojo we wanted to solve the “Game Of Life” problem. The goal was mainly to – given an initial configuration – evolve the world in one generation and print back this evolved state. For those of you who never heard about Conway’s Game of Life, its article on Wikipedia can be very clarifying.

So we did it, and we solved the problem, but – as always – I wanted very badly to try to solve the same problem using Haskell (instead of Python). At home, I coded it up in Haskell in around 30min, and the code was concise and elegant… In fact, this was so easy that I decided I needed some bigger challenge, something graphical, why not? Then I gave the awesome Gloss graphics library another chance, and I was AGAIN impressed with the results.

So let’s start our Haskell Kata routine, by first describing the problem itself and then the beautiful Haskell solution…

Problem Description

(This problem description is blatantly copied from the correspondent Coding Dojo Wiki entry, as of 2012-01-19)

This Kata is about calculating the next generation of Conway’s Game of Life, given any starting position. Take a look at Wikipedia for background.

You start with a two-dimensional grid of cells, where each cell is either alive or dead. In this version of the problem, the grid is finite, and no life can exist off the edges. When calculating the next generation of the grid, follow these rules:

  • Any live cell with fewer than two live neighbours dies, as if caused by underpopulation.
  • Any live cell with more than three live neighbours dies, as if by overcrowding.
  • Any live cell with two or three live neighbours lives on to the next generation.
  • Any dead cell with exactly three live neighbours becomes a live cell.

You should write a program that can accept an arbitrary grid of cells, and will output a similar grid showing the next generation.

Input and Output

The input starting position is given through a text file that looks like this:

........
....*...
...**...
........

And the output could look like:

........
...**...
...**...
........

This format was chosen because it is very easily parsed in most languages. OBS: It is VERY, VERY easily parsed in Haskell :)

 

Proposed “solution” in Haskell

SPOILER ALERT: DO NOT KEEP READING if you want to try solving this problem for yourself.

As I said in the beginning of the post, my code isn’t really a solution to the given problem, but rather an improvement over it. It improves a simple solution in two ways:

  1. It handles more than one generation. It is actually a live simulation
  2. It has a graphical interface to display the simulation

Even though the code is bigger (because of these enhancements) it is still very concise and readable, totaling some 150 lines of code (with lots of whitespace). And to keep it simple, we are going to start our tour of the solution by the module that deals with the “original” problem: we are going to look first at the functions to read and evolve the board for one generation. First of all, the parser for a starting position (imports omitted for brevity):

dead = fmap (const False) (char '.')
alive = fmap (const True) (char 'O')
line = many1 (dead  alive)
 
board ∷ Parser [[Bool]]
board = line `endBy1` newline
 
parseBoardFromFile ∷ FilePath → IO [[Bool]]
parseBoardFromFile filename =
    do  result ← parseFromFile board filename
        return $ either (errorshow) id result

Our parser was written using the easy-to-use, efficient and popular Parsec library for Haskell, that allows us to describe a format to be recognized (a grammar) in Haskell itself, in a very readable and concise way. Parsec is a parser combinator library, which means that we build our “big” parser by combining smaller parsers, which are built by combining even smaller ones, and so on… until we reach the primitive parsers that come with Parsec like, for example, “char”, used to define the parser for a dead cell in the first line of the code above.

Indeed, the parsers “dead” (for a dead cell) and “alive” (for a live cell) are the most fundamental, and what they do is pretty simple: they map a dot or an upper-case ‘O’ to False or True, respectively. We can take a look at the type signatures of “char” and “fmap” if we want to know more precisely what they do:

fmap(a → b) → f a → f b
char ∷ Char → Parser Char
...
map(a → b)[a][b]

First of all, let’s talk about “char”. The exact type of “char” (as in the Parsec documentation) is more general and uses type classes. The type shown above, however, is a valid specialization of the general type, suiting our usage. You can see that “char” takes a character and gives us a Parser that recognizes that character. However, we want “dead” to be of type Parser Bool, that is, a parser that returns a boolean; so we need a function that converts Parser Char to Parser Bool… And that’s the point where “fmap” comes to help us!

You can see that not only the name, but also the type of “fmap” is very similar to that of “map”. In fact, “fmap” is a generalization of “map”: while “map” works only over lists, “fmap” works over any container. Well, then “fmap” is applicable to our situation, that’s because anything of type Parser a is a monad, and any monad is a container (the fancy word for container is “Functor”). The last interesting detail is the function that we map over (char ‘.’) and (char ‘O’): The usage of “const” means that we don’t care about what the Parser returns, as long as it succeeds, we return either a True or a False.

The remaining parser code is even simpler; the top-level function in the parser module (“parseBoardFromFile”) is the only one that still deserves some commentary:

parseBoardFromFile ∷ FilePath → IO [[Bool]]
parseBoardFromFile filename =
    do  result ← parseFromFile board filename
        return $ either (errorshow) id result

This function takes a file name as input (FilePath is just a synonym for String) and performs some IO actions (reading the file and parsing its contents). As a result of these actions, it returns a matrix of booleans (that is, our Board \o/). The function “parseFromFile” comes from Parsec, and it does exactly what its name says, with a return type of Either ParseError [[Bool]]. Because “result” has this type, we use the “either” function in the next line. The “either” function is used whenever you need to decide what to do based on a value of type Either. In our case, if the result is a correctly parsed matrix, we return it as-is (that’s what “id” does for us). But if it’s an erroneous parse, then we convert the ParseError to a string and terminate the program showing the error message (that’s what “(error ∘ show)” does for us).

Enough for parsing now, and on to what really matters: evolving a board one generation. We take care of this task in the Evolution module. The most important functions of this module are “evolve” and “eval”. “eval” is given a position on the board and returns whether the cell in that position should be dead or alive, based on its surroundings. “evolve” is given a board and just applies “eval” all over it to create the next-generation board. With one important consideration: it only looks at the inner cells in the board, i.e, all border cells are considered always dead. Here’s the code for “eval” and “evolve”:

evolve ∷ Board → Board
evolve b = dummyLine : newKernel ++ [dummyLine]
    where
        (ls,cs) = (length b, length $ head b)
        (innerLs,innerCs) = ([1 .. ls-2], [1 .. cs-2])
        dummyLine = replicate cs False
        dummyCel  = False
        newKernel = [dummyCel : [eval b l c | c ← innerCs] ++ [dummyCel]  |  l ← innerLs]
 
eval ∷ Board → IntIntBool
eval b l c
    | n &lt; 2 || n &gt; 3  = False
    | n == 3          = True
    | otherwise       = b !! l !! c
        where n = aliveNeighbours b l c

The function “aliveNeighbours” takes a board and a pair of coordinates, returning how many of the 8 surrounding cells are alive. Its definition is pretty boring and straightforward, so I won’t explain it – you’ll have to believe that it works :)

So, the code that we saw so far almost solves the original Kata problem – we only needed to add some pretty-printing of the board… BUT! But we are going to do something much nicer, much cooler: we are going to display our simulation in a graphical interface, like this:

The "Gosper glider gun" o/o/o/

Before getting too excited, I have to warn you: the module dealing with drawing the graphics is the largest one (It has around 50 lines of code). However, if you run the program for yourself and watch the AWESOME resulting simulation, you’ll realize that 50 lines is a very good mark :)

Anyways, 50 lines is still too much to be embedded in a blog post, so I’ll only give you the highlights – namely, the functions “drawModel”, “layout” and “activate”. Here’s their code:

drawModel ∷ Board → Picture
drawModel b = (layoutY ∘ layoutX) $ activate b $ rawMatrix
    where
        (l,c) = (length b, length $ head b) 
        rawMatrix = replicate l $ replicate c $ cell
        layoutX = map (pictures ∘ layout DirX)
        layoutY = pictures ∘ layout DirY
 
layout ∷ LayoutDirection → [Picture][Picture]
layout dir pics = [move dir middle i p | (i,p)zip [0..(n-1)] pics]
    where
        n = length pics
        middle = fromIntegral $ floor (fromIntegral n / 2)
 
activate ∷ Board → [[Picture]][[Picture]]
activate board = applyFunctionMatrix (paintersFromBoard board)
    where
        paintersFromBoard = map (map painter)
        painter active = if active then paintActive else paintInactive

Let’s begin with the easiest one: “activate” takes a matrix of pictures (all of them are squares) and assigns a color to each one, with the color depending on whether that position of the matrix is active or not in the underlying Game of Life board. The way in which we implemented “activate” is VERY typical in functional programming, and benefits largely from lazy evaluation. First, we apply “paintersFromBoard”, which has type:

paintersFromBoard ∷ [[Bool]][[Picture → Picture]]

It takes the board and transforms it into a matrix of functions. More specifically, each function in this matrix is a painter, transforming a picture into a new one by changing its color. Now we have two things to combine:

  1. A matrix of default-colored squares (by default they are all black)
  2. A matrix of “painter functions”, one for each square

To combine these, we use “applyFunctionMatrix” (not shown above). Its definition is very straightforward, but also very typical of functional programming languages. We define “applyFunctionMatrix” by “lifting” the usage of “applyFunctionList” to the outer list of the matrix (the list of lines). “applyFunctionList” is beautifully defined as follows:

applyFunctionList ∷ [[a → b]][[a]][[b]]
applyFunctionList fs xs = [f x | (f,x)zip fs xs]

Continuing in the Drawing module, we reach “layout”. The purpose of this function is to – given a list of “unplaced” pictures (all overlapping each other) – spread them with some padding between each other so that they fill a horizontal or vertical line. We want the resulting “array” of pictures to be centered around the point where all overlapping pictures currently are – that’s why we calculate the “middle” index as being half of the list’s length. We then use “move” to displace each picture by “i” steps in direction “dir”. The “move” function knows the size of the cells, so we only need to pass it how many “steps” we want to displace a cell, and a pixel-exact translation will be done for us.

At last, the most important function in the drawing module – the core of the simulation: “drawModel”. Its type, Board → Picture, already tells us how much important it is. At each simulation step, the model is transformed by a call to “evolve”; then, our function “drawModel” is called by the Gloss Simulation Engine to render to model into a Picture. The definition is a simple pipeline with the following steps:

  1. Create a l × c matrix of same-sized squares – by default they are all black and placed at (0,0)
  2. Paint the squares according to the model
  3. Place them – equally spaced – in a nice grid

Ladies and gentlemen, having understood how to do a simulation step (“evolve”) and how to display the model (“drawModel”), we are almost DONE. I am only going to show you how the “simulate” function from Gloss connects these pieces together. Our main function is basically just a call to “simulate”, like this:

main = 
    do  args ← getArgs
        let fps = read (head args)
        -- reads input here
        simulate display bg fps initial drawModel (\_ _ → evolve)
 
bg = greyN 0.95
display = InWindow "Conway's Game of Life" windowSize windowPosition

The “simulate” function takes 6 parameters, and while it might seem too much, they all actually make sense:

  1. display: defines how gloss is going to show the simulation – the size and position of the window, etc.
  2. bg: background color for the drawing area
  3. fps: how many simulation steps happen in one second of real time
  4. initial: the initial model (that we read from the file)
  5. drawModel: a function that transforms the model into a Picture
  6. step function: a function to advance the model one iteration. We use the lambda to ignore some parameters that are irrelevant for us

THAT’S IT! TADA! Now we are done. As always, you can download the cabalized package with the source code GlossGameOfLife, build and run it as follows:

$ tar xzf GlossGameOfLife.tar.gz
$ cd GlossGameOfLife
$ cabal-dev install

Have fun!

flattr this!

by João Paulo at January 20, 2012 07:42 PM

Lee Pike

Who’s Afraid of Software?

Who’s afraid of software?  I mean viscerally, stomach-knotting afraid.  Afraid like you might be when you come across a snake or a bear, or when you are mugged.  Do you obsess about a phishing attack each time you open your email?  Do you worry there’s an eavesdropper when you join the open wifi access point in a coffee shop?  Do you worry your software will fail in your modern automobile or aircraft?

I listened to a Freakonomics podcast about risk, uncertainty, and beliefs.  One point made during the show was that our fears are shaped by evolution—to our ancestors, it made sense to be afraid of threatening animals.  In modern life, however, our fears don’t match risks—we’d be much better off being afraid of cheeseburgers, as pointed out in the show.  Some people are afraid of modern risks.  I know people afraid of cancer, for example.

That got me thinking about fearing software.  Software is certainly among the most complex artifacts created by humans.  Modern cars contain 100+ million lines of code.  Nearly every day there is a story about a large corporation being hacked and of cyber-warfare between nations.

My question is serious—I really do wonder if people are genuinely afraid of software.  I work in the area of software assurance, and while I take precautions against viruses, phishing attacks, etc., I don’t particularly worry about software failures, even when my life might depend on it.  This is despite issues just last year like this and this in automotive software.  I get to see a somewhat how the sausage is made, and in general, we only exercise a small fraction of the state-space of deployed software in validation and in actual usage.  There are legitimate risks, but there seems to be very little fear.

Perhaps like a medical doctors stereotypically neglecting their own health, I don’t worry day-to-day about software assurance despite working in the field.  But it seems nobody else really fears software, either.

In the podcast, the topic of polarizing claims, like global warming, is discussed.  Outside of academic circles, one’s view on the risks of software are not so polarizing—your views on the topic won’t cause your friends or colleagues to disparage you (indeed, if anything, the main risk is likely boring others in discussing the topic!).  I wonder just what the “global warming” of software might be in the future.


by Lee Pike at January 20, 2012 06:46 AM

Paul Chiusano

Possible projects for Boston Haskell Hackathon: smarter evaluation strategies and refactoring combinators

This weekend I'm going to be participating in the Boston Haskell hackathon. I'm very excited about it and I have a couple idea for projects to work on. If any of these sound interesting or you are thinking of something similar, I'm looking for people to collaborate with! Send me an email or come talk to me in person! I think I'm going to get there sometime in the late afternoon to early evening on Friday and I'll be around all weekend.

Evaluation strategies for Haskell that don't leak space

The first project is doing some research / prototyping of an alternate evaluation strategy with the same termination properties as normal order evaluation, but with much easier reasoning about space usage. For lack of a better name, I'm calling it specializing, strictness-propagating evaluation. In this model, calling a function is something like two communicating coroutines. When calling a function, the callee begins evaluating its body, yielding control back to the caller when it needs its first argument, and also indicating whether that argument should be strict or lazily passed, using whatever information is available at runtime. Subsequent arguments work similarly. As a result, functions are automatically specialized as arguments are passed, and we do not construct thunks if they are going to be consumed strictly by a subsequent callee. This can be implemented efficiently using just two call stacks, and there are various optimizations to the scheme. It is intended to augment, not replace, the existing static strictness analysis and argument passing.

Here's an example working through this for the if function, which let's assume has the following implementation:

foldl f z l = case l of 
[] -> z
h:t -> foldl t f (f z h)

I'm also going to assume we've done some static strictness analysis to determine that all branches evaluate z and that therefore the h:t branch evaluates f (since all branches evaluate z and in the h:t case, f appears at the head of an expression passed as z). Suppose we call this with foldl (+) 0 [1,2,3,4].

  1. Caller pushes foldl onto the call stack. foldl begins evaluating with no arguments. It gets as far as the case l. It will then request l strictly, since it is about to evaluated it anyway.
  2. To request the argument, foldl pops its currently running frame from the call stack and pushes it onto the save stack. It then resumes the caller now at the top of the call stack with an argument of strict.
  3. The caller passes the argument as requested - if the caller were itself receiving these arguments as function parameters, it would propagate the strictness request of foldl to its caller.
  4. To resume foldl, it pops foldl from the save stack and pushes it onto the call stack, giving it the (strictly evaluated) list it requested.
  5. Now we hit the interesting case: inside the h:t branch, we know that z is strict (this is known statically). We also know that f can now be evaluated, so we request this argument strictly from our caller. With f now evaluated, we can propagate its stricness information. We know we will be evaluating f z h - what we did not know until runtime was that f was plus (let's just say it was + specialized to Int), and therefore static SA has no choice but to pass f z h as a thunk. We now know that f is strict in both its arguments, so the call to f z h means we can fully evaluate z (which we do by requesting z strictly from our caller), h, and then f z h.

Each step of the iteration works similarly and foldl ends up running in constant space. I'm handwaving a lot here, but in general I want an evaluation order that is totally predictable in its space usage - values are immediately forced as soon as their consuming functions are known at runtime. The consuming functions tell us if an argument will ultimately be forced so we find out sooner rather than building up enormous thunks.

This needs some serious whiteboarding, but assuming it is at all sensible, here's what I propose doing:

  • Come up with an instruction set for this evaluation model, and write a simple interpreter for it
  • Write a compiler for a toy functional language to this instruction set, including the basic static analysis needed to kickstart the dynamic analysis
  • Try writing some programs with it

Some other interesting ideas - I wonder if there's some way to embed this evaluation model in GHC itself.

A code database for Haskell and refactoring combinators

The other project I'm interested in working on is a code database for Haskell, and a Datalog interpreter to go with it. Using this database and the datalog query language, I then want to implement a set of refactoring combinators. A "refactoring" is simply a compilation-preserving function from one code database to another. I've started tinkering with a set of combinators that individually preserve compilation and can be composed to allow arbitrary code transformations. I wrote up some ideas for that here:

... Refactoring times in this new model will go from weeks or months to hours, and writing code to transform a codebase will become a separate but critical skill, distinct from the usual act of programming. That is, programmers do not simply conceive of a refactoring (which is often quite simple to express to another programmer), then begin a tedious, manual and error-prone process of text munging to implement it. Instead, the programmer conceives of a refactoring, then conceives of a code transforming program to implement the refactoring, then applies this transformation to the code database, all in the span of a few hours.

... First, I am not advocating for datalog syntax. I don't care about that. The key functionality enabled by datalog over and above the relational algebra is the ability to express transitive closure and mutual recursion guaranteed to terminate. Together these features enable many of the common queries we'd like to express in transforming and querying our codebases. For instance, here is a hypothetical query to find all references to a given function id, fid. Don't worry if the syntax looks alien or doesn't make sense. The key is more that this query is just a few lines of code to express, and it can be reused and built upon.

-- propagate reference to containing apply
refs(Id) :- apps(Id, fid, _).
refs(Id) :- apps(Id, _, fid).
refs(Id) :- refs(X), apps(Id,X,_).
refs(Id) :- refs(X), apps(Id,_,X).
-- any lambda whose body is or contains fid
-- is considered to reference fid
return(Id) :- lambdas(Id,_,Id1), refs(Id1).
return(Id) :- lambdas(Id,_,fid).

Much of the analysis required to implement refactorings has this sort of "transitive-closure" feel to it - you need to do something to the "direct" callers, then do some transformation for their callers as necessary, and so on.

Here's what I propose for this project:

  • Implement datalog, possibly backed by just in-memory data structures, or maybe tied to something like SQLite. Or if there's an existing free datalog interpreter and backend for it somewhere, let's see if we can use that.
  • Come up with the normalized datalog representation for the Haskell AST and type information - besides just the AST I think you'll need to know all the type information. Is there some way to use the GHC API to get the type of all expressions in the
  • Implement or steal a Haskell parser, and write code to translate this to the normalized datalog representation. As a proof of concept, take some existing Haskell project and "code-database-ify" it.
  • Come up with a good set of refactoring combinators. Implement them using datalog. As a proof of concept, use the combinators to express some nontrivial refactoring (like - make this value monadic rather than pure, and propagate the change in calling convention to all direct and indirect callers as needed - this is exactly the sort of refactoring that is trivial to describe to another Haskell programmer, and is totally mechanical, but is still done via a tedious process of text munging)

If all this is too much, I propose not doing this for Haskell but instead for a toy functional language with a very simple AST and type system.

by Paul Chiusano (noreply@blogger.com) at January 20, 2012 01:07 AM

January 19, 2012

Keegan McAllister

Embedding GDB breakpoints in C source code

Have you ever wanted to embed GDB breakpoints in C source code?

int main() {
printf("Hello,\n");
EMBED_BREAKPOINT;
printf("world!\n");
EMBED_BREAKPOINT;
return 0;
}

One way is to directly insert your CPU's breakpoint instruction. On x86:

#define EMBED_BREAKPOINT  asm volatile ("int3;")

There are at least two problems with this approach:

  • They aren't real GDB breakpoints. You can't disable them, count how many times they've been hit, etc.

  • If you run the program outside GDB, the breakpoint instruction will crash your process.

Here is a small hack which solves both problems:

#define EMBED_BREAKPOINT \
asm("0:" \
".pushsection embed-breakpoints;" \
".quad 0b;" \
".popsection;")

We place a local label into the instruction stream, and then save its address in the embed-breakpoints linker section.

Then we need to convert these addresses into GDB breakpoint commands. I wrote a tool that does this, as a wrapper for the gdb command. Here's how it works, on our initial example:

$ gcc -g -o example example.c

$ ./gdb-with-breakpoints ./example
Reading symbols from example...done.
Breakpoint 1 at 0x4004f2: file example.c, line 8.
Breakpoint 2 at 0x4004fc: file example.c, line 10.
(gdb) run
Starting program: example
Hello,

Breakpoint 1, main () at example.c:8
8 printf("world!\n");
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y 0x00000000004004f2 in main at example.c:8
breakpoint already hit 1 time
2 breakpoint keep y 0x00000000004004fc in main at example.c:10

If we run the program normally, or in GDB without the wrapper, the EMBED_BREAKPOINT statements do nothing. The breakpoint addresses aren't even loaded into memory, because the embed-breakpoints section is not marked as allocatable.

You can find all of the code on GitHub under a BSD license. I've done only minimal testing, but I hope it will be a useful debugging tool for someone. Let me know if you find any bugs or improvements. You can comment here, or find my email address on GitHub.

I'm not sure about the decision to write the GDB wrapper in C using BFD. I also considered Haskell and elf, or Python and the new pyelftools. One can probably do something nicer using the GDB Python API, which was added a few years ago.

This code depends on a GNU toolchain: it uses GNU C extensions, GNU assembler syntax, and BFD. The GDB wrapper uses the Linux proc filesystem, so that it can pass to GDB a temporary file which has already been unlinked. You could port it to other UNIX systems by changing the tempfile handling. It should work on a variety of CPU architectures, but I've only tested it on 32- and 64-bit x86.

by keegan (noreply@blogger.com) at January 19, 2012 05:51 PM

Jeremy Gibbons

Comprehensions

Prompted by some recent work I’ve been doing on reasoning about monadic computations, I’ve been looking back at the work from the 1990s by Phil Trinder, Limsoon Wong, Leonidas Fegaras, Torsten Grust, and others, on monad comprehensions as a framework for database queries.

The idea goes back to the adjunction between extension and intension in set theory—you can define a set by its extension, that is by listing its elements:

\displaystyle  \{ 1, 9, 25, 49, 81 \}

or by its intension, that is by characterizing those elements:

\displaystyle  \{ n^2 \mid 0 < n < 10 \land n \equiv 1 (\mathop{mod} 2) \}

Expressions in the latter form are called set comprehensions. They inspired a programming notation in the SETL language from NYU, and have become widely known through list comprehensions in languages like Haskell. The structure needed of sets or of lists to make this work is roughly that of a monad, and Phil Wadler showed how to generalize comprehensions to arbitrary monads, which led to the “do” notation in Haskell. Around the same time, Phil Trinder showed that comprehensions make a convenient database query language. The comprehension notation has been extended to cover other important aspects of database queries, particularly aggregation and grouping. Monads and aggregations have very nice algebraic structure, which leads to a useful body of laws to support database query optimization.

List comprehensions

Just as a warm-up, here is a reminder about Haskell’s list comprehensions.

\displaystyle  [ 2 \times a + b \mid a \leftarrow [1,2,3] , b \leftarrow [4,5,6] , b \mathbin{\underline{\smash{\mathit{mod}}}} a == 0 ]

This (rather concocted) example yields the list of all values of the expression {2 \times a + b} as {a} is drawn from {[1,2,3]} and {b} from {[4,5,6]} and such that {b} is divisible by {a}, namely {[6,7,8,8,10,12]}.

To the left of the vertical bar is the term (an expression). To the right is a comma-separated sequence of qualifiers, each of which is either a generator (of the form {a \leftarrow x}, with a variable {a} and a list expression {x}) or a filter (a boolean expression). The scope of a variable introduced by a generator extends to all subsequent generators and to the term. Note that, in contrast to the mathematical inspiration, bound variables need to be generated from some existing list.

The semantics of list comprehensions is defined by translation; see for example Phil Wadler’s Chapter 7 of The Implementation of Functional Programming Languages. It can be expressed equationally as follows:

\displaystyle  \begin{array}{lcl} [ e \mid \epsilon ] &=& [e] \\ {} [ e \mid b ] &=& \mathbf{if}\;b\;\mathbf{then}\;[ e ]\;\mathbf{else}\;[\,] \\ {} [ e \mid a \leftarrow x ] &=& \mathit{map}\,(\lambda a \mathbin{.} e)\,x \\ {} [ e \mid q, q' ] &=& \mathit{concat}\,[ [ e \mid q' ] \mid q ] \end{array}

(Here, {\epsilon} denotes the empty sequence of qualifiers. It’s not allowed in Haskell, but it is helpful in simplifying the translation.)

Applying this translation to the example at the start of the section gives

\displaystyle  \begin{array}{ll} & [ 2 \times a + b \mid a \leftarrow [1,2,3] , b \leftarrow [4,5,6] , b \mathbin{\underline{\smash{\mathit{mod}}}} a == 0 ] \\ = & \mathit{concat}\,(\mathit{map}\,(\lambda a \mathbin{.} \mathit{concat}\,(\mathit{map}\,(\lambda b \mathbin{.} \mathbf{if}\;b \mathbin{\underline{\smash{\mathit{mod}}}} a == 0\;\mathbf{then}\;[2 \times a + b]\;\mathbf{else}\;[\,])\,[4,5,6]))\,[1,2,3]) \\ = & [6,7,8,8,10,12] \end{array}

More generally, a generator may match against a pattern rather than just a variable. In that case, it may bind multiple (or indeed no) variables at once; moreover, the match may fail, in which case it is discarded. This is handled by modifying the translation for generators to use a function defined by pattern-matching, rather than a straight lambda-abstraction:

\displaystyle  [ e \mid p \leftarrow x ] = \mathit{concat}\,(\mathit{map}\,(\lambda a \mathbin{.} \mathbf{case}\;a\;\mathbf{of}\;p \rightarrow [ e ] \;;\; \_ \rightarrow [\,])\,x)

or, more perspicuously,

\displaystyle  [ e \mid p \leftarrow x ] = \mathbf{let}\;h\,p = [ e ] ; h\,\_ = [\,]\;\mathbf{in}\; \mathit{concat}\,(\mathit{map}\,h\,x)

Monad comprehensions

It is clear from the above translation that the necessary ingredients for list comprehensions are {\mathit{map}}, singletons, {\mathit{concat}}, and the empty list. The first three are the operations arising from lists as a functor and a monad, which suggests that the same translation might be applicable to other monads too. But the fourth ingredient, the empty list, does not come from the functor and monad structures; that requires an extra assumption:

\displaystyle  \begin{array}{ll} \mathbf{class}\;\mathit{Monad}\,m \Rightarrow \mathit{MonadZero}\,m\;\mathbf{where} \\ \quad \mathit{mzero} :: m\,a \end{array}

Then the translation for list comprehensions can be generalized to other monads:

\displaystyle  \begin{array}{lcl} [ e \mid \epsilon ] &=& \mathit{return}\,e \\ {} [ e \mid b ] &=& \mathbf{if}\;b\;\mathbf{then}\;\mathit{return}\,e\;\mathbf{else}\;\mathit{mzero} \\ {} [ e \mid p \leftarrow m ] &=& \mathbf{let}\;h\,p = \mathit{return}\,e ; h\,\_ = \mathit{mzero}\;\mathbf{in}\; \mathit{join}\,(\mathit{map}\,h\,m) \\ {} [ e \mid q, q' ] &=& \mathit{join}\,[ [ e \mid q' ] \mid q ] \end{array}

(so {[ e \mid \epsilon ] = [ e \mid \mathit{True} ]}). The actual monad to be used is implicit; if we want to be explicit, we could use a subscript, as in “{[ e \mid q ]_\mathsf{List}}“.

This translation is different from the one used in the Haskell language specification, which to my mind is a little awkward: the empty list crops up in two different ways in the translation of list comprehensions—for filters, and for generators with patterns—and these are generalized in two different ways to other monads (to the {\mathit{mzero}} method of the {\mathit{MonadPlus}} class in the first case, and the {\mathit{fail}} method of the {\mathit{Monad}} class in the second). I think it is neater to have a monad subclass {\mathit{MonadZero}} with a single method subsuming both these operators. Of course, this does mean that the translation forces a monad comprehension with filters or possibly failing generators to be interpreted in a monad in the {\mathit{MonadZero}} subclass rather than just {\mathit{Monad}}—the type class constraints that are generated depend on the features used in the comprehension. (Perhaps this translation was tried in earlier versions of the language specification, and found wanting?)

Taking this approach gives basically the monad comprehension notation from Wadler’s Comprehending Monads paper; it loosely corresponds to Haskell’s do notation, except that the term is to the left of a vertical bar rather than at the end, and that filters are just boolean expressions rather than introduced using {\mathit{guard}}.

We might impose the law that {\mathit{mzero}} is a “left” zero of composition, in the sense

\displaystyle  \mathit{join}\,\mathit{mzero} = \mathit{mzero}

or, in terms of comprehensions,

\displaystyle  [ e \mid a \leftarrow \mathit{mzero} ] = \mathit{mzero}

Informally, this means that any failing steps of the computation cleanly cut off subsequent branches. Conversely, we do not require that {\mathit{mzero}} is a “right” zero too:

\displaystyle  \mathit{join}\,(\mathit{map}\,(\lambda a \mathbin{.} \mathit{mzero})\,m) \ne \mathit{mzero} \quad\mbox{(in general)}

This would have the consequence that a failing step also cleanly erases any effects from earlier parts of the computation, which is too strong a requirement for many monads—particularly those of the “launch missiles now” variety. (The names “left-” and “right zero” make more sense when the equations are expressed in terms of the usual Haskell bind operator {(\gg\!=)}, which is a kind of sequential composition.)

Ringads and collection classes

One more ingredient is needed in order to characterize monads that correspond to “collection classes” such as sets and lists, and that is an analogue of set union or list append. It’s not difficult to see that this is inexpressible in terms of the operations introduced so far: given only collections {m} of at most one element, any comprehension using generators of the form {a \leftarrow m} will only yield another such collection, whereas the union of two one-element collections will in general have two elements.

To allow any finite collection to be expressed, it suffices to introduce a binary union operator {\uplus}:

\displaystyle  \begin{array}{ll} \mathbf{class}\;\mathit{Monad}\,m \Rightarrow \mathit{MonadPlus}\,m\;\mathbf{where} \\ \quad (\uplus) :: m\,a \times m\,a \rightarrow m\,a \end{array}

We require composition to distribute over union, in the following sense:

\displaystyle  \mathit{join}\,(m \uplus n) = \mathit{join}\,m \uplus \mathit{join}\,n

or, in terms of comprehensions,

\displaystyle  [ e \mid a \leftarrow m \uplus n, q ] = [ e \mid a \leftarrow m, q ] \uplus [ e \mid a \leftarrow n, q ]

For the remainder of this post, we will assume a monad in both {\mathit{MonadZero}} and {\mathit{MonadPlus}}. Moreover, we will assume that {\mathit{mzero}} is the unit of {\uplus}, and is both a left- and a right zero of composition. To stress the additional constraints, we will write “{\emptyset}” for “{\mathit{mzero}}” from now on. The intention is that such monads exactly capture collection classes; Phil Wadler has called these structures ringads. (He seems to have done so in an unpublished note Notes on Monads and Ringads from 1990, which is cited by some papers from the early 1990s. But Phil no longer has a copy of this note, and it’s not online anywhere… I’d love to see a copy, if anyone has one!)

\displaystyle  \begin{array}{ll} \mathbf{class}\;(\mathit{MonadZero}\,m, \mathit{MonadPlus}\,m) \Rightarrow \mathit{Ringad}\,m\;\mathbf{where} \end{array}

(There are no additional methods; the class {\mathit{Ringad}} is the intersection of the two parent classes {\mathit{MonadZero}} and {\mathit{MonadPlus}}, with the union of the two interfaces, together with the laws above.) I used roughly the same construction already in the post on Horner’s Rule.

As well as (finite) sets and lists, ringad instances include (finite) bags and a funny kind of binary tree (externally labelled, possibly empty, in which the empty tree is a unit of the binary tree constructor). These are all members of the so-called Boom Hierarchy of types—a name coined by Richard Bird, for an idea due to Hendrik Boom, who by happy coincidence is named for one of these structures in his native language. All members of the Boom Hierarchy are generated from the empty, singleton, and union operators, the difference being whether union is associative, commutative, and idempotent. Another ringad instance, but not a member of the Boom Hierarchy, is the type of probability distributions—either normalized, with a weight-indexed family of union operators, or unnormalized, with an additional scaling operator.

Aggregation

The well-behaved operations over monadic values are called the algebras for that monad—functions {k} such that {k \cdot \mathit{return} = \mathit{id}} and {k \cdot \mathit{join} = k \cdot \mathit{map}\,k}. In particular, {\mathit{join}} is itself a monad algebra. When the monad is also a ringad, {k} necessarily distributes also over {\uplus}—there is a binary operator {\oplus} such that {k\,(m \uplus n) = k\,m \oplus k\,n} (exercise!). Without loss of generality, we write {\oplus/} for {k}; these are the “reductions” of the Bird–Meertens Formalism. In that case, {\mathit{join} = \uplus/} is a ringad algebra.

The algebras for a ringad amount to aggregation functions for a collection: the sum of a bag of integers, the maximum of a set of naturals, and so on. We could extend the comprehension notation to encompass aggregations too, for example by adding an optional annotation, writing say “{[ e \mid q ]^\oplus}“; although this doesn’t add much, because we could just have written “{\oplus/\,[e \mid q]}” instead. We could generalize from reductions {\oplus/} to collection homomorphisms {\oplus/ \cdot \mathit{map}\,f}; but this doesn’t add much either, because the map is easily combined with the comprehension—it’s easy to show the “map over comprehension” property

\displaystyle  \mathit{map}\,f\,[e \mid q] = [f\,e \mid q]

Leonidas Fegaras and David Maier develop a monoid comprehension calculus around such aggregations; but I think their name is inappropriate, because nothing forces the binary aggregating operator to be associative.

Note that, for {\oplus/} to be well-defined, {\oplus} must satisfy all the laws that {\uplus} does—{\oplus} must be associative if {\uplus} is associative, and so on. It is not hard to show, for instance, that there is no {\oplus} on sets of numbers for which {\mathit{sum}\,(x \cup y) = \mathit{sum}\,x \oplus \mathit{sum}\,y}; such an {\oplus} would have to be idempotent, which is inconsistent with its relationship with {\mathit{sum}}. (So, although {[a^2 \mid a \leftarrow x, \mathit{odd}\,a]_\mathsf{Bag}^{+}} denotes the sum of the squares of the odd elements of bag {x}, the expression {[a^2 \mid a \leftarrow x, \mathit{odd}\,a]_\mathsf{Set}^{+}} (with {x} now a set) is not defined, because {+} is not idempotent.) In particular, {\oplus/\emptyset} must be the unit of {\oplus}, which we write {1_\oplus}.

We can derive translation rules for aggregations from the definition

\displaystyle  [ e \mid q ]^\oplus = \oplus/\,[e \mid q]

For empty aggregations, we have:

\displaystyle  \begin{array}{ll} & [ e \mid \epsilon ]^\oplus \\ = & \qquad \{ \mbox{aggregation} \} \\ & \oplus/\,[ e \mid \epsilon ] \\ = & \qquad \{ \mbox{comprehension} \} \\ & \oplus/\,(\mathit{return}\,e) \\ = & \qquad \{ \mbox{monad algebra} \} \\ & e \end{array}

For filters, we have:

\displaystyle  \begin{array}{ll} & [ e \mid b ]^\oplus \\ = & \qquad \{ \mbox{aggregation} \} \\ & \oplus/\,[ e \mid b ] \\ = & \qquad \{ \mbox{comprehension} \} \\ & \oplus/\,(\mathbf{if}\;b\;\mathbf{then}\;\mathit{return}\,e\;\mathbf{else}\;\emptyset) \\ = & \qquad \{ \mbox{lift out the conditional} \} \\ & \mathbf{if}\;b\;\mathbf{then}\;{\oplus/}\,(\mathit{return}\,e)\;\mathbf{else}\;{\oplus/}\,\emptyset \\ = & \qquad \{ \mbox{ringad algebra} \} \\ & \mathbf{if}\;b\;\mathbf{then}\;e\;\mathbf{else}\;1_\oplus \end{array}

For generators, we have:

\displaystyle  \begin{array}{ll} & [ e \mid p \leftarrow m ]^\oplus \\ = & \qquad \{ \mbox{aggregation} \} \\ & \oplus/\,[ e \mid p \leftarrow m ] \\ = & \qquad \{ \mbox{comprehension} \} \\ & \oplus/\,(\mathbf{let}\;h\,p = \mathit{return}\,e ; h\,\_ = \emptyset\;\mathbf{in}\;\mathit{join}\,(\mathit{map}\,h\,m)) \\ = & \qquad \{ \mbox{lift out the \textbf{let}} \} \\ & \mathbf{let}\;h\,p = \mathit{return},e ; h\,\_ = \emptyset\;\mathbf{in}\;{\oplus/}\,(\mathit{join}\,(\mathit{map}\,h\,m)) \\ = & \qquad \{ \mbox{monad algebra} \} \\ & \mathbf{let}\;h\,p = \mathit{return}\,e ; h\,\_ = \emptyset\;\mathbf{in}\;{\oplus/}\,(\mathit{map}\,(\oplus/)\,(\mathit{map}\,h\,m)) \\ = & \qquad \{ \mbox{functors} \} \\ & \mathbf{let}\;h\,p = \mathit{return}\,e ; h\,\_ = \emptyset\;\mathbf{in}\;{\oplus/}\,(\mathit{map}\,(\oplus/ \cdot h)\,m) \\ = & \qquad \{ \mbox{let \(h' = \oplus/ \cdot h\)} \} \\ & \mathbf{let}\;h'\,p = \oplus/\,(\mathit{return}\,e) ; h'\,\_ = \oplus/\,\emptyset\;\mathbf{in}\;{\oplus/}\,(\mathit{map}\,h'\,m) \\ = & \qquad \{ \mbox{ringad algebra} \} \\ & \mathbf{let}\;h'\,p = e ; h'\,\_ = 1_\oplus\;\mathbf{in}\;{\oplus/}\,(\mathit{map}\,h'\,m) \end{array}

And for sequences of qualifiers, we have:

\displaystyle  \begin{array}{ll} & [ e \mid q, q' ]^\oplus \\ = & \qquad \{ \mbox{aggregation} \} \\ & \oplus/\,[ e \mid q, q' ] \\ = & \qquad \{ \mbox{comprehension} \} \\ & \oplus/\,(\mathit{join}\,[ [ e \mid q'] \mid q ] \\ = & \qquad \{ \mbox{monad algebra} \} \\ & \oplus/\,(\mathit{map}\,(\oplus/)\,[ [ e \mid q'] \mid q ]) \\ = & \qquad \{ \mbox{map over comprehension} \} \\ & \oplus/\,[ \oplus/\,[ e \mid q'] \mid q ] \\ = & \qquad \{ \mbox{aggregation} \} \\ & [ [ e \mid q']^\oplus \mid q ]^\oplus \end{array}

Putting all this together, we have:

\displaystyle  \begin{array}{lcl} [ e \mid \epsilon ]^\oplus &=& e \\ {} [ e \mid b ]^\oplus &=&\mathbf{if}\;b\;\mathbf{then}\;e\;\mathbf{else}\;1_\oplus \\ {} [ e \mid p \leftarrow m ]^\oplus &=& \mathbf{let}\;h\,p = e ; h\,\_ = 1_\oplus\;\mathbf{in}\;{\oplus/}\,(\mathit{map}\,h\,m) \\ {} [ e \mid q, q' ]^\oplus &=& [ [ e \mid q']^\oplus \mid q ]^\oplus \end{array}

Heterogeneous comprehensions

We have seen that comprehensions can be interpreted in an arbitrary ringad; for example, {[a^2 \mid a \leftarrow x, \mathit{odd}\,a]_\mathsf{Set}} denotes (the set of) the squares of the odd elements of (the set) {x}, whereas {[a^2 \mid a \leftarrow x, \mathit{odd}\,a]_\mathsf{Bag}} denotes the bag of such elements, with {x} a bag. Can we make sense of “heterogeneous comprehensions”, involving several different ringads?

Let’s introduced the notion of a ringad morphism, extending the familiar analogue on monads. For monads {\mathsf{M}} and {\mathsf{N}}, a monad morphism {\phi : \mathsf{M} \mathbin{\stackrel{.}{\to}} \mathsf{N}} is a natural transformation {\mathsf{M} \mathbin{\stackrel{.}{\to}} \mathsf{N}}—that is, a family {\phi_\alpha :: \mathsf{M}\,\alpha \rightarrow \mathsf{N}\,\alpha} of arrows, coherent in the sense that {\phi_\beta \cdot \mathsf{M}\,f = \mathsf{N}\,f \cdot \phi_\alpha} for {f :: \alpha \rightarrow \beta}—that also preserves the monad structure:

\displaystyle  \begin{array}{lclcl} \phi \cdot \mathit{return}_\mathsf{M} &=& \mathit{return}_\mathsf{N} \\ \phi \cdot \mathit{join}_\mathsf{M} &=& \mathit{join}_\mathsf{N} \cdot \phi \cdot \mathsf{M}\,\phi &=& \mathit{join}_\mathsf{N} \cdot \mathsf{N}\,\phi \cdot \phi \end{array}

A ringad morphism {\phi : \mathsf{M} \mathbin{\stackrel{.}{\to}} \mathsf{N}} for ringads {\mathsf{M},\mathsf{N}} is a monad morphism {\phi : \mathsf{M} \mathbin{\stackrel{.}{\to}} \mathsf{N}} that also respects the ringad structure:

\displaystyle  \begin{array}{lcl} \phi\,\emptyset_\mathsf{M} &=& \emptyset_\mathsf{N} \\ \phi\,(x \uplus_\mathsf{M} y) &=& \phi\,x \uplus_\mathsf{N} \phi\,y \end{array}

Then a ringad morphism behaves nicely with respect to ringad comprehensions—a comprehension interpreted in ringad {\mathsf{M}}, using existing collections of type {\mathsf{M}}, with the result transformed via a ringad morphism {\phi : \mathsf{M} \mathbin{\stackrel{.}{\to}} \mathsf{N}} to ringad {\mathsf{N}}, is equivalent to the comprehension interpreted in ringad {\mathsf{N}} in the first place, but with the initial collections transformed to type {\mathsf{N}}. Informally, there will be no surprises arising from when ringad coercions take place, because the results are the same whenever this happens. This property is straightforward to show by induction over the structure of the comprehension. For the empty comprehension, we have:

\displaystyle  \begin{array}{ll} & \phi\,[ e \mid \epsilon ]_\mathsf{M} \\ = & \qquad \{ \mbox{comprehension} \} \\ & \phi\,(\mathit{return}_\mathsf{M}\,e) \\ = & \qquad \{ \mbox{ringad morphism} \} \\ & \mathit{return}_\mathsf{N}\,e \\ = & \qquad \{ \mbox{comprehension} \} \\ & [e \mid \epsilon ]_\mathsf{N} \end{array}

For filters, we have:

\displaystyle  \begin{array}{ll} & \phi\,[ e \mid b ]_\mathsf{M} \\ = & \qquad \{ \mbox{comprehension} \} \\ & \phi\,(\mathbf{if}\;b\;\mathbf{then}\;\mathit{return}_\mathsf{M}\,e\;\mathbf{else}\;\emptyset_\mathsf{M}) \\ = & \qquad \{ \mbox{lift out the conditional} \} \\ & \mathbf{if}\;b\;\mathbf{then}\;\phi\,(\mathit{return}_\mathsf{M}\,e)\;\mathbf{else}\;\phi\,\emptyset_\mathsf{M} \\ = & \qquad \{ \mbox{ringad morphism} \} \\ & \mathbf{if}\;b\;\mathbf{then}\;\mathit{return}_\mathsf{N}\,e\;\mathbf{else}\;\emptyset_\mathsf{N} \\ = & \qquad \{ \mbox{comprehension} \} \\ & [ e \mid b ]_\mathsf{N} \end{array}

For generators:

\displaystyle  \begin{array}{ll} & \phi\,[ e \mid p \leftarrow m ]_\mathsf{M} \\ = & \qquad \{ \mbox{comprehension} \} \\ & \phi\,(\mathbf{let}\;h\,p = \mathit{return}_\mathsf{M}\,e ; h\,\_ = \emptyset_\mathsf{M}\;\mathbf{in}\;\mathit{join}_\mathsf{M}\,(\mathit{map}_\mathsf{M}\,h\,m)) \\ = & \qquad \{ \mbox{lift out the \textbf{let}} \} \\ & \mathbf{let}\;h\,p = \mathit{return}_\mathsf{M}\,e ; h\,\_ = \emptyset_\mathsf{M}\;\mathbf{in}\;\phi\,(\mathit{join}_\mathsf{M}\,(\mathit{map}_\mathsf{M}\,h\,m)) \\ = & \qquad \{ \mbox{ringad morphism, functors} \} \\ & \mathbf{let}\;h\,p = \mathit{return}_\mathsf{M}\,e ; h\,\_ = \emptyset_\mathsf{M}\;\mathbf{in}\;\mathit{join}_\mathsf{N}\,(\phi\,(\mathit{map}_\mathsf{M}\,(\phi \cdot h)\,m)) \\ = & \qquad \{ \mbox{let \(h' = \phi \cdot h\)} \} \\ & \mathbf{let}\;h'\,p = \phi\,(\mathit{return}_\mathsf{M}\,e) ; h'\,\_ = \phi\,\emptyset_\mathsf{M}\;\mathbf{in}\;\mathit{join}_\mathsf{N}\,(\phi\,(\mathit{map}_\mathsf{M}\,h'\,m)) \\ = & \qquad \{ \mbox{ringad morphism, induction} \} \\ & \mathbf{let}\;h'\,p = \mathit{return}_\mathsf{N}\,e ; h'\,\_ = \emptyset_\mathsf{N}\;\mathbf{in}\;\mathit{join}_\mathsf{N}\,(\phi\,(\mathit{map}_\mathsf{M}\,h'\,m)) \\ = & \qquad \{ \mbox{naturality of \(\phi\)} \} \\ & \mathbf{let}\;h'\,p = \mathit{return}_\mathsf{N}\,e ; h'\,\_ = \emptyset_\mathsf{N}\;\mathbf{in}\;\mathit{join}_\mathsf{N}\,(\mathit{map}_\mathsf{N}\,h'\,(\phi\,m)) \\ = & \qquad \{ \mbox{comprehension} \} \\ & [ e \mid p \leftarrow \phi\,m ]_\mathsf{N} \end{array}

And for sequences of qualifiers:

\displaystyle  \begin{array}{ll} & \phi\,[ e \mid q, q' ]_\mathsf{M} \\ = & \qquad \{ \mbox{comprehension} \} \\ & \phi\,(\mathit{join}\,[ [ e \mid q' ]_\mathsf{M} \mid q ]_\mathsf{M}) \\ = & \qquad \{ \mbox{ringad morphism} \} \\ & \phi\,(\mathit{map}\,\phi\,[ [ e \mid q' ]_\mathsf{M} \mid q ]_\mathsf{M}) \\ = & \qquad \{ \mbox{map over comprehension} \} \\ & \phi\,[ \phi\,[ e \mid q' ]_\mathsf{M} \mid q ]_\mathsf{M} \\ = & \qquad \{ \mbox{induction} \} \\ & [ [ e \mid q' ]_\mathsf{N} \mid q ]_\mathsf{N} \\ = & \qquad \{ \mbox{comprehension} \} \\ & [ e \mid q, q' ]_\mathsf{N} \end{array}

For example, if {\mathit{bag2set} : \mathsf{Bag} \mathbin{\stackrel{.}{\to}} \mathsf{Set}} is the obvious ringad morphism from bags to sets, discarding information about the multiplicity of repeated elements, and {x} a bag of numbers, then

\displaystyle  \mathit{bag2set}\,[a^2 \mid a \leftarrow x, \mathit{odd}\,a]_\mathsf{Bag} = [a^2 \mid a \leftarrow \mathit{bag2set}\,x, \mathit{odd}\,a]_\mathsf{Set}

and both yield the set of squares of the odd members of {x}. As a notational convenience, we might elide use of the ringad morphism when it is “obvious from context”—we might write just {[a^2 \mid a \leftarrow x, \mathit{odd}\,a]_\mathsf{Set}} even when {x} is a bag, relying on the “obvious” morphism {\mathit{bag2set}}. This would allow us to write, for example,

\displaystyle  [ a+b \mid a \leftarrow [1,2,3], b \leftarrow \langle4,4,5\rangle ]_\mathsf{Set} = \{ 5,6,7,8 \}

(writing {\langle\ldots\rangle} for the extension of a bag), instead of the more pedantic

\displaystyle  [ a+b \mid a \leftarrow \mathit{list2set}\,[1,2,3], b \leftarrow \mathit{bag2set}\,\langle4,4,5\rangle ]_\mathsf{Set} = \{ 5,6,7,8 \}

There is a forgetful function from any poorer member of the Boom hierarchy to a richer one, flattening some distinctions by imposing additional laws—for example, from bags to sets, flattening distinctions concerning multiplicity—and I would class these forgetful functions as “obvious” morphisms. On the other hand, any morphisms in the opposite direction—such as sorting, from bags to lists, and one-of-each, from sets to bags—are not “obvious”, and so should not be elided; and similarly, I’m not sure that I could justify as “obvious” any morphisms involving non-members of the Boom Hierarchy, such as probability distributions.


by jeremygibbons at January 19, 2012 05:03 PM

Jason Dagit

Keeping the "science" in Software Development

If you already feel that math is a science you can safely skip the first section.  What follows are my musings on what it means to be "correct", especially in software.  I pose a challenge at the end for practitioners and researchers.

Math is science too
According to Wikipedia the scientific method is:
Scientific method refers to a body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge.[1] To be termed scientific, a method of inquiry must be based on gathering empirical andmeasurable evidence subject to specific principles of reasoning.[2]
At first glance it may appear that mathematics doesn't follow the scientific method above. The mathematics that you're most likely familiar with is a collection of definitions and the facts, lemmas and theorems, about those definitions that result from applying logical deduction.  You might notice that if you reason that way, then textbook applications of newton's three laws do not follow the scientific method either.

You might argue that in this case the applications of newton's laws is now mathematics and no longer science.  I would agree that it is mathematics and science.  Let's see if I can convince you.

By the time you see the definitions and corresponding theorems in a math text they have been reduced to facts.  The trial and error, as well as correction and integration of previous knowledge, have already happened.  The same is true of a textbook that shows you how to apply newton's three laws.  I recommend taking a moment to familiarize yourself with Lockhart's lament.

Math only seems to not be a science because of our relationship to learning it.  Most of us get taught math as facts.  If you've taken a proof based course you may have had the experience of searching for proofs.  There is a lot of trial and error involved.  Mathematics as a field may have already integrated previous knowledge, but if you are taking a course chances are you have not.  So you employ trial and error as well as hypothesis refinement to build logical deductions to establish what your intuition, or pen and paper examples, says in true.

The set of assumptions that you choose can make or break a result.  Even though mathematical results are timeless, they still bend to the sway of assumptions.  This is exactly what happens to empirical scientists too.  They think they know something, or have the right model, but later it turns out they reasoned from a flawed assumption and suddenly new data transform their field.

Evidence vs Proof
If you've studied proofs in different logical settings you may have noticed that what constitutes a proof depends on what you seek to gain from the proof.  See for example classical logic versus constructive logic.

In a court case or a forensics study, people look for evidence or witnesses to give data points in support of a particular position within a larger case (or claim).

Think about that for a minute.  A claim is made, then people try to establish it or refute it.  Often times, the assertion needs only be accepted or rejected by a "reasonable" skeptic that is willing to accept partial proof.  Such a skeptic might be a jury or peers reviewing a publication for acceptance in a journal.  Because we used partial proof new information could turn up later that reverses our understanding.

At the other extreme we have mathematical proofs, which are composed of logical statements following from assumptions.  Proofs are timeless.  If we get our logic right and we're clear about our assumptions then the result will stand forever.  I say "forever", but in reality, proofs can be found to be incomplete, insufficiently rigorous, or using the wrong logic at a later date.  Sometimes years after the arguments are accepted.

It seems that the difference here isn't in whether the resulting "facts" are immutable but rather the strength with which we intend to make the claim.  In the forensics case, we are making a claim to the best of our incomplete knowledge about what happened.  In the case of mathematical proof, we are making a claim that should be true forever.

In my opinion, the difference between "evidence" and "proof" exists to quantify the strength of our assertion that the claim or conjecture is true.  Claims can roughly be placed on a continuum from weak to strong.  Some claims, such as "a human authored this blog post", are weak in nature and need little evidence.  Other claims, such as "the author of this blog post can see through walls", is a hard to believe as there is good evidence that humans can't see through walls. To prove the second claim I would hope that you demand really strong "evidence".  Perhaps in the form of a demonstration, a scientific model for how I can do it, and validation from peer review.

Due to the continuum for strength of evidence described above, I consider proofs to be a form of evidence.  Proofs, in the mathematical sense, being the strongest form of evidence.  Claims made by politicians would be on the extreme other end of my continuum.

The strength of the evidence for the claim has practical applications too.  I wouldn't engineer a shuttle to the moon based on arguments built by politicians.  I would base my engineering on mathematical proofs as getting to the moon requires a high degree of correctness.

So then, what level of evidence is right if I need to be correct?

Evidence Based Correctness
Now we're really at the heart of the matter.  Sometimes it just doesn't matter if we're correct or not, for example when arguing with a spouse.  Other times it matters deeply, like when we want to engineer a shuttle to the moon.

While mathematics is obsessed with being unequivocally true, other branches of science have been trying to deepen their understanding while accepting a greater risk of being wrong.  This results in a different level of evidence required for progress.

In other words, you have to figure out how important it is to you (or your application) that you know you're correct versus it being reasonable that you are correct.  I like continuums and this is no exception.  I believe that you can build a continuum of how strongly you believe things.  This continuum is a natural dual to the continuum for strength of evidence.

In other words, the bolder your claim is, the stronger the evidence you need.  Unfortunately, no one has a magical formula for knowing when evidence is strong enough.  If we did science would become more precise, court would become more cut and dry, and mathematicians could relax and quantify results in terms of proof strength.

Now wait a second.  Quantify results in terms of proof strength?  That sounds like statistical reasoning doesn't it?  Statisticians are able to cobble together data points and talk about the strength of that evidence in terms of supporting a particular hypothesis.

In software development, testing is probably the cheapest way to have some evidence that your program is correct.  Testing is almost never enough to have a proof of correctness though.  On the other end of the spectrum we have theorem proving, which is known to be costly in terms of human effort.

My point is this, you don't have to force yourself into a staunch mathematical camp to have "correctness" in your software development.  Nor do you have to give up on correctness in order to fit into a budget.  It's not a dichotomy where you are forced to give up on either truth or practicality.  Testing is not "wrong" and proofs are not "required".

What is required is having a level of evidence that is strong enough to support the level of belief we need for our chosen level of correctness.  Currently we have to carefully evaluate where on the continuum of correctness our result rests and then pick a commiserate level of evidence.  You probably do this already intuitively to some degree.  My challenge to you is to do this explicitly and consciously.

Furthermore, can we take the precedent set by the statisticians and find a way to quantify strength of evidence?  Can testing and theorem proving be unified on a continuum by this measure?  Is there a way to look at a successful run of a test suite, that may or may not include theorem proving, and quantify the strength of that evidence that our program is correct?

by dagitj (noreply@blogger.com) at January 19, 2012 06:02 AM

January 18, 2012

Eric Kidd

Best article I've seen on SOPA

Wikipedia, Google and many other internet sites are protesting PIPA and SOPA today. But their official explanations don’t include very many details about the actual legislation.

If you’d like to learn more, check out this excellent background piece by a freelance film editor.

by Eric Kidd at January 18, 2012 07:29 PM

Mihai Maruseac

SOPA

Is this what you want?

Think again please.


by Mithrandir at January 18, 2012 06:28 AM

January 16, 2012

Lee Pike

Meta-Programming and eDSLs

I’ve been working on a domain-specific language that is embedded in Haskell (an “eDSL”) that essentially takes a set of Haskell stream (infinite list) equations and turns them into a real-time C program implementing the state-machine defined by the streams. It’s called Copilot, and in fact, it’s built on top of another Haskell eDSL called Atom,1 which actually does the heavy lifting in generating the C code.

For example, here’s the Fibonacci sequence in Copilot:

fib = do let f = varW64 "f" f .= [0,1] ++ f + (drop 1 f) 

I’ve been writing Copilot libraries recently, and I’ve had the following realization about eDSLs and meta-programming (let me know if someone has had this idea already!). Many languages treat meta-programming as a second-class feature—I’m thinking of macros used by the C preprocessor, for example (it’s probably generous even to call the C preprocessor ‘meta-programming’). One reason why Lisp-like languages were exciting is that the language is a first-class datatype, so meta-programming is on par with programming. In my experience, you don’t think twice about meta-programming in Lisp. (Haskell is more like C in this regard—you do think twice before using Template Haskell.)

So languages generally treat meta-programming as either second-class or first-class. What’s interesting about eDSLs, I think, is that they treat meta-programming as first-class, but programming as second-class! This isn’t surprising, since an eDSL is a first-class datatype, but the language is not first-class—its host language is.

Practically, what this means is that you spend very little time actually writing eDSL programs but rather generating eDSL programs. It is natural to layer eDSLs on top of other eDSLs.

This is just how Copilot came about: I was writing various Atom programs and realized that for my purposes, I just needed a restricted set of behaviors provided by Atom that are naturally represented by stream equations (and make some other tasks, like writing an interpreter, easier).

But as soon as Copilot was written, we2 started writing libraries implementing past-time linear temporal logic (LTL) operators, bounded LTL operators, fault-tolerant voting algorithms, regular expressions, and so on.

You wouldn’t think about combining the intermediate languages of a compiler into the same program. The idea of a language is more fluid, more organic in the context of eDSLs, where now libraries can be quickly written and different levels can be easily combined.

1Tom Hawkins wrote Atom.
2Credit for Copilot also goes to Sebastian Niller, Robin Morisset, Alwyn Goodloe.


by Lee Pike at January 16, 2012 11:40 PM

JP Moresmau

Lego Mindstorms, Haskell, Windows: all set!

A new version of the NXT library has been released, and it supports Windows!! I lamented a few weeks ago that it only ran on Unix/MacOs (requiring the unix package and using Posix file descriptors), but that's a thing of the past. It was really easy to port Mitar's code to use the serialport package instead of file descriptors. Now Mitar has merged back my changes, we've tested the code on Linux and Windows, and it's been released on Hackage!

So now I can control my Lego Mindstorm robot from Haskell on my Windows machine! World domination is only a couple more steps away!

I wish to point out that Mitar's code is extremely well commented and a pleasure to work with, which of course made the adaptation to Windows a lot easier. Thank you!

by JP Moresmau (noreply@blogger.com) at January 16, 2012 10:06 PM

João Paulo Pizani Flor

Haskell Tone Synthesizer – Playing with sound waves

Some time ago, in a cloudy friday afternoon, I had nothing to do. Me and my friends started talking about the Unix philosophy, and in particular, how everything in Unix is a file, even devices are represented as files. So I thought: the sound card is a file! What happens if I write to it? Then we started feeding lots of files (PDFs, /dev/urandom, some source code and some images) to the sound card and where amused :)

But then we decided to have more fun and feed some more “organized” bytes to the sound card – we would generate our audio samples programmatically. And what better tool to do some data generation than Haskel!? So I started writing my first audio synthesizer, and after some hours it was ready and working awesomely… I was really excited with the results, so I just NEEDED to share this nice piece of code with the world :)

My Haskell tone synthesizer takes a “score file” as input, that describes the tune you want the synthesizer to generate. This input file has a syntax very similar to the old Nokia Ringtone Composer syntax… The program reads the score from the file and outputs a lot of bytes to standard output. If you redirect the standard output to your sound card, you will hear your favorite tunes! So let me guide you through a tour of this program’s source code:

This tour will be in top-bottom style: we will start by looking at the Main module and its major processing steps, and then dwell a bit deeper into some of the most important functions of each module. So, to start, here is the entire Main module:

module Main where

import ... -- imports ommited for brevity

main =
    do  args ← getArgs
        let file : bpm_ : _ = args
            bpm = (read bpm_) ∷ Int

        parsedScore ← parseFromFile score file
        let abstractMusic = either (error ∘ show) id parsedScore
            concreteMusic = map (concrete bpm) abstractMusic

        B.putStr $ produceStream concreteMusic

-- | Given the sequence of concrete musical sounds to play
-- (Frequency, Duration), produce the corresponding sample stream
produceStream ∷ [(Float,Float)] → B.ByteString
produceStream music = B.concat [slice t (signal f)  | (f,t) ← music]

As you can see in the second line of “main”, we need 2 command-line parameters: “file” and “bpm_” (which will be read into an Int called “bpm”). “file” is the name of the file containing our score, and “bpm_” is the tempo in which the song will be played (in beats per minute – BPM). We then proceed to parse the score from the file, using the “parseFromFile” function (from Parsec). This function takes a parser to apply and a filename from which to read an input stream. Our conveniently-named “score” parser is defined in a separate file (Melody.hs) and will not be further discussed… You can see it if you download the full source code at the end of the post.

The “parseFromFile” function has type Parser a → String → Either ParseError a, which means its result is Either our desired outcome (Right [Note]) or an error (Left ParseError). We then apply the “either” function over “parsedScore” to do case analysis: in case a perfect parse ocurred, the “id” function is applied and the result is kept as-is; in case a parse error has been found, we apply (error ∘ show) – which just displays the error message on standard output and terminates the program. The next “big step” in the program is to turn the music into concrete form.

The parsed score is a sequence of musical notes with relative durations (1, 1/2, 1/4, etc.) and abstract pitch names (C, D, E, etc.), so the the task of “concrete bpm” is to take each abstract note and make both pitch and duration “concrete”. Its type is:

concrete bpm ∷ Note → (Int, Int)

It takes a Note and produces a pair representing a concrete sound, where the first element is the frequency of this sound (in Hz) and the second element is its duration (in seconds).

After we have our list of sound pairs to play (“concreteMusic”), we reach the last and most important step, which is to actually generate the samples and output them, using “concreteMusic” to guide the generation process. This is done by “produceStream”. For each pair of frequency and duration “(f,t)”, the following is done:

  1. Samples for a (infinite) square wave of frequency f are generated by a call to “signal f”.
  2. This signal is then “sliced” to obtain a piece of length t, by a call to “slice t”

Finally, this whole sequence of “signal slices” is concatenated together into a ByteString which goes to standard output… Now going deeper, we will dive to file “Signal.hs” and take a look at the definitions of “signal” and “slice”, along with some helper functions:

[...]
-- | Get a slice of an infinite signal with n seconds
slice ∷ Float → B.ByteString → B.ByteString
slice time = B.take (truncate (fromIntegral rate * time))

-- | Makes a continuous (infinite) signal for a given frequency
signal freq = cycleAndPack (singlePeriod freq)
    where
        singlePeriod 0 = silence
        singlePeriod f = period (truncate $ fromIntegral rate / f) Square

-- | Converts a [Int] to a Lazy ByteString and then cycles it
cycleAndPack ∷ [Int] → B.ByteString
cycleAndPack = B.cycle ∘ B.pack ∘ (map fromIntegral) 

-- | Synthesizes one period
period ∷ Int → Waveform → [Int]
period n Square = scale codomain $ map square (normalizeDom n)
period n Sine = scale codomain $ map sin (zeroTo2Pi n)

-- | Synthesizes one small silence, one sample :)
silence ∷ [Int]
silence = [0]
[...]

Let’s start explaining this module (“Signal.hs”) from the ground up: our signal is an infinite sound wave. The most characteristic feature of a wave is that it’s periodic, which means it’s just an infinite repetition of a pattern. Therefore we have the “period” function here, which, given the number of samples to generate and the desired waveform, generates a list of samples that follows this waveform. So that’s how “signal” uses “period” in order to make a wave: it takes the requested frequency and then calculates (according to the sample rate) how many samples are there in one period. It uses the “period” function to generate a “singlePeriod”, which it then cycles and packs into an infinite and repetitive ByteString. The “silence” function is only there to produce a “dummy” period when we need a pause. It doesn’t need to have more than one sample, however, because it’s going to be transformed into an infinite ByteString of 0′s anyway…

The “cycleAndPack” function takes one period ([Int]) and transforms it into a (lazy) infinite ByteString. Its type should be very self-evident, after all. The “(map fromIntegral)” part transforms our [Int] into [Word8], which then serves as input for “B.pack”, which actually creates the ByteString, then made infinite (repeated indefinitely) by “B.cycle”.

Now to the slicing! :) The task of the “slice” function is to transform an infinite wave (infinite ByteString) of a certain frequency into a finite wave (finite ByteString), with the requested duration in seconds. The type of slice is:

slice ∷ Float → B.ByteString → B.ByteString

Which makes evident its role as a “ByteString-transforming” function. The definition of “slice” – in point-free style – also helps explaining what it does: to slice an infinite ByteString is just to take n of its first elements, where n is the product of the desired time in seconds by the sampling rate in Hz.

Before continuing to show you the code, allow me to take a slight detour and say how Haskell’s purely-functional features help us express this program in such an elegant (and still efficient) way:

  1. Lazyness: Due to the possibility of using lazy evaluation, we were able to represent sound waves as infinite streams of samples, separating stream generation from stream usage.
  2. Referential transparency: Haskell’s purity means that every function is transparent. Thus, the value of any expression needs only to be computed once, and can be substituted for all occurrences of that expression. This is important because – in a single song – several notes will have the same frequency (and therefore will result in identical signals). Thanks to referential transparency, it is guaranteed that we only need to compute “signal f” once for each f, and thus we will only have one stream for each frequency in our song.

Well, having understood how “signal” and “slice” work, the puzzle is now almost entirely solved. The only remaining mystery should be how the actual input file (in Nokia-Composer-style) is parsed, but – as I said before – the parser is mostly straightforward and uses just basic Parsec. The input parser is the largest single piece of code in the whole synthesizer (only relatively big, but still small), so I’ll only show you a summarized version here and won’t even explain it. You can grab the full code at the end of the post and take a more careful look… So here it goes, the core of our input parser:

[...]
-- | Parsing a pitch
name = let singleton = (:[])  in  fmap (singleton ∘ toUpper) $ oneOf "cdefgabp"
sharp = fmap (maybe "" (const "s")) $ optionMaybe (char '#')
quote = fmap (maybe "4" (const "5")) $ optionMaybe (char '\'')

pitch ∷ Parser Pitch
pitch = do [n,s,q] ← sequence [name, sharp, quote]
           return $ read (concat [n,q,s])
[...]

-- | Parsing musical notes
duration = fmap read $ many1 digit

note ∷ Parser Note
note = liftM2 Note pitch duration

[...]

-- | Parsing a score
white = skipMany1 (space <|> newline)
separator = char ',' » white

score ∷ Parser [Note]
score = do notes ← note `sepBy1` separator
           optional newline » eof
           return notes

And now, a bit of fun: to finish up the post nicely, I have included some sample song files in the synthesizer’s tarball; then I ran the synthesizer over them and encoded the output as MP3. Now you can listen to two catchy tunes generated by Haskell!

  • Europe – The Final Countdown:
    • Song score:
      p4, p8, b16, a16, b4, e4, p4, p8, c'16, b16, c'8, b8, a4, p4, p8, c'16, b16, c'4, e4, p4, p8, a16, g16, a8, g8, f#8, a8, g4, p8, f#16, g16, a4, p8, g16, a16, b8, a8, g8, f#8, e4, c'4, b2, p4, b16, c'16, b16, a16, b1
    • MP3: Download audio file (finalcountdown.mp3)
  • Zelda Main Theme:
    • Song score:
      a#4, f4, f8, f16, a#16, a#16, c'16, d'16, d#'16, f'2, p8, f'8, f'8, f#'16, g#'16, a#'2, p8, a#'8, a#'8, a#'8, g#'16, f#'16, g#'8, g#'16,  f#'16, f'2
    • MP3: Download audio file (zelda.mp3)

Finally, as always, there comes what REALLY matters: The Source Code. You can grab the cabalized tarball containing the synthesizer just described – along with sample songs – HERE. I strongly recommend that you use cabal-dev to build all your Haskell projects, so in case you want to follow my recommendation, the steps to build and run (in Unix) the synthesizer are something like this:

tar xzvf ToneSynthesizer.tar.gz
cd ToneSynthesizer
cabal-dev install
./cabal-dev/bin/tonesynthesizer songs/<chosen-song>  | aplay -t raw -f U8 -r 16000

I wish you all a whole lot of excitement and fun while coding in Haskell, and would certainly appreciate suggestions and critique regarding the code I just shared with you. That’s all, folks! :)

flattr this!

by João Paulo at January 16, 2012 12:29 PM

Felipe Almeida Lessa

Abstracting permissions with Yesod

Yesod is a terrific framework for web applications in Haskell.  It has many, many built-in features.  One of them is that there’s nice support for authentication and authorization.  In this post I’m interested in talking about how you could write your authorization code such that it’s harder to make mistakes.

As shown on the recent example of creating a blog web app, Yesod’s approach to authorization lies within the isAuthorized function:

class ... => Yesod a where
  ...
  isAuthorized :: Route a -> Bool -> GHandler s a AuthResult
  ...

So isAuthorized takes a Route a, such as EntryR 10, and a Bool telling if the request may do writes (e.g. POST or PUT) or not (e.g. GET or HEAD). It must return an AuthResult that decides if the user is Authorized, Unauthorized or if he needs to be authenticated first (AuthenticationRequired).

While keeping authorization code in a single place is nice, using isAuthorized alone makes it very difficult to test your authorization code. Using the blog example again, only the admin should be able to create blog posts. So let’s take a look at the example code that decides if the user should be authorized to post to the blog:

isAuthorized BlogR True = do
  mauth <- maybeAuth
  case mauth of
    Nothing -> return AuthenticationRequired
    Just (_, user)
      | isAdmin user -> return Authorized
      | otherwise    -> unauthorizedI MsgNotAnAdmin

There are many things going on here:

  1. There’s authentication code that checks if the current user is logged in via maybeAuth.
  2. If the user is not logged in, it returns AuthenticationRequired.
  3. Otherwise the user’s credentials are checked to see if he should be authorized or not.

Step 1 is pretty standard housekeeping-style code. Steps 2 and 3, however, are part of your business logic that decides who should be able to do what. This means that you should be able to create (a) an unit test that asserts that the admin can create blog posts and (b) an unit test that asserts that a non-admin can’t create blog posts. Unfortunately, with the code above writing unit tests is really difficult. You not only need to artificially run the GHandler monad (boring), but you need to fake the current session so that Step 1′s maybeAuth gets the right information (difficult). Even then, your isAuthorized function is allowed to do anything, since it’s inside GHandler.

Enter permissions. What we’re really trying to say in the code above is that (a) to create a blog post you need the “post” permission and (b) admin has “post” permission, non-admins don’t. So let’s split these things! First of all, we need a list of the permissions that we’ll need:

data Permission = Post | Comment

You should read these constructors names with “permission to” before their names. For example, the admin has permission to Post and any logged user has permission to Comment.

Each request needs to decide which permissions it requires. This is one of the most important pieces of your application’s security, since forgetting to ask for permissions could lead to catastrophic problems. Instead of having this core piece of your app diluted in isAuthorized, we use a simple, clear, pure function called permissionsRequiredFor. The idea is to make permissionsRequiredFor as simple as possible, such that with code review alone you could determine if you’re asking for the right permissions.

permissionsRequiredFor :: Route Blog -> Bool -> [Permission]
permissionsRequiredFor BlogR      True = [Post]
permissionsRequiredFor (EntryR _) True = [Comment]
permissionsRequiredFor _          _    = []

We also need to decide if the currently logged user has the necessary permissions or not. This is the other important piece of your authorization puzzle, and a piece that we need to make easily testable. In order to do so, we avoid maybeAuth and take the user as an argument.

hasPermissionTo :: (UserId, User)
                -> Permission
                -> YesodDB sub Blog AuthResult
(_, user) `hasPermissionTo` Post
  | isAdmin user = return Authorized
  | otherwise    = lift $ unauthorizedI MsgNotAnAdmin
_ `hasPermissionTo` Comment = return Authorized


isAuthorizedTo :: Maybe (UserId, User)
               -> [Permission]
               -> YesodDB sub Blog AuthResult
_       `isAuthorizedTo` []     = return Authorized
Nothing `isAuthorizedTo` (_:_)  = return AuthenticationRequired
Just u  `isAuthorizedTo` (p:ps) = do
  r <- u `hasPermissionTo` p
  case r of
    Authorized -> Just u `isAuthorizedTo` ps
    _          -> return r -- unauthorized

The hasPermissionTo function decides if the user has a given permission or not. While in this example hasPermissionTo could have been a pure function, in general you may need to access the database. The isAuthorizedTo function then (a) decides if the user needs to be authenticated and (b) checks all permissions required from the list using hasPermissionTo.

Finally, we need to implement isAuthorized gluing everything together:

isAuthorized route isWrite = do
  mauth <- maybeAuth
  runDB $ mauth `isAuthorizedTo` permissionsRequiredFor route isWrite

Ok, so what did we gain by writing three more functions?

  • You can easily review permissionsRequiredFor to see if you didn’t leave a restricted route in the open.
  • If you don’t use the wildcards on the last line of permissionsRequiredFor and instead list all of your routes one by one, then you’d get a compiler warning and a runtime error every time you forgot to set the permissions of a newly added route.
  • If you have many routes that needed the same permissions, you don’t need to recode the permission code everywhere. You just need to code it once on hasPermissionTo and then ask for that permission on each of your routes. In my experience, the set of permissions (i.e. the Permission data type) is a lot smaller than the set of possible routes.
  • You may easily create unit tests for hasPermissionTo, increasing your confidence on your code’s correctness.

I should also note that this approach is easily extendable. For example, suppose that you wanted to restrict the visibility of some of your blog posts. You could change the Permission data type into:

data Permission = Post | CommentOn EntryId | View EntryId

Now your hasPermissionTo function is able to give a different answer depending on which blog post we’re talking about.

So far this approach has been successfully used on my day job’s Yesod application. It looks like a cousin of Yesod’s i18n support using data types.

Thanks for reading along this far! Please use the comment section below to say what think of this approach. =)

by Felipe at January 16, 2012 11:55 AM

Darcs

darcs weekly news #90

News and discussions

  1. Ganesh Sittampalam uploaded the second beta of darcs 2.8:
  2. Zooko O'Whielacronx called for a new maintainer for darcsver:
  3. Eric Kow summed up the recent issues of Darcs with regards to SSH:

Issues resolved in the last week (2)

issue845 Eric Kow
issue2090 Eric Kow

Patches applied in the last week (44)

See darcs wiki entry for details.

by guillaume (noreply@blogger.com) at January 16, 2012 11:47 AM

January 15, 2012

Programming Languages and Systems research group at the University of New South Wales

Seminar by John Launchbury

Thanks to NICTA, John Launchbury will be in Sydney next week. He contributed in many ways to the development of the Haskell programming language and provided practical proof of the benefits of functional programming and formal methods by founding and leading the company Galois, Inc.

On Friday (20 January), John will give a presentation at the School of Computer Science and Engineering (CSE) of the University of New South Wales. The details are as follows.

 

Time: 20 January 2012 (Friday), 10.30AM

Location: CSE Seminar room (K17_113), Level 1, CSE Building (K17)

 

Title: A Snapshot on Secure Computation

Speaker: John Launchbury (Galois, Inc.)

Abstract

There is growing interest in performing computation on encrypted data, partly motivated by the challenges of cloud computing. As we lose control of the location of our data, we still want to retain control of the confidentiality and/or integrity of our data. If we could encrypt our data (either for confidentiality, or for integrity) and then have the cloud operators perform computations on the data in the encrypted form, then we may have the best of both worlds: the cloud supplies storage and computational resources, while the encryption provides guarantees about what happens to the data.

Recent theoretical breakthroughs in homormorphic encryption have raised the possibility of this working in practice, though there are very many challenging practical considerations to overcome first. In the meantime, a simpler method of secure shared computation is approaching practicality---the main limiting factor is the network communication required. The main weakness of sharing schemes is that they rely on the fairly weak security model of "honest but curious", though there are newer techniques to weaken this assumption.

In this talk we describe the current state of the art, and provide some specific insights into how to build plausibly efficient algorithms for shared computation.

Bio

Dr. John Launchbury is Chief Scientist of Galois, Inc. John founded Galois in 1999 to address challenges in Information Assurance through the application of Functional Programming and Formal Methods. Under his leadership, formerly as CEO, the company has grown strongly, successfully winning and delivering on multiple contracts for more than a decade. Prior to founding Galois, John was a full professor in Computer Science and Engineering at the Oregon Graduate Institute School of Science and Engineering at OHSU. John received First Class Honors in Mathematics from Oxford University in 1985. He holds a Ph.D. in Computing Science from University of Glasgow and won the British Computer Society's distinguished dissertation prize. In 2010, John was inducted as a Fellow of the Association for Computing Machinery (ACM) for his contributions to functional programming.

 

Permalink | Leave a comment  »

January 15, 2012 11:22 PM

Joachim Breitner

c't features heisse-news

Several years ago, I created a computer news parody page called “heisse news”, emulating the look-and-feel of the German “heise news”, an online newsticker run by the most important German computer magazin publisher. Although my page did not see an updated since almost four years, it's featured in the latest issue of the publisher’s main print magzine, c't, in the section “Websites aktuell” on page 168, which I find very cool. As it is only fair to give and take, I posted a new entry on heisse news, featuring their website in the same manneras they featured mine...


Flattr this post

by nomeata (mail@joachim-breitner.de) at January 15, 2012 10:13 AM

January 14, 2012

Jeremy Shaw

happstack.com: now less evil

If you experienced some outage today trying to reach happstack.com, it is because we very ungracefully transfered our domain from the evil pro-SOPA godaddy to the less-evil anti-SOPA namecheap. Yay!

You are probably already familiar with SOPA/PIPA, but if not, here is some more information:

http://lifehacker.com/5860205/all-about-sopa-the-bill-thats-going-to-cripple-your-internet

The DNS changes have probably propagated to you by now, but if not, they should in the next 24 hours.

by Jeremy Shaw (noreply@blogger.com) at January 14, 2012 05:37 PM

Erik de Castro Lopo

A Simple Telnet Client Using Data.Conduit.

Below is a simple telnet client written using Haskell's new Conduit library. This library was written by Michael Snoyman the man behind the Yesod Web Framework for Haskell.

The Conduit library is a second generation approach to the problem of guaranteeing bounded memory usage in the presence of lazy evaluation. The first generation of these ideas were libraries like Iteratee, Enumerator, and IterIO. All of these first generation libraries use the the term enumerator for data producers and iteratee for data consumers. The new Conduit library calls data producers "sources" and data consumers "sinks" to make them a little more approachable.

The other big difference between Conduit and the early libraries in this space is to do with guaranteeing early clean up of potentially scarce resources like sockets. Although I have not looked in any detail at the IterIO library, both Iteratee and Enumerator simply rely on Haskell's garbage collector to clean up resources when they are no longer required. The Conduit library on the other hand uses Resource transformers to guarantee release of these resources as soon as possible.

The client looks like this (latest available here):

  import Control.Concurrent (forkIO, killThread)
  import Control.Monad.IO.Class (MonadIO, liftIO)
  import Control.Monad.Trans.Resource
  import Data.Conduit
  import Data.Conduit.Binary
  import Network (connectTo, PortID (..))
  import System.Environment (getArgs, getProgName)
  import System.IO


  main :: IO ()
  main = do
      args <- getArgs
      case args of
          [host, port] -> telnet host (read port :: Int)
          _ -> usageExit
    where
      usageExit = do
          name <- getProgName
          putStrLn $ "Usage : " ++ name ++ " host port"


  telnet :: String -> Int -> IO ()
  telnet host port = runResourceT $ do
      (releaseSock, hsock) <- with (connectTo host $ PortNumber $ fromIntegral port) hClose
      liftIO $ mapM_ (`hSetBuffering` LineBuffering) [ stdin, stdout, hsock ]
      (releaseThread, _) <- with (
                            forkIO $ runResourceT $ sourceHandle stdin $$ sinkHandle hsock
                            ) killThread
      sourceHandle hsock $$ sinkHandle stdout
      release releaseThread
      release releaseSock

There are basically three blocks, a bunch of imports at the top, the program's entry point main and the telnet function.

The telnet function is pretty simple. Most of the function runs inside a runResourceT resource transformer. The purpose of these resources transformers is to keep track of resources such as sockets, file handles, thread ids etc and make sure they get released in a timely manner. For example, in the telnet function, the connectTo function call opens a connection to the specified host and port number and returns a socket. By wrapping the connectTo in the call to with then the socket is registered with the resource transformer. The with function has the following prototype:

  with :: Resource m
       => Base m a             -- Base monad for the current monad stack
       -> (a -> Base m ())     -- Resource de-allocation function
       -> ResourceT m (ReleaseKey, a)

When the resource is registered, the user must also supply a function that will destroy and release the resource. The with function returns a ReleaseKey for the resource and the resource itself. Formulating the with function this way makes it hard to misuse.

The other thing of interest is that because a telnet client needs to send data in both directions, the server-to-client communication path and the client-to-server communication run in separate GHC runtime threads. The thread is spawned using forkIO and even though the thread identifier is thrown away, the resource transformer still records it and will later call killThread to clean up the thread.

The main core of the program are the two lines containing calls to sourceHandle and sinkHandle. The first of these lines pulls data from stdin and pushes it to the socket hsock while the second pulls from the socket and pushes it to stdout.

It should be noted that the final two calls to release are not strictly necessary since the resource transformer will clean up these resources automatically.

The experience of writing this telnet client suggests that the Conduit library is certainly easier to use than the Enumerator or Iteratee libraries.

January 14, 2012 02:22 AM

January 13, 2012

Kevin Reid (kpreid)

Followup on Cubes compatibility testing

Based on the feedback I’ve received, I conclude that the white-screen/crash problem with Cubes is hardware/driver-specific and not very widespread. Since

  • this is a hobby project and an experiment, not a Game to be Published,
  • the problem looks hard to debug,
  • I don't have a variety of machines to test on, and
  • I replaced the MacBook Pro in question with a newer model which does not exhibit the same problem,

I have decided not to debug it further at this time. I've added a note about the problem to the documentation, and I’ll instead work on the problems Cubes exhibits on the new machine :-) (such as a speckling of not-the-right-texture-tile at the edges of blocks, and a strange input lag) or adding new features, like more useful saving/persistence, sub-block collision detection, or subworlds.

by Kevin Reid (kpreid) (kpreid@switchb.org) at January 13, 2012 10:06 PM

JP Moresmau

EclipseFP 2.2.1 released

It is my pleasure to announce a new version of EclipseFP. Release 2.2.1 does not bring huge changes in the UI, but comes with important bug fixes and small enhancements. More importantly, it integrates with the latest versions of buildwrapper and scion-browser. You can see the release notes here.

We tried to enhance the performance and stability of EclipseFP, while making it easier to work with. The "open definition" action for example should be able to resolve a lot more symbols than before, including in type annotations. Quick fixes and auto-completion we've also worked on.

I wish to thank all users of EclipseFP, most notably all the people that submit bug reports or questions to the forum and thus help make the product move forward!

The instructions to install are the same as always: point your Eclipse to the update site http://eclipsefp.sourceforge.net/updates/. Once you restart EclipseFP should install or update buildwrapper and scion-browser.

Happy Haskell Hacking!

by JP Moresmau (noreply@blogger.com) at January 13, 2012 01:55 PM

Tom Schrijvers

Postdoctoral positions at Ghent University

<style></style>
 Get in touch if you are interested in applying for a postdoctoral position in the area of programming languages (computer science).
 Sixpostdoctoral positions in Ghent University’s Faculty of Science will beavailable with effect from 1 October 2012, for a duration of 3 years.
Profile:
The candidate:
·       holds a Ph.D. degree based on a doctoraldissertation, preferentially in (Natural) Sciences (the Ph.D. diploma must havebeen awarded before the date of appointment to the position i.c. 1st  October 2012);
·       conducts research activities in one of theresearch areas that are covered by the faculty of Sciences; this must clearlyappear from current research work and from a research plan for the coming yearsthat demonstrates a synergy with at least one of the research groups of theFaculty of Sciences;
·       has relevant scientific publications innational and international journals that are widely distributed and that applya peer review procedure to the submitted manuscripts;
·       has actively participated in national andinternational scientific meetings that are relevant to the field of study;
·       Should dispose of adequate didactic andcommunicative skills;
·       Communications or oral presentations atscientific meetings and sojourns for study purposes in foreign countries serveas an additional recommendation;
·       Experience with guidance of practical exercisesand master theses, serve as an additional recommendation.
Content of the position:
The successful candidate is expected to:
·       carry out scientific research in one of the researchareas that are covered by the faculty of Sciences (minimally 90% of the timehas to be devoted to scientific research);
·       contribute to the academic education activitiesof the host department;
·       contribute to the supervision of master thesesand doctoral students;
·       elaborate and write applications and reports ofresearch projects;
·       supervise and support scientific projects;
·       contribute to the services provided by the hostdepartment.
The governing languageat Ghent University is Dutch. However, research is primarily conducted inEnglish. Persons who do not speak Dutch as a native language are very welcometo apply.
Applications must besent by registered mail to the Personnel Office, Sint-Pietersnieuwstraat 25,9000 Ghent, at the latest on 29 February 2012. Applications should include aCV, proof of suitability (e.g., copies of the applicant’s diplomas), adescription of the current research and a research project plan (2 pages). Aduly filled in application form that can be downloaded from https://edit.ugent.be/we/enmust be sent as excel-file to decaan.we@ugent.be.

by Tom Schrijvers (noreply@blogger.com) at January 13, 2012 11:59 AM

January 12, 2012

Jeremy O'Donoghue

Who is my Community?

“Executive” Summary

This post is more of an op-ed piece than anything technically illuminating, and certainly not Haskell specific. Those not wishing to be exposed to a high level of pontification may prefer to pass. Similarly, those unused to my writing style may be disconcerted at just how many digressions I manage to fit into a short blog post. Imagine something like Steve Yegge, but without the wit, expertise or insight, and you shouldn’t be too far off. Again, please pass if this is an issue for you.

If you are a time-starved executive type who needs to cut to the content and ignore the crap, see immediately below.

tl;dr: We built something which might be of interest to others. Should we bother to make it accessible to others when it would make our own lives, and those of our existing community, harder?

wxHaskell – why bother?

I am the lead maintainer of wxHaskell, a Haskell binding for the wxWidgets library. As part of a coming push to make wxHaskell compatible with wxWidgets 2.9, we have done quite a bit of work which has raised questions as to the ‘best’ way forward in a number of areas.

As Open Source communities go, the Haskell community is probably a relative outlier in a number of ways which matter to an Open Source Haskell contributor:

  • It is relatively small;
  • Its members have an astonishingly wide range of abilities and interests;
  • It is incredibly good natured and helpful to all, pretty much regardless of level of ability.

The small size of the community means that many projects struggle to maintain a significant level of engagement over time. There are, I think, relatively few people interested in the nuts and bolts of maintaining a GUI binding, and this effort is split across several GUI bindings (Gtk2Hs and wxHaskell being probably the best supported, but with QTHaskell and HOC). There are good reasons for the split opinion on which GUI library to bind with, and I won’t explore them here.

Why not Gtk?

One aspect of wxHaskell which I hugely appreciate is that it works on the three main platforms: Linux and other unices; Windows and OS X, and delivers ‘native’ application look and feel without too much problem on all, and this is an attribute I’m determined we should keep.

In common with many programming languages, Haskell has an FFI binding which has no problem interfacing with libraries written in C, but cannot natively interface with libraries written in C++. There are very good reasons for this (basically that there is no way for anything other than the C++ compiler which originally compiled a C++ library to know how to call or link to the functions in that library – or more technically, every C++ compiler does name mangling differently).

This is probably the reason that GTK has become the default GUI binding for many non-mainstream languages. GTK may have <opinion-not-fact>a rather ugly approach to object-orientation</opinion-not-fact>, but all of its APIs are accessible in standard C, and it is therefore relatively straightforward to write a low-level GTK binding.

GTK is an outlier in the GUI world, however. Most GUI libraries are designed in object oriented fashion, using languages, like C++, which natively support object orientation. There are also good reasons for this: GUIs are <opinion-again> one of the few application areas </opinion-again> where classical object orientation is a very good way of expressing design (the other is designing libraries for classifying shapes, which seems to be a staple of OO tutorials, but not something for which I have seen an overwhelming demand in my professional life).

Wrapping C++ for beginners

Fortunately, it is very simple to make C++ code comprehensible to systems which only understand C. Suppose you have a class something like (borrowed from C++ Language Tutorial):

class CRectangle {
    int x, y;
  public:
    void set_values(int, int);
    int get_area() const;
};

It is straightforward to write a wrapper for the class, callable in C, which looks (if you ignore error checking etc.) something like:

extern "C" {
  void CRectangle_set_values(CRectangle* self, int x, int y) {
    self->set_values(x, y);
  }
  int CRectangle_get_area(CRectangle* self) {
    return self->get_area();
  }
}

You will notice that in each case, we pass around a pointer to the C++ object and use it to call the C++ function from within a function declared (with the “C” modifier) as being compatible with C calling and linker conventions.

Most of wxHaskell is wrapper code of this type. It’s not hard to write, but there is enough of it required that the writing soon becomes tedious and error prone. SWIG is designed to do this sort of thing, but it doesn’t have a Haskell backend, and when I started to work on one I quickly discovered that the documentation is a long way out of sync with SWIG itself, although the SWIG community is very helpful on the mailing lists. I may come back to a SWIG backend one day, but life is short!

That was a pretty long digression which was aimed at saying the what we have actually built is a C binding to wxWidgets. This is therefore just as accessible to most language FFIs out there as GTK, and would form a pretty good basis for a wxWidgets binding for many non-mainstream languages.

Standing on the shoulders of giants (and the not-so-giant)

Now, all of this good stuff is built on a wide range of Open Source code. Obviously the most important component is wxWidgets itself, but wxHaskell itself was forked from wxEiffel. We also use wx-config-win, a replacement for wx-config on Windows machines, and a project called wxC was forked some time back from wxEiffel (I think) to make a generic C wrapper.

In common with many Open Source projects, unfortunately many of these are moribund – the original contributors moved on and were never replaced, and so we end up with a fork or replace question. In most cases the answer to this is easy: fork – replace is almost always too costly. However Haskell is very productive indeed, which puts the question more finely in balance, and leads me (barring a few more digressions) to what I am really pondering.

How can I best serve ‘the community’ and what is that community anyway?

As I mentioned, right now we have a fairly complete C wrapper for wxWidgets. It builds as a DLL/shared library, and uses plain C headers, which means it could be used by pretty much any language.

There’s a barrier though. In order to make it as easy as possible for the Haskell community to set up and use wxHaskell, we have used a very Haskell-centric approach to building the library (Cabal, which is the standard way of distributing Haskell source libraries).

Unfortunately, I’m sure this would be a disincentive to anyone from, say, the Ocaml, Forth, Lisp or pretty much any other language community from using or contributing to the C wrapper (“waddya mean I need to download 200MB of Haskell Platform just to build a C library?”). I certainly know that I get discouraged when looking at a potentially interesting project only to discover that I need to locate 15 different packages before I even get to try to build it – an operation which is far from guaranteed to be successful with many projects.

So now I, or rather the wxHaskell community, recognise that we have a conflict of interest: we have built something which might be useful to many other groups (but we’re not sure if there is anyone out there who cares), but making life easier for those other groups will make life harder for our existing community, at least in the short term.

However, the other side of the calculation is that if other language communities get involved in contributing to the C binding, we get a larger community with a greater level of contrbution and engagement.

So we finally come to the point: is anyone out there interested in a C binding to wxWidgets, and if so, are they interested enough to want to help?


Filed under: Uncategorized

by jeremyodonoghue at January 12, 2012 10:35 PM

Magnus Therning

Compiling U-Boot for use in QEMU (VersatilePB)

Since I’m now working a bit with embedded systems I thought I’d take a look at compiling for one of the ARM-based machines that QEMU supports. I settled for VersatilePB after finding this old-ish article. Rather optimistically I thought that maybe, just maybe things had change in a year and that the limitation of flash was removed. How wrong I was.

I did find an easier way to get it working, though with the limitation that Linux has to be started via tftpboot or some other network-based fashion. The patch looks like this:

--- u-boot.orig/src/u-boot-2011.12/include/configs/versatile.h
+++ u-boot/src/u-boot-2011.12/include/configs/versatile.h
@@ -31,6 +31,8 @@
 #ifndef __CONFIG_H
 #define __CONFIG_H

+#define CONFIG_ARCH_VERSATILE_QEMU
+
 /*
  * High Level Configuration Options
  * (easy to change)

Then just go ahead and modify the default boot argument (CONFIG_BOOTARGS in the same file) to your hearts content to minimise the amount of manual work for booting.

Share/Bookmark

by Magnus at January 12, 2012 01:40 PM

Well-Typed.Com

Well-Typed are hiring: Haskell consultant

In order to keep up with customer demand, we are looking to hire a Haskell expert to work with us at Well-Typed as a Haskell consultant.

This is an exciting opportunity for someone who is passionate about Haskell and who is keen to improve and promote Haskell in a professional context.

The role is quite general and could cover any of the projects and activities that we are involved in as a company. The tasks may involve:

  • working on the Haskell compilers, libraries and tools;
  • Haskell application development;
  • working directly with clients to solve their problems.

Well-Typed has a variety of clients. For some we do proprietary Haskell development and consulting. For others, much of the work involves open-source development and cooperating with the rest of the Haskell community: the commercial, open-source and academic users.

At the moment, we are running the Parallel GHC Project. It is likely that initial tasks will have some connection with parallel and/or concurrent programming in Haskell. We are also doing quite a bit of GHC maintenance, and some knowledge or interest in compiler internals, operating systems, the foreign language interface, and/or deployment issues would be welcome.

Our ideal candidate has excellent knowledge of Haskell, whether from industry, academia, or personal interest. Familiarity with other languages, low-level programming, and good software engineering practices are also useful. Good organisation and ablity to manage your own time, and reliably meet deadlines, is important. You are likely to have a bachelor's degree or higher in computer science or a related field, although this isn't a requirement. Experience of consulting, or running a business, is also a bonus.

The position is initially as a contractor for one year with a salary of 150 GBP per day. We offer flexible hours and work from home. Living in England is not required.

In the longer term there is the opportunity to become a member of the partnership with a full stake in the business: being involved in business decisions, and fully sharing the risks and rewards.

If you are interested, please apply via info@well-typed.com. Tell us why you are interested and why you would be a good fit for the job, and attach your CV. Please also indicate when you might be able to start. We are more than happy to answer informal enquiries. Contact Duncan Coutts, Ian Lynagh or Andres Löh for further information, either by email or IRC.

The deadline for applications is Friday 27th January 2012.

About Well-Typed

Well-Typed LLP is a Haskell services company, providing consultancy services, writing bespoke applications, and offering commercial training in Haskell and related topics.

http://www.well-typed.com/

by andres at January 12, 2012 12:44 PM

January 11, 2012

Philip Wadler

Reading beside the lines: Using indentation to rank revisions by complexity

I'm always on the lookout for good papers that offer an empirical approach to programming languages. Thanks to Oscar Nierstrasz for suggesting this one, which shows that indentation is as effective as more sophisticated measures at classifying program complexity.
Abram Hindle, Michael W. Godfrey,
Richard C. Holt, Reading beside the lines: Using indentation to rank revisions by complexity, Science of Computer Programming, Volume 74, Issue 7, 1 May 2009, Pages 414–429.

Maintainers often face the daunting task of wading through a collection of both new and old revisions, trying to ferret out those that warrant detailed inspection. Perhaps the most obvious way to rank revisions is by lines of code (LOC); this technique has the advantage of being both simple and fast. However, most revisions are quite small, and so we would like a way of distinguishing between simple and complex changes of equal size. Classical complexity metrics, such as Halstead’s and McCabe’s, could be used but they are hard to apply to code fragments of different programming languages. We propose a language-independent approach to ranking revisions based on the indentation of their code fragments. We use the statistical moments of indentation as a lightweight and revision/diff friendly metric to proxy classical complexity metrics. We found that ranking revisions by the variance and summation of indentation was very similar to ranking revisions by traditional complexity measures since these measures correlate with both Halstead and McCabe complexity; this was evaluated against the CVS histories of 278 active and popular SourceForge projects. Thus, we conclude that measuring indentation alone can serve as a cheap and accurate proxy for computing the code complexity of revisions.

by Philip Wadler (noreply@blogger.com) at January 11, 2012 04:34 PM

Mark Jason Dominus

Where should usage messages go?

Last week John Speno complained about Unix commands which, when used incorrectly, print usage messages to standard error instead of to standard output. The problem here is that if the usage message is long, it might scroll off the screen, and it's a pain when you try to pipe it through a pager with command | pager and discover that the usage output has gone to stderr, missed the pager, and scrolled off the screen anyway.

Countervailing against this, though, is the usual argument for stderr: if you had run the command in a pipeline, and it wrote its error output to stdout instead of to stderr, then the error message would have gotten lost, and would possibly have caused havoc further down the pipeline. I considered this argument to be the controlling one, but I ran a quick and informal survey to see if I was in the minority.

After 15 people had answered the survey, Ron Echeverri pointed out that although it makes sense for the usage message to go to stderr when the command is used erroneously, it also makes sense for it to go to stdout if the message is specifically requested, say by the addition of a --help flag, since in that case the message is not erroneous. So I added a second question to the survey to ask about where the message should go in such a case.

83 people answered the first question, "When a command is misused, should it deliver its usage message to standard output or to standard error?". 62 (75%) agreed that the message should go to stderr; 11 (13%) said it should go to stdout. 10 indicated that they preferred a more complicated policy, of which 4 were essentially (or exactly) what M. Echeverri suggested; this brings the total in favor of stderr to 66 (80%). The others were:

  1. stdout, if it is a tty; stderr otherwise
  2. stdout, if it is a pipe; stderr otherwise
  3. A very long response that suggested syslog.
  4. stderr, unless an empty stdout would cause problems
  5. It depends, but the survey omitted the option of printing directly on the console
  6. It depends
I think #2 must have been trying to articulate #1, but (a) got it backwards and (b) missed. #3 seemed to be answering a different question than the one that was asked; syslog may make sense for general diagnostics, but to use it for usage messages seems peculiar. #5 also seems strange to me, since my idea of "console" is the line printer hardwired to the back of the mainframe down in the machine room; I think the writer might have meant "terminal".

68 people answered the second question, "Where should the command send the output when the user specifically requests usage information?". (15 people took the survey before I added this question.) 50 (74%) said the output should go to stdout, 12 (18%) to the user's default pager and then to stdout, and 5 (7%) to stderr. One person (The same as #5 above) said "it depends".

Thanks to everyone who participated.

by Mark Dominus at January 11, 2012 01:06 PM

Jeremy O'Donoghue

wxHaskell and wxWidgets 2.9

Those who follow the wxHaskell developer or users lists will know that over the last few months there has been a flurry of activity on wxHaskell, spurred on by Dave Tapley. I wanted to summarise what has been happening, and what stands in the way of a release supporting wxWidgets 2.9

<blatent-plug>Dave is working on wxHaskell with the agreement of his employer, Mentics Inc. I encourage any readers who have the opportunity to place business in Mentics’ direction to do so in appreciation of this generous donation of time and talent to the Haskell community.</blatent-plug>

Why wxWidgets 2.9?

Most Linux distributions currently supply wxWidgets 2.8.x libraries from their packaging systems, and Windows has the pre-built and readily installable wxPack containing wxWidgets 2.8.12, so why are we moving to an unstable release of wxWidgets?

There are many reasons, but two stand out in particular:

  • It is the future. wxWidgets 3.0 will be released from the 2.9 line, and provides major improvements. These include:
    • Much improved Unicode support.
    • Lots of new controls: ribbon bars, wxPropertyGrid, wxWebView, wxTreeListCtrl, wxRichToolTip, wxRichMessageDialog. Together, these simplify the creation of GUIs with a ‘modern’ look and feel.
    • The stc library (used by StyledTextControl) is now part of the main library build (as is SVG, which I would like to wrap for wxHaskell in the near future).
    • Support for 64bit OS X builds, which we have been unable to support on wxHaskell due to underlying lack of support in wxWidgets.
  • We need to clean up some legacy ‘cruft’ in the code. Daan originally wrote wxHaskell for wxWidgets 2.4.x, and things have moved forward a long way since then. This is an opportunity to remove deprecated functions and offer cleaner APIs.

What is changing?

The changes listed exist today in Dave’s development repository at DarcsDen, and will be mainlined in our master repository at code.haskell.org starting the coming weekend.

  • Newly wrapped classes
    • PropertyGrid, related helper types and sample code (Dave Tapley)
    • ListView (Dave Tapley)
  • Reinstated features
    • StyledTextCtrl (Dave Tapley)
    • OpenGL (Me)
    • Ability to use wxHaskell in GHCi (Dave Tapley)
  • Build system improvements
    • Some old Eiffel legacy code in the build system has been removed. Everything is now either C++ or Haskell (Eric Kow).
    • The C wrapper for wxWidgets has been moved into a separate project, wxC and built as a shared library (Dave Tapley).
    • Work to support OS X Lion (Eric and Alessandro Vermulen).
    • A Haskell native implementation of wx-config for use on Windows platforms. We are not sure if we will use this as yet – it is experimental (Eric).
  • Bugfixes and ‘build experience’ contributions from all of the above and several others , including Shelarcy, Maciek Makowski, Henning Thielemann, Peter Simons.

What is left to do before there is a release to Hackage?

Quite a bit!

The code in Dave’s repo is reasonably well tested on Linux (Ubuntu), but has currently received insufficient love on Windows or Mac. We will need it to work reliably on all three platforms before it can be released.

I think that the main blocker right now is probably determining the correct configuration for a Windows build. We have a couple of options here. We could develop Eric’s Haskell wx-config replacement so that it has been tested for a good subset of all possible wxWidgets build configurations, or we could fix wx-config-win to the same end. This has raised some philosophical questions on ‘Open Source and community’ which I am thinking of blogging on separately.

Most of all, when we have candidate builds ready, I hope that wxHaskell users will help out by trying to install the new version on as many machines as possible, so that we can be sure that we have everything working.


Filed under: wxHaskell

by jeremyodonoghue at January 11, 2012 11:19 AM

Philip Wadler

Computing at School

Hurrah! The government appears to finally understand that there is a difference between teaching IT and teaching computing, and that we need to move from the former to the latter in schools. Join the campaign at Computing at School. The Guardian is running a digital literacy campaign. Computing is the new Latin!

by Philip Wadler (noreply@blogger.com) at January 11, 2012 10:19 AM

January 10, 2012

Mark Jason Dominus

Elaborations of Russell's paradox

When Iris was five or six, I told her about Russell's paradox in the following form: in a certain library, some books are catalogs that contain lists of other books. For example, there is a catalog of all the books on the second floor, and a catalog of all the books about birds. Some catalogs might include themselves. For example, the catalog of all the books in the library certainly includes itself. Such catalogs have red covers; the other catalogs, which do not include themselves, such as the catalog of all the plays of Shakespeare, have blue covers. Now is there a catalog of all the catalogs with blue covers?

I wasn't sure she would get this, but it succeeded much better than I expected. After I prompted her to consider what color cover it would have, she thought it out, first ruling out one color, and then, when she got to the second color, she just started laughing.

A couple of days ago she asked me if I could think of anything that was like that but with three different colors. Put on the spot, I suggested she consider what would happen if there could be green catalogs that might or might not include themselves. This is somewhat interesting, because you now can have a catalog of all the blue catalogs; it can have a green cover. But I soon thought of a much better extension.

I gave it to Iris like this: say you have a catalog, let's call it X. If X mentions a catalog that mentions X, it has a gold stripe on the spine. Otherwise, it has a silver stripe. Now:

  1. Could there be a red catalog with a gold stripe?
  2. Could there be a red catalog with a silver stripe?
  3. Could there be a blue catalog with a gold stripe?
  4. Could there be a blue catalog with a silver stripe?
And more interesting:

  1. Is there a catalog of all the catalogs with gold stripes?
  2. Is there a catalog of all the catalogs with silver stripes?
I knew that early 20th century logicians, trying to repair the Russell paradox, first tried a very small patch: since comprehension over the predicate XX causes problems, just forbid that predicate. This unfortunately doesn't solve the problem at all; there are an infinite number of equally problematic predicates. (Whitehead and Russell's theory of types is an attempt to fix this; Quine's New Foundations is a different attempt.) One of these predicates is ¬∃Y.X∈Y and Y∈X. You can't construct the set of all X such that ¬∃Y.X∈Y and Y∈X because there is no such set, for reasons similar to the reason why there's no set of all X such that XX, so that's where I got the silver stripe predicate.

Translating this into barber language is left as an exercise for the reader.

by Mark Dominus at January 10, 2012 06:46 PM

Keegan McAllister

Zombie 6.001 starts tomorrow!

The student-run revival of MIT's famous intro CS class starts tomorrow! 6.001 and its text SICP had a singular influence on the teaching of introductions to computer science — not to be confused with intro to programming, worthwhile though that subject may be. After the unfortunate demise of 6.001 at MIT, some former TAs reanimated the class as an intense four-week experience. As their description says:

Zombie-like, 6.001 rises from the dead to threaten students again. Unlike a zombie, though, it's moving quite a bit faster than it did the first time. Like the original, don't walk into the class expecting that it will teach you Scheme; instead, it attempts to teach thought patterns for computer science, and the structure and interpretation of computer programs. Three projects will be assigned and graded. Prereq: some programming experience; high confusion threshold.

I'm helping teach it this year, and it should be a lot of fun. You can follow along online or if you're in the area, come to lectures Tuesdays and Thursdays, 19:00 to 21:00 in 32-044 (that's MIT building 32, room 044).

by keegan (noreply@blogger.com) at January 10, 2012 09:04 AM

January 08, 2012

Neil Mitchell

Pascal's Triangle in Haskell

Summary: I'm continually amazed how concise and powerful Haskell is, compared to mainstream languages. This post describes how to write Pascal's Triangle, and gives some of the advantages of using Haskell.

Often, when programming in Haskell, I feel like I'm cheating. As an example, I recently came across this article by William Shields, which suggests that prospective interview candidates be given simple programming tasks like generating Pascal's Triangle. William gives examples in Python, including some of the answers a typical candidate might give.

Pascal's Triangle in Haskell

William describes Pascal's Triangle as:

"The root element is 1. Every other element is the sum of the one or two above it (diagonally left and diagonally right)."

As a Haskell programmer, the obvious technique to use is induction. The first row of the triangle is [1], and each row can be computed from the previous row by adding the row shifted left, and the row shifted right:


next xs = zipWith (+) ([0] ++ xs) (xs ++ [0])
pascal = iterate next [1]


Here, we define next to take one row and produce the next row. We then use the iterate function to repeatedly apply next starting at the root element. The solution is short, and follows from the definition.

Laziness for Modularity

William originally posed three questions:


  • Print out the triangle to a specific row: print $ take 100 pascal

  • Return a given row of the triangle: pascal !! 50

  • Return a given element (by row and index) of the triangle: pascal !! 10 !! 5



Thanks to laziness, we can concisely answer all these questions in terms of the original pascal definition. In contrast, using a language such as Python, the best solution (Dynamic Programming from the original article) can only perform the first task.

Interview problems

The original article was not about the choice of programming language, but about choosing suitable questions for interviewing programmers. I agree with William that Pascal's Triangle is a suitable problem - it isn't a trick puzzle, it isn't an API knowledge quiz - it's about understanding how to program. Given how much easier the problem is to solve in Haskell, I wonder if using Haskell in a job interview should be considered cheating? ;-)

by Neil Mitchell (noreply@blogger.com) at January 08, 2012 02:34 PM

Hiding Win32 Windows

Summary: This post describes how to hide windows on the Windows operating system, by using the Win32 API from Haskell.

Imagine that whenever your computer restarts, it pops up a message box:



If you interact with this window, your computer will be destroyed. You can't kill the process, or dismiss the window harmlessly. (This scenario isn't hypothetical...) The solution is to hide the window, so it still exists but is out of the way of misplaced clicks. To hide the window, we first find it's OS handle, then we call some Win32 API functions.

Find the Window

To find the handle of the window, we use Spy++. Spy++ comes with Visual Studio, and is bundled in the Express edition (the free version) from 2010 onwards. Start Spy++, got to Search, Find Window, then use the finder tool to select the window in question. Check that the caption of the window matches what Spy++ reports:



The important information is the handle: 0004061E.

Hide the Window

To hide the window you need a programming language capable of making Win32 API calls. In the past I have used Word VBA as the host language, but Haskell is probably easier. Start GHCi, and type:


$ import System.Win32
$ import Graphics.Win32
$ showWindow (castUINTToPtr 0x0004061E) sW_HIDE


Replace 0x0004061E on the final line with 0xyour-handle. The final line should cause the window to be hidden, saving your computer from destruction.

Thanks: Thanks to Roman Leshchinskiy for reminding me that there were better solutions than just trying not to click the window. Thanks to the Win32 Haskell developers - the Win32 binding was a lot of work, which not many people ever see.

by Neil Mitchell (noreply@blogger.com) at January 08, 2012 12:03 PM

Christopher Done

Moogle (movie reviews) and web frameworks

My latest harebrained scheme, a movie/TV review web site that only shows reviews by your friends and people you’ve chosen to see.

Review Everything

By giving a short review and some tags of every movie you watch, you really help out your friends deciding what to watch.

  • Step 1: Watch movie.
  • Step 2: Jump onto moogle, save a little write-up.
  • Step 3: Your friends get an email and check out the review, decide to grab the movie or decide they wouldn’t like it.
  • Step 4: You can come back later and see what your favourite movies are, or invite your friends.

Design Mockup

I’ve done a design mockup pitching it as moogle for reviewing movies and TV. I want this site to look professional and be simple and clean. I’ve produced the best I can do with Inkscape, the rest CSS3 can take care of.

The idea of the site is quite trivial and implementation will be easy. However, see the next section.

The Framework

The next problem is choosing the web framework. The Big Three1 are:

Choosing between them is a very slow and boring task. They all appear to be production-ready. I could just use FastCGI as I do at work with my pre-existing web utilities that I wrote before these frameworks existed, where I’d have everything I need to get going,3 but I feel it’s time to consolidate efforts.4

Surprisingly I am unable to find a straight blog that compares all the technical features and philosphies of Happstack, Snap and Yesod.5 I’m not sure whether I have the time and motivation to make such a long and detailed post, but I will try if so.6

I might even use Ji or UHC JavaScript and abstract over it, we’ll see.

More to come.


  1. There’s also Salvia which is also interesting and fairly complete, though I don’t know if actively maintained or used (pretty sure the author uses Happstack).

  2. Not sure why they choose to stop using my design, couldn’t reworked the concept rather than scrapping it, but go figure.

  3. Haskell is a powerful enough language that it’s easy to work from the ground up.

  4. For example, I’d like to provide OpenID. This is implemented already in both Snap and Yesod (and probably Happstack). I don’t feel like implementing and testing this myself. There is also the problem of contribution. Choosing a popular framework will aid contribution. It’s just a shame I have to pick one.

  5. On the other hand one can find plenty of contention between the Yesod and Snap lead developers.

  6. It would seem that (few non-author) people have tried all three of these frameworks, otherwise there would be more comparative literature out there (suppose), this would imply that people pick the frameworks fairly arbitrarily, or based on the community.

January 08, 2012 12:00 AM

January 07, 2012

Dan Piponi (sigfpe)

Lossless decompression and the generation of random samples

Let S be some finite set with a probability distribution on it. Here's a diagram showing some example probabilities for the set S={A, B, C, D, E}:
How can we generate lots of random samples from this distribution? A popular method involves first storing the cumulative distribution function in a table like so:

We then use any of a number of popular methods to generate uniform pseudorandom numbers in the range [0,1) and for each one walk through the table until we find the first entry greater than our pseudorandom number. The symbol above the number is the one we generate. So if we generated 0.512, we'd pick symbol C. It's straightforward to prove this gives the correct probability distribution.

As described this algorithm can be slow. If the size of the table is N we may have to walk through up to N entries to generate each sample.

One approach to accelerating this algorithm is to quantise our pseudorandom number and use it to look up, in a precomputed table, a jump into our cumulative distribution table. I've used this several times in my visual effects career. But today I'm going to take a different approach.

Another natural way to speed up sample generation is to use a binary search to find the appropriate point in the cumulative distribution, for example the C++ standard template library's upper_bound method will do the job.


But what is a binary search algorithm going to do? Typically it's going to start by comparing our pseudorandom number with the midpoint of the table. If our number is bigger then it will recursively use binary search on the left (looking next at the midpoint of the left half), otherwise on the right, and so on. If we're generating many samples from the same distribution we're going to be repeatedly looking up the same midpoints in the table. At the end of the day, the process can be described by a decision tree. So we may as well throw away the table and build the decision tree up front. Here's what it might look look like:


But maybe we can do better. For example C is three times as likely as A but they both take the same amount of time to generate as they both require walking down to depth 2. Meanwhile D has a probability of 0.25 and requires walking to depth 3. We'd like to rebalance the tree. Note also that there's no reason to list our original PDF in the order I gave. Different orderings might give different trees.


It's straightforward to describe what we want to optimise. We want to place sample i at depth di so that the expected value of di is as small as possible. In other words we want to minimise Σpidi. But this is precisely the problem solved by Huffman coding.


And that's the conclusion I wanted to reach: Huffman coding gives the optimal decision tree to generate random samples. It also tells us this interesting consequence: if we use a decision tree method then the performance of our algorithm is bounded by the entropy of the probability distribution. I find this connection between entropy and algorithmic complexity pretty surprising.


I learnt the above during my interview at Google!


Why is there this connection between a compression algorithm and the generation of random samples? It  took me a little longer to realise why but it's quite straightforward.


Huffman coding tries to compress text one letter at a time on the assumption that each letter comes from some fixed and known probability distribution. If the algorithm is successful then we'd expect the compressed text to look like a uniformly distributed sequence of bits. If it didn't then there'd be patterns that could be used for further compression.

So when we're decompressing Huffman encoded data we have a machine that takes as input uniformly distributed bits and which outputs letters sampled from some probability distribution. But that's exactly the same problem that I posed above. So, at least from some perspective, decompression is precisely the same thing as generating samples from a probability distribution.

Or to put it another way: there is a class of algorithm whose purpose is to convert samples from one probability distribution into samples from another. When we use one of these algorithms to convert from samples from some distribution P that's easy to generate samples from, into samples from Q, we call this "an algorithm to sample from distribution Q". When we use one of these algorithms to convert from some distribution to a uniform distribution, and the function given by the algorithm is invertible, then we call it "lossless compression".

by sigfpe (noreply@blogger.com) at January 07, 2012 06:02 PM

January 05, 2012

Brent Yorgey

Parsing context-sensitive languages with Applicative

Many parser combinator libraries in Haskell (such as parsec) have both a Monad interface as well as an Applicative interface. (Actually, to be really useful, you also need MonadPlus along with Monad, or Alternative along with Applicative, in order to encode choice; from now on when I talk about Monad and Applicative note that I really have MonadPlus or Alternative in mind as well.) The Applicative interface is often nicer, but it is less powerful than the Monad interface: in particular, using Applicative you can only parse context-free languages, whereas Monad lets you parse arbitrary context-sensitive languages. Intuitively, this is because the structure of Applicative computations cannot depend on intermediate results, whereas Monad computations allow you to choose which computation to run next based on intermediate results.

This is a good bit of intuition, with only one minor caveat: it isn’t true! I believe it was two years ago, during the second Hac phi, when I first learned from Edward Kmett how Applicative (by which I mean, of course, Alternative) can be used to parse arbitrary context-sensitive languages. The question just came up again in the #haskell IRC channel, and I figured it would be useful to have this all written down somewhere. In particular, Reid Barton gave a nice sketch which I decided to turn into some working code.

Here’s the key insight: normally, grammars are defined as finite objects: a finite set of terminals, a finite set of nonterminals, and a finite set of productions. However, Haskell’s general recursion means that we can write down a "grammar" with an infinite set of production rules. This is what lets us get away with parsing context-sensitive languages with Applicative: we just make a different production rule for every possible input!

First, some imports. Notice that I do not import Control.Monad.

> import Text.Parsec
> import Text.Parsec.String
> import Control.Arrow ((&&&))
> import Control.Applicative hiding ((<|>))
> import Data.List (group)

The usual guard function is for MonadPlus, but we can make something equivalent for Alternative.

> guard' :: Alternative f => Bool -> f ()
> guard' True  = pure ()
> guard' False = empty

And now for the meat of the example. parseArbitrary takes an arbitrary predicate on Strings built from lowercase letters and turns it into a parser. The created parser will accept Strings for which the predicate evaluates to True (returning ()) and fail for any other string.

> parseArbitrary :: (String -> Bool) -> Parser ()
> parseArbitrary p =

If we encounter eof, we simply ensure that the predicate holds of the empty string.

>       (eof <* guard' (p []))

Otherwise, we choose between 26 alternatives based on the next character in the input. If the character c is encountered, we make a recursive call to parseArbitrary (p . (c:)). The remainder of the input must satisfy (p . (c:)), that is, it must consist of some String s such that (c:s) satisfies the predicate p.

>   <|> foldr (<|>) parserZero
>         (map (\c -> char c *>
>                     parseArbitrary (p . (c:))
>              )
>              ['a'..'z']
>         )

For any given predicate p, you can think of parseArbitrary p as an infinite tree with a 26-way branch at each node. Each node "remembers" the path taken to reach it from the root of the tree, in the form of prepend functions composed with the original predicate. We have constructed an infinite grammar: each node in the tree corresponds to a production, one for every possible input prefix.

Let’s try it out. Here’s a function which only accepts Strings of the form "aaabbbccc", with an equal number of a’s, b’s, and c’s. This is a well-known example of a language which is not context-free (easily shown using the pumping lemma for context-free languages).

> f :: String -> Bool
> f s
>   | [('a',na), ('b',nb), ('c',nc)]
>     <- map (head &&& length). group $ s
>
>     = na == nb && nb == nc
>
>   | otherwise = False

Now we make f into a parser and test it on some example inputs:

> p = parseArbitrary f
>
> main = do
>   parseTest p "aaa"
>   parseTest p "aaabbbcc"
>   parseTest p "aaaabcccc"
>   parseTest p "aaaaabbbbbccccc"

The last test succeeds by returning (). For the first three, we get error messages like this:

parse error at (line 1, column 4):
unexpected end of input
expecting "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y" or "z"

Obviously, these are not very helpful. But what were you expecting? After all, this is one of those things that is interesting in theory, but in practice amounts to an awful hack that no one would ever want to use in real code.

In the end, it’s still true that Applicative can only parse context-free languages as long as we restrict ourselves to finite grammars—which any sensible programmer would do anyway.

[ETA: it looks like using infinite grammars is not as impractical as I thought---see the comments below!]


by Brent at January 05, 2012 07:56 PM

Alson Kemp

My Apache process is only using one core!

I was recently working on a client site (a good-sized one) and was checking on the health of their application servers.  I noticed that each of their app servers was running a few of the cores much harder than the other cores.  This was in the evening and they get most of their traffic during the day; it runs Django under mod_wsgi in daemon mode with 8 processes and 25 threads per process.  Further, the boxes were not VPSs/VMs, but were dedicated, multicore boxes.  So they had multicore hardware and the web server was running in a multicore friendly way.

At the time, the load for each box was around 0.5.  And various process IDs rotated as the top CPU users, so process IDs weren’t bound to a core.  The ratio of traffic between the cores (ignoring actual core number and focusing on the utilization of each core, since each box was different) was something like:

Core # : Core Utilization
1      : 15%
2      : 2%
3      : 1%
*      : 0%

So why would one processor bear most of the load?  I googled and googled, and found little useful information.   I banged on one server with Apache’s benching tool (“ab”) while watching core utilization and, sure enough, all cores shared the load equally.  So what was going on?

I’m not sure if it’s the Linux kernel or a natural outcome of CPU caches, but the simplest explanation is that in low load situations processes are similar, due to cache coherence, will flock to the same core.  Rather than spreading a set of processes across a set of cores that don’t necessarily share the same cache, processes naturally gravitate to the cores that experience the lowest cache misses.

Upshot: it’s rational for the system to schedule most operations of a process or group of similar processes on one core when a system is relatively lightly loaded.  This is especially true if the cores are “Hyperthreading” and are sharing resources (read: their caches)!

by alson at January 05, 2012 03:41 AM

January 02, 2012

Paul Chiusano

The future of programming

What will programming look like 10 or even 20 years from now? With another new year almost here, now is the time to wax philosophical about the future of our industry. We are on the cusp of a number of major transformations in programming that will make 2011 programming technology, techniques, and ideas seem primitive by comparison. The transformations will occur in several key areas: tooling and infrastructure, languages and type systems, and runtime systems.

Before I get started, let me preface this all with the disclaimer that this should be taken with a grain of salt, in the spirit of being provocative, and these are not really predictions per se. There are lots of reasons why inferior technologies might continue to dominate the industry longer than expected. I won't talk much about that here, although it's certainly interesting. No, instead I want to give my vision of the future of programming. If there were no accumulated network effects attached to all the mediocre technologies currently in use for programming, if the programming world were given a clean slate and the directive to invent the future, what would I want to result?

An overarching theme to all these transformations is the move away from incidental structure and its close cousin incidental complexity. Manifested in various forms, these factors are major barriers to the building programs of greater complexity, and many of the major transformations will be to enable lifting of these barriers. The resulting programming world will look much different--orders of magnitude more code reuse, ease of deployment, efficiency, and much more.

Where we begin

The first major change is the move away from storing programs as text, split across "files". You can be forgiven for thinking of this as a minor thing. Who cares how programs are stored or represented, right? In fact, this major source of incidental complexity has spillover effects all the way down the programming toolchain, starting with the pollution of programmers mental thought processes, continuing to development environments (which I'll discuss next), and then to build systems and deployment. Like many of the things I discuss here, the full extent of lost productivity due to this incidental complexity is below most programmers radars. Programmers have unconsciously accepted this lost productivity, to the point that many now rationalize their dependence on files and text for programming.

But what about intentional programming, you ask? Or Smalltalk development environments? The idea of moving away from programs as text spread across files is not exactly new, but the devil is in the details. Previous efforts to rid ourselves of this incidental structure have failed in part because they merely substituted one arbitrary structure for another. For instance, intentional programming advocated the good idea of separating code presentation from its storage, substituting instead another format like XML with only slightly less incidental structure. Any advantage conferred by the new format and structure was not exploited to significant benefit, and there was a huge, very real cost in terms of lost network effects from all the text-based tools that could no longer be leveraged. Likewise for Smalltalk development environments. Also, somewhat ironically for Smalltalk, being an OO language, its fundamental premise includes one of the most arbitrary choices of all, in some sense the ultimate generator of incidental complexity, namely, the decision of which class should implement a particular method.

There is a real barrier to entry here for any new technology. It doesn't need to just be better than the antiquated status quo; it needs to be significantly so to overcome the network effects and switching cost advantage that status quo technologies inherently posses. In my opinion, none of the proposed text-in-file alternatives so far have come close to overcoming these handicaps.

But this is nothing fundamental. We should not conclude from these failed experiments that the idea is unsound. It's merely been poorly executed. The first major shift in getting this right is not storing programs as text at all, not even some normalized textual form like XML, and not some normalized binary form which nonetheless contains a huge amount of incidental structure. Likewise, get rid of files. Instead, a codebase is stored relationally, as a database, with some version of datalog being used for querying and update. Careful thought will be given to this query-update language so that many of the large-scale refactorings that are currently difficult or impossible can be fully automated, or guided automated, with just a few lines of this codebase transformation code. As a simple example, we eliminate the notion of where a function or datatype "is located". Instead, each function gets a unique id (perhaps content addressed, to enable a simple, automated form of code deduplication), and a separate names table maps these ids to names for purposes of rendering the code in some fashion to a human (and in principle, there is really no reason why there can't be multiple names tables, with different programmers choosing different names for the same operation). This representation trivially enables the "renaming" refactoring in just a single table cell update.

Let me sketch out a few additional thoughts on how this could work. First, I am not advocating for datalog syntax. I don't care about that. The key functionality enabled by datalog over and above the relational algebra is the ability to express transitive closure and mutual recursion guaranteed to terminate. Together these features enable many of the common queries we'd like to express in transforming and querying our codebases. For instance, here is a hypothetical query to find all references to a given function id, fid. Don't worry if the syntax looks alien or doesn't make sense. The key is more that this query is just a few lines of code to express, and it can be reused and built upon.

-- propagate reference to containing apply
refs(Id) :- apps(Id, fid, _).
refs(Id) :- apps(Id, _, fid).
refs(Id) :- refs(X), apps(Id,X,_).
refs(Id) :- refs(X), apps(Id,_,X).
-- any lambda whose body is or contains fid
-- is considered to reference fid
return(Id) :- lambdas(Id,_,Id1), refs(Id1).
return(Id) :- lambdas(Id,_,fid).

Current IDEs, with their enormous development staff and reams of special purpose code, support a very limited subset of code transformation/querying operations, poorly, slowly, and with huge overhead. As a result, there is an entire subculture of programmers (myself included) who have largely rejected IDEs in favor of minimal, less feature-rich but more dependable text editors. Future development environments will take the refactorings now supported by the most advanced IDEs as the most insignificant starting point, and build from there.

Large scale refactorings are the inevitable consequence of achieving the reuse needed to build programs of significant complexity that don't die of heat death. Refactoring times in this new model will go from weeks or months to hours, and writing code to transform a codebase will become a separate but critical skill, distinct from the usual act of programming. That is, programmers do not simply conceive of a refactoring (which is often quite simple to express to another programmer), then begin a tedious, manual and error-prone process of text munging to implement it. Instead, the programmer conceives of a refactoring, then conceives of a code transforming program to implement the refactoring, then applies this transformation to the code database, all in the span of a few hours.

The code as database concept has spillover simplification effects in other areas of tooling, in particular version control. The problem of handling merges of ordered sequences of characters spread across files and directories with yet more arbitrary structure is extremely difficult, resulting in a huge amount of complexity in the area of version control software. The difficulties have led many to settle for what are arguably inferior VCS models (Git, Mercurial), where changesets form a total order and cherrypicking just doesn't work. In the future, with code databases, we'll see a resurrection of Darcs' patch theory, only this time, it will work exactly as expected, and will be trivial to implement.

Related to this notion of code databases, we get much more fine-grained dependency management. Dependency management is another serious problem for software development, a huge hindrance to code reuse, and once again, something that is simply below many programmer radars. A library will often introduce some new interface or abstraction and give several instances of that abstraction for particular concrete types. The library now depends on these concrete types being available for compilation. Including this library now means pulling in all these dependencies, even if you just require the one instance. The overhead of doing this in conjunction with the existing complexity of builds (again partially caused by the programs-as-text-in-files paradigm) means code is reused much less than would be possible or easy otherwise.

In the future, we'll be able to conditionally specify code, and not using an ad hoc, 1970s technology like the C preprocessor. The datalog query and update language will factor in here, and allow us to express things like: if a concrete type X is available in the codebase, define some set of functions and new datatypes. Likewise, dependencies will in general not be on "a library" or "a module". A function will depend only on the set of functions and types it references, and a codebase will be a much more fluid thing. We can extract any connected component of functions and datatypes from a codebase, and this is trivial, automated transformation supported by the query language. There are some unknowns here, all solvable, around versioning, and they are related to how we reconceptualize version control as a partial order of changesets, with no extraneous dependencies due to the representation of programs as ordered sequences of characters.

Code editing, IDEs, and type systems

Code editing will be done in structural editors, which will look nothing like the existing batch of IDEs that are little more than glorified text editors (and they are actually rather poor text editors). In a structural editor, the programmer will construct expressions which may have holes in them not yet filled with terms. Importantly, these structural editors will be type-directed, so for any given hole the programmer can be presented with set of values matching the expected type, ordered in some sensible way. The editor will perform local program search to enable autocompleting of multiple expressions. If you've ever seen someone live-code Agda, you'll know how powerful and productive this idea could be. Yeah, the actual interface for programming Agda is still kind of 1970s (a custom Emacs mode), but the idea of type-directed editors is powerful. It makes it clear that types are effective not just at preventing many software errors, but also in guiding development. They are a powerful tool to augment our puny programming brains.

Along these lines, the rise of type-directed development in languages with real, state of the art type systems will mark the beginning of the end for dynamically typed (aka, single-typed) languages. Dynamic typing will come to be perceived as a quaint, bizarre evolutionary dead-end in the history of programming. This is already widely accepted in some programming circles, where the general consensus is that most dynamic typing advocates are not familiar with type-directed development in a language with a real type system and are basically unaware of the state of the art in type systems and programming language theory. With some additional developments in type systems we'll see any last of any advantages to dynamic typing completely evaporate.

A related transition is that types will become even more critical and type systems will grow features to better handle data normalization, the lack of which is a major source of incidental structure and complexity. A big problem in large codebases is data normalization. Lack of data representation normalization is responsible for the significant amounts of plumbing code involved in aligning two pieces of code so they can talk to each other. Most of the mainstream programming world isn't really aware of this issue because code reuse in most imperative codebases is extremely limited (because side-effects don't compose well).

In the functional programming world, there is insane amounts of code reuse happening, but along with this comes a fair bit of plumbing. As a small example, consider a function, f, of type X -> Y -> Int, and a value, xs of type [(Y, X)] and suppose you want to apply f to the list. An experienced functional programmer will bust out map (uncurry f . swap) xs in about 3 seconds and not think twice about it. But this code is total boilerplate, there to convince the compiler this is what you really want to do, and is noise when reading the code. Yes, you are acheiving reuse (an imperative programmer would still be writing out a for loop for the millionth time instead of reusing higher-order functions) but it should be cleaner. If this code needs to exist (and I'm not sure it even does, see below), I'd rather get to this point in the editing process and then tell my editor to map f over xs. The editor will search for a program to make the types align, show me the program for confirmation if I request it, and then not show this subexpression in the main editor view, perhaps just showing map f* xs, where the * can be clicked and expanded to see the full program.

Even better, perhaps we can get away with much less plumbing in the first place. Plumbing can be eliminated from code in two general ways--the first, which I've just discussed, is to have plumbing code written for you automatically, then hidden unless requested. The second is to have more normalized types and type systems so that there is no incidental structure for code to broker between in the first place. To support this we'll see things like row types, unordered tuples, type sets, and so on, and any related type system machinery needed to make all this feasible and convenient. An idea I'm interested is explicit separation between the model underlying a type (which could be something very normalized, with no incidental structure), and views of that type, which may contain some additional structure. Functions will generally be written to operate over the model, from which any number of views can be reconstructed post-transformation. The compiler or runtime is generally smart about choosing runtime representations to avoid unnecessary round trip conversions between different representations.

Language runtimes

Non-strict languages will come to dominate the programming world, due to the increase in code reuse and modularity that comes with pervasive non-strictness. As I've argued before, optional laziness doesn't cut it. As with other issues I've mentioned, the problems with strict as default aren't apparent to most programmers, even those who view themselves as well-versed in functional programming. The problems with strictness only become fully apparent as you get much further along in the development of the FP style, in particular, after the discovery of combinator libraries and the further abstractions that develop to remove duplication across these libraries. This has been a source of tension in the Scalaz library, which supports functional programming in Scala, a strict-by-default language, and also in our Scala codebase at work.

The only real problem with laziness has to do with reasoning about space usage performance, and evaluation stack usage. These problems get a lot of play among people who enjoy rationalizing their ignorance of Haskell and FP in general, but the truth is some additional research and smarter evaluation strategies can address the problems. There's nothing fundamental here suggesting we should throw up our hands and resort to strict evaluation.

The usual normal order evaluation is guaranteed to terminate if any is, but reasoning about is space usage in this evaluation order is problematic. We can do much better. Besides static strictness analysis, which only covers a fraction of the cases we'd like, we can conceive of additional evaluation orders which terminate for the same set of programs as normal-order evaluation, and which can propagate additional strictness information. The one I am particularly interested in I refer to as specializing, strictness-propagating evaluation. I'll elaborate on this in another post, but in this evaluation model, calling a function is something like two communicating coroutines. When calling a function, the callee begins evaluating its body, yielding control back to the caller when it needs its first argument, and also indicating whether that argument should be strict or lazily passed, using whatever information is available at runtime. Subsequent arguments work similarly. As a result, functions are automatically specialized as arguments are passed, and we do not construct thunks if they are going to be consumed strictly by a subsequent callee. This can be implemented efficiently using just two call stacks, and there are various optimizations to the scheme. It is intended to augment, not replace, the existing static strictness analysis and argument passing.

In general, the goal of new evaluation strategies should not be efficiency (though that is a nice goal as well), but simple reasoning about space and evaluation stack usage, so that these things do not depend as they currently do on incidental details of how a function is factored or what the compiler chooses to inline (being forced to reason about these things in Haskell breaks the black box abstraction of functions that FP and higher-order programming depend on).

Language runtimes and VMs will grow to support these new evaluation strategies, and the current crop of "general-purpose" VMs that are actually quite specialized for strict, imperative languages (the JVM, the CLR) will likely die off.

Code distribution and the future of the web

What of the web, javascript, html 5 and beyond? Are we going to keep hobbling along with these technologies indefinitely? I don't think it's too controversial to say that writing applications of any significant complexity as a web application involves far more work than would be required with access to a truly powerful client-side language. Again, I don't think many web programmers realize just how bad the situation is. In comparison to mainstream languages like Java, C#, or Python, Javascript isn't so bad; in some ways it's even a better language. But compared to a programming language with a good type system and real support for FP like Haskell, Scala, and whatever the next generation of languages can bring, Javascript is quite sad.

Of course, proponents of existing web technologies are always finding ways to rationalize the status quo, pointing out how much better things are today than they used to be. That's maybe true, but why settle?

I envision a future where Javascript dies off, as do browser-specific plugins like Flash, and instead we'll see client code written in arbitrary, compiled languages, using something like NaCl. This will be combined with a signed code caching and dependency tracking mechanism, so that for instance, you can distribute an application that downloads the entire Java virtual machine and other dependencies, but only if they aren't already cached on your local machine (and the signatures match, of course). This changes how we think about software. Software won't be something you "download onto our computer and then run". Instead, software exists "out there", and you run it. As an implementation detail of running it, it may choose to download some additional code it needs to do its work.

This will end the monopoly that html+javascript has on client-side web applications and largely eliminate the switching costs of moving to new client-side technologies. A few lower-level protocols and standards will be enough to tie everything together and maintain the good parts of the web today (I'll say more about this next), but nothing so high-level as specifying the client side language for interaction (Javascript, or Dart, or whatever) or the client-side language for layout and display (html + CSS). In the future, client-side code is written in whatever language, compiled to native code if desired.

In place or in addition to native client support, another option would be some sort of in-browser low-level VM designed by people who know what they're doing, who are fluent in the state of the art in programming languages and runtimes. In other words, not the people who designed Dart, Go, Ceylon, or any of the other recent languages that are 30 years or more behind state of the art. We need something that actually supports languages of the future, not something that rearranges the deck chairs on the titanic sinking ship that form the current batch of mainstream languages.

What of the suggestion that we simply use Javascript as the "assembly language" of the web, use it as a compilation target, and avoid programming in it directly? This obviously is not ideal. Compiling a real programming language like Haskell or whatever is next through Javascript or Dart is obviously not how anyone would not how design things today given a clean slate. Even if this were possible to do well without bloating client code size beyond what's acceptable, there is the inevitable efficiency hit of compiling to a language whose performance is as bad as Javascript. When you think about it, it makes no real sense to be reverting to 1/10th or 1/100th the speed of what native code or a well-JIT'd VM could provide, purely for ease of deployment. We don't need to be making this tradeoff, and with the rise of something like NaCl and/or a real browser VM, we won't have to.

There's one wrinkle. There are tremendous network effects on the web. This is part of its power, and its usefulness, and we don't want to lose it. We do still need the moral equivalent of urls and hyperlinking but one thing we don't necessarily need is the DOM. What will take its place? Don't we need the DOM to enable mashups and services like search engines? Actually, no. DOM munging and traversal is not the only way programs could obtain information from applications on the web, and when you think about it, it's actually rather primitive. Why screen scrape when you can call a real API? This transformation is already sort of happening; most modern web applications expose APIs in the form of REST+JSON. It just needs to be taken a little further.

What would this look like? Well, we would need a standard type system for the web. By this I mean a standard way for the applications that live at a url to expose a module of types and functions they support. The underlying type system would be expressive enough to encode richer data than what JSON currently provides (which besides being dynamically typed, cannot even represent sum types effectively) and will support for algebraic data types and some of the type system features alluded to earlier. With this in place, standards will arise for certain function signatures, with standard meanings attached to them. So, for instance, rather than the googlebot crawling the DOM looking for links, it invokes the getAllLinks function of the application at the given url, which returns an actual list of url values. getAllLinks is some separate standard, and new ones can arise on an ad hoc basis, as we grow new modes of interaction with web sites. There will be certain near-universal standards (like getAllLinks) and more specialized ones, specific to the web application in question (for instance, Facebook exports certain functions and datatypes that are specific to Facebook, which are unlikely to be implemented by other sites, though this is certainly not a requirement).

Already, we can see this happening somewhat: there are various ad hoc APIs and mechanisms for basically controlling how web crawlers should interpret the DOM.

With a standard way of interacting with web sites programmatically, there's no longer any need for the DOM and we can see a proliferation of innovation of different display and layout technologies.

Closing thoughts: the rise of functional programming

I've hinted at this throughout: functional programming will absolutely win. Not everyone shares my views, but many of the improvements I've talked about start with the assumption that we are working in a functional, post-imperative world. FP provides a dramatic increase in productivity due to massive increases in code reuse and the ease of reasoning about functional code (not to mention ease of parallelization). The industry is slowly starting to understand this, but it doesn't really matter if many programmers are still for whatever reason resistant to learning FP (which is unfortunately true). Eventually, the selection pressures of a competitive market will weed out less productive and effective techniques, and this largely means the death of imperative programming as we know it, except in certain niche areas. Regardless of initial biases or attitudes, programmers and companies who wish to stay competitive will be forced to employ FP.

As for why FP hasn't gained more prominence already, well, perhaps I'll write more about that in another post. What I will say is FP is currently reaching a tipping point enabling its wider adoption and relevance. There are several factors in play: functional language compiler technology is advanced enough, computers are fast enough, and most importantly, the FP community is rapidly discovering all the necessary techniques to organize large programs that preserve referential transparency. The Haskell community has mostly led this charge, Haskell being the only widely-used language that, due to its non-strict evaluation, had no choice but to fully commit to preserving referential transparency throughout. Ten years ago, admittedly, there was a good chance that expressing some programs purely functionally involved what was basically new research. Today, many of the required techniques are known, and expressing even the most imperative-seeming program functionally is possible, and for an experienced functional programmer, pretty effortless. There are still interesting open questions of how to express certain programs, but these are diminishing rapidly.

That said, I understand why many people claim FP is too hard, or it's unnatural or awkward, etc. Like many worthwhile subjects, attaining fluency in FP is difficult and requires dedication. Once this level of fluency is reached, though, expressing functional programs is quite natural and effortless (of course, software design is still hard, but finding functional designs becomes easy with practice). For people looking in, the techniques of FP seem opaque and unnecessarily difficult. For people on the inside, there's nothing difficult or complex about it and the benefits are enormous. For those who would criticise FP, I think a little humility is in order. To draw an analogy, no one without mathematical background would feel equipped to dismiss or criticise an entire branch of mathematics ("real analysis is a stupid idea"), and yet programmers with barely a cursory understanding of FP regularly (and loudly) criticise it.

I see why some people are frustrated with some of the existing resources for learning the subject. But it's wrong to dismiss the subject on that basis, or on the basis of personalities of people (myself included!) who feel that FP is more productive; objectively, either FP is worth knowing because of the productivity and other benefits it provides, or it isn't. For my part, I am co-authoring a book on FP that I hope is helpful to some who are interested in learning the subject. With the solidification of FP techniques I expect to see more and more resources like this, to the point that fluency and the huge resulting benefits are something that any motivated programmer can work toward, without encountering some of the discouraging hurdles to learning FP that are present today.

Will the future of programming look anything like what I've laid out here? Maybe, maybe not. But I certainly hope it will.

by Paul Chiusano (noreply@blogger.com) at January 02, 2012 09:30 PM

Johan Tibell

Dynamic graphs for counters and gauges

I've been busy implementing some of the feature requests I've gotten the last few days. Today I'm releasing the result as ekg v0.3. Here's what's new:

Counters and gauges

The old counter concept has been split up into two concepts: counters and gauges. Counters are monotonically increasing while gauges are not. An example of a counter would be requests served since program start and an example of a gauge would be the number of concurrent connections at a given point in time.

The counter/gauge distinction lets us do smarter things in the web interface. For example, given a counter called "iterations", we know that it ought to be graphed as iterations per second, instead of the raw value:

Graphing the raw value wouldn't be very interesting as it would result in a graph that just went straight up to the right.

Graphs for user-defined counters and gauges

The web interface now lets you dynamically add (and remove) graphs for user-defined counters and gauges. To add a graph, click on the little graph icon to the right of the counter or gauge name:

Reorganized REST API

The REST API has been reorganized to allow for finer-grained access to resources e.g. you can request all counters without also getting all gauges.

Server time instead of client time

In previous versions I used the client (i.e. browser) time when plotting graphs. This isn't a terribly good idea as network latency will skew the graphs. Version 0.3 uses the server time instead of client time to render graphs.

Usability improvements

I've also made a few minor usability improvements, such as adding column headers, improving number formatting, adding error messages, and reorganizing the user interface.

by Johan Tibell (noreply@blogger.com) at January 02, 2012 01:51 AM

Roman Cheplyaka

Composing monadic effects

In Haskell, monad transformers are used to combine effects provided by multiple monads.

Boring example: Writer, Reader, State

For instance, if we want to have a read-only environment plus a write-only logging facility, we can start with the Reader monad and add a logging facility to it with the WriterT monad transformer:

type MyMonad w r a = WriterT w (Reader r) a

Or, we could start with the Writer monad and put the ReaderT transformer on top of it:

type MyMonad w r a = ReaderT r (Writer w) a

In the end, it doesn’t matter which way we choose, as both are isomorphic to

type MyMonad w r a = r -> (a, w)

If we want to have a state as well, we can plug in the StateT transformer anywhere in the stack, and again the order doesn’t matter. The resulting monad is now isomorphic to the standard RWS (Reader-Writer-State) monad.

So, we say about those transformers that they commute (recall that Reader is just the ReaderT transformer applied to the Identity monad).

More interesting example: Either + Writer

Not every two transformers commute, though.

Suppose we want to write log messages, plus to be able to abort the computation.

There are two quite different ways to compose these effects.

There’s

type MyMonad e w a = ErrorT e (Writer w) a

isomorphic to (Either e a, w), and then there’s

type MyMonad e w a = WriterT w (Either e) a

isomorphic to Either r (a, w).

We can already see the difference from the types. In the first case, the w is there independently of the value of Either. In the second case, if the computation failed, the whole thing is Left msg, and we don’t get our log w back.

Let’s try the following to confirm our expectations:

> runWriter $ runErrorT $ tell "foo" >> throwError "err" >> tell "bar"
(Left "err","foo")
> runErrorT $ runWriterT $ tell "foo" >> throwError "err" >> tell "bar"
Left "err"

Developing intuition

When reasoning about a stack of monad transformers, it’s useful to keep in mind that a transformer knows nothing about its inner monad. It has to work “inside” that monad. It has no ability to alter or cancel the effects that happened before in that monad. (It can prevent those effects from happening, though — see tell "bar" above.)

In the first example above, which corresponds to ErrorT e (Writer w), the Error monad had to work inside the Writer monad. It simply had no way to tell the writer monad to “forget” what had been told to it before.

In the second example, corresponding to WriterT w (Either e) a, the main monad is now Either e. If it decides to fail, it will return just the error, and nothing else.

The same logic answers the question why there’s no IOT, an IO monad transformer. Imagine this:

do
targetHit <- launchMissiles
when targetHit $
throwError "oops I did it again"

Since throwError happens in the base monad, the whole thing is just Left "oops I did it again" — nothing mentions any IO. But what about the missiles?

If this shallow explanation is not enough to make you comfortable with transformers, I’d advise to read (or even to write) the implementation of some standard transformers.

Even more interesting example: Parsec + State

The Parsec monad already allows us to have our own state, but let’s see how we can add one independently.

Again, we have two ways to layer ParsecT and StateT transformers on top of Identity: ParsecT s () (State u) a and StateT u (Parsec s ()) a. Is there a difference?

Recall that Parsec is a backtracking monad. It can execute one branch, fail, and then execute another branch. Do we observe the state changes that happened in the aborted branch?

This will be our test code:

((put 1 >> char 'b') <|> char 'a') >> get

We’ll try to run it with the input "a" and initial state 0. First, Parsec executes the left branch of <|>, fails, and proceeds with the right branch. If the state is “global”, then the final result will be 1. If the state is subject to backtracking, then we’ll see 0.

In the ParsecT s () (State u) a case, the base monad is State. ParsecT works inside that monad and cannot revert the effects that happened there. (It may seem that it could remember the state of a branch using get and then restore it using put; but a monad transformer has to be polymorphic in the underlying monad and thus cannot rely on existence of get and put.)

And indeed, the code

import Text.Parsec
import Control.Monad.State

main = print $ flip evalState 0 $ runParserT p () "-" "a"
where p = ((put 1 >> char 'b') <|> char 'a') >> get

prints Right 1.

Let’s now try the second option: layering StateT on top of Parsec. We cannot use the code above as is, because types do not match: e.g. char 'b' has type Parsec s () Char, but we need StateT u (Parsec s ()) Char. Such an upgrade is done by the lift function.

Sometimes, however, we need to “downgrade” computations from StateT u (Parsec s ()) a to Parsec s () a.

For example, to implement a <|> operator of type

(<|>) :: StateT u (Parsec s ()) a -> StateT u (Parsec s ()) a -> StateT u (Parsec s ()) a

we need first to downgrade the arguments to plain Parsec computation, invoke Parsec’s own <|> and then upgrade the result back to StateT.

import Text.Parsec as P
import Control.Monad.State

main = print $ runParser (evalStateT p 0) () "-" "a"
where
p = ((put 1 >> char 'b') <|> char 'a') >> get

char = lift . P.char

a <|> b = StateT $ \s ->
runStateT a s P.<|> runStateT b s

Notice how we explicitly run both branches of <|> with the same state. It’s no surprise now that the state will be “backtracked” and the result will be Right 0. By the way, this is the same result as would be achieved using Parsec’s internal state.

January 02, 2012 12:00 AM

January 01, 2012

Johan Tibell

More monitoring goodies

The ekg library now supports user-defined counters. User-defined counters can be used to track custom program state (e.g. the number of requests served by a server.)

Adding a new counter to a program is easy. Here's a small example program:

main = do
    handle <- forkServer "localhost" 8000
    counter <- getCounter "iterations" handle
    let loop = do
            inc counter
            loop
    loop

The counter will show up in a new section of the web interface, together with a number of built-in counters (e.g. garbage collector and memory usage counters):

All user-defined counters are also available via the REST API.

I've expanded the REST API to allow you to retrieve single counters, with the caveat that single counters can only be retrieved using the "text/plain" MIME type, as the JSON specification doesn't allow simple values (e.g. integers) to occur at the top-level.

by Johan Tibell (noreply@blogger.com) at January 01, 2012 11:42 PM

apfelmus

FRP - Upcoming API changes in reactive-banana 0.5

Version 0.5 of my library reactive-banana for functional reactive programming is looming on the distant horizon and I have a significant API change to discuss with you, dear users.

(Do you like bananas? Support the project with a small donation: Flattr this!)

The first, harmless change is that I will remove the FRP class. In the beginning, it was useful for testing my push-driven implementation against the model implementation. But now I have found a way to do that without the overhead of an additional class, so we can get rid of it; a move that greatly simplifies type signatures.

Unfortunately, the second change will make type signatures more complicated again: I intend the main types Event, Behavior and NetworkDescription to get an additional parameter t, like this:

accumE   :: a -> Event t (a -> a) -> Event t a
fromPoll :: NetworkDescription t (Behavior t a)

Of course, I have profound reasons for this change, in total three:

  1. The parameter will be used to guarantee the absence of time leaks in my upcoming implementation of dynamic event switching. There will be types like

    switchE :: (forall t. Trim s t (Event s (Event t a))) -> Event s a
    
  2. Wolfgang Jeltsch has developed beautiful theory that reinterprets functional reactive programming as the Curry-Howards Isomorphism for constructive temporal logic. Brilliant! Of interest for us is that the seemingly superfluous parameter t has meaning, it corresponds to a “starting time”.
  3. Without the parameter, it is actually possible to break library internals. This is similar to how the parameter for the ST monad prevents mutable state from escaping. For instance, the following program uses an event e way out of its intended scope:

    compile $ do
        e <- fromAddHandler something
        let anotherNetwork =
                actuate <=< compile $ reactimate e  -- Oops!
        reactimate $ anotherNetwork <$ e
    

    This program can be outlawed by giving compile a second-rank type.

    compile :: (forall t. NetworkDescription t ()) -> IO EventNetwork
    

Still, I’m not entirely happy with the change, because it makes type signatures slightly more annoying, especially for new users. Also, GHC’s recent monomorphic local bindings restriction might force you to spell out type signatures.

Concerning 1, Gergely Patai offers another option, but it’s not really simpler than the additional parameter. Point 2 can be ignored. I’m willing to sacrifice point 3, the idea being that the library comes with an API for beginners and an API for advanced users

type Beginner.Event    = Adept.Event Start
type Beginner.Behavior = Adept.Behavior Start

where the beginner version does away with the seemingly superfluous type parameter. If you want to use dynamic event switching, you have to become an adept and also change a few type signature. Code will mostly stay compatible, though.

Should I go ahead with adding the type parameter or do you think that the convenience of the beginner version is worth the cost? Comments appreciated!


Flattr this

January 01, 2012 05:03 PM

December 30, 2011

Darcs

darcs weekly news #89

News and discussions

  1. Owen Stephens summarized his work on the darcs-bridge summer of code and did some followup work on it:
  2. Eric Kow introduced an 'announce' topic filter on the mailing list to provide a low-traffic announcement subset of the list:
  3. Mark Stosberg announced a donation of $500 on behalf of Summersault. Remember you too can donate by going on the donations page of darcs.net:
  4. Interested by participating in the next Darcs sprint in March? Add yourself to the wiki page:

Issues resolved in the last week (1)

issue1705 Eric Kow

Patches applied in the last week (27)

See darcs wiki entry for details.

by guillaume (noreply@blogger.com) at December 30, 2011 01:36 PM

Leon P Smith

Announcing postgresql-simple

I’ve done a bit of database programming off and on over the years, and when I started into a larger project and decided to use Haskell and PostgreSQL, I didn’t understand exactly how bad of a choice Haskell was for PostgreSQL development at the time. So postgresql-simple and postgresql-libpq is hopefully a small step towards remedying the situation. This work is a fork of Bryan O’Sullivan’s mysql-simple library and Grant Monroe’s libpq binding, with some code informed by Chris Done’s pgsql-simple library, and I’d to thank all for their terrific work.

So, there are five things that, taken together, distinguish postgresql-simple from the other PostgreSQL options on hackage:

1. postgresql-simple uses libpq, PostgreSQL’s official client library for C

Chris Done’s pgsql-simple is a pure Haskell implementation of a small subset of the PostgreSQL frontend/backend protocol. I do think there is technical merit to a native Haskell implementation, and in particular I’ve identified a potentially significant advantage that cannot be achieved via libpq. However writing a native implementation is a significant undertaking, and such efforts invariably lag behind in terms of features. And though pgsql-simple is experimental software that hasn’t been officially released and I don’t want to be unfair to it, pgsql-simple performs badly. I sped up a program I had been using to test pgsql-simple by over a factor of 20 simply by switching to postgresql-simple.

Among other things using libpq means that postgresql-simple gets the full range of connection and authentication options out of box, including support for Unix Domain Sockets, SSL, and GSSAPI.

2. postgresql-simple uses libpq-based string escaping.

HSQL doesn’t provide parameterized SQL statements at all, requiring programmers to dynamically generate sql and handle escaping issues themselves. And experience has shown punting on this issue is unacceptable, as dynamically generating SQL is risky and forcing programmers to handle escaping issues is an order of magnitude worse.

HDBC and pgsql-simple support parameterized SQL, but they attempt to do the escaping themselves. The problem is that this is usually a bit more subtle than it first appears, leading to bugs and potential vulnerabilities. For example, in PostgreSQL, escape syntax depends on the value of the standard_conforming_strings server setting, which libpq will detect and accommodate accordingly.

3. postgresql-simple makes no pretense of supporting prepared statements

In PostgreSQL, a prepared statement allows one to send off a parameterized sql statement to a backend, and then re-use that statement as many times as you like by just filling in the parameters. This means the text of the prepared statement does not repeatedly traverse the wire, the backend does not parse the sql query multiple times, the backend re-uses the query plan generated when the statement was prepared, and you get to use protocol-level parameters which can eliminate the need for escaping strings and converting numbers to base-10 and back.

Note that prepared statements are very often, but not universally, advantageous. Starting from nothing, they require two round trips to get any information out, so preparation can be disadvantageous for one-shot queries. Also, occasionally re-planning a query can be advantageous, so infrequently executed queries can be harmed by preparation as well.

HDBC nominally supports prepared statements, but in fact all HDBC-postgresql does is cache some of the preprocessing of the query. No preparation in the database sense ever occurs. So while the lack of support for prepared statements is a definite disadvantage, neither does postgresql-simple pretend to prepare statements when it doesn’t.

4. postgresql-simple provides a simple API that most programmers would find familiar

Takusen appears to be the one PostgreSQL database access library for Haskell that gets basic implementation details more-or-less correct. Unfortunately it exports an esoteric API that is not applicable in all situations. In particular, web applications often use various forms of connection caching or pooling, which is fundamentally incompatible with the deterministic connection resource guarantees provided by Takusen.

Also, the *-simple libraries have a relatively nice interface compared to HDBC. Ultimately that was the breaking point that caused me to spend the time to create postgresql-simple; no one database library did everything I needed, and while working on some new code where my mangled fork of HDBC made the most sense, I realized I really wished I was using pgsql-simple instead. Also, I paid attention to ensure that application code could add support for user-defined PostgreSQL types with a minimum amount of fuss and without modifying the library, something that neither HDBC nor pgsql-simple could really do.

5. Support for Listen/Notify

Listen/Notify is the perfect solution when you want to write a program that responds to changes made to a PostgreSQL database. Informing these programs when changes are available consumes less resources and provides lower latencies than periodic polling. And since you can use a rule or trigger to send the notification, these notifications can be robust; you don’t have to assume that the program making the changes even knows that anybody wants to be informed of the changes. Listen/notify is aware of and consistent with transactions; notifications don’t get sent until the transaction commits. Even if you are willing to put that kind of logic in the database clients, using listen/notify solves the otherwise sticky problem of finding and coordinating with the other clients.

Nothing else on hackage supports asynchronous notifications, unless you use a low-level libpq binding directly.

Looking Forward

Creating a mid-level database access library based on libpq is a significant undertaking, and there is an awful lot that isn’t supported, including prepared statements, binary data formats, copy in/copy out support, more and better support for PostgreSQL data types, and PostGIS support. Many other things could be improved, such as using libpq asynchronously, better end-to-end typechecking that discovers more errors at compile time, and other interface improvements. For example, SqlQQ is a simplistic quasiquoter to improve the literal syntax for multi-line SQL queries; one could certainly imagine extending the syntax to support including a $haskellvariable as a SQL parameter instead of going through the syntactic indirection of ? in the SQL string with haskellvariable in a separate Haskell parameter.

Funnily enough, I ran across this thread in which PostgreSQL luminaries were complaining about the quality of Python’s PostgreSQL libraries. And Python is considerably ahead of Haskell in this regard.


by lpsmith at December 30, 2011 12:17 PM

December 29, 2011

Bryce Verdier

Pangrams


If you haven’t figured it out by now, I enjoy solving problems. And over the course of the last year or two, I’ve learned that interview questions make for great problems to work on. Actual interview problems are nice because they are usually quick, but have a quirk or two in there that makes them challenging, unlike simple questions like the Fizz Buzz problem that just checks if you have the most basic coding skills. (Has anyone actually been asked that question in an interview?)

The most recent problem I got to sink my teeth into (found it on a recruiting site, but not going to share where I got it; wouldn’t be fair to the company posting the problem) is for finding pangrams in sentences. If you don’t know what a pangram is Wikipedia defines them as, “a sentence using every letter of the alphabet at least once.” Yeah, I didn’t know what they were either until I started programming this little puzzle. Here is the code:

  1. module Main (main) where
  2.  
  3. import System (getArgs)
  4. import qualified Data.Set as S
  5. import qualified Data.Text as T
  6. import qualified Data.Text.IO as TI (readFile)
  7.  
  8. buildList :: FilePath -> IO [T.Text]
  9. buildList filename = TI.readFile filename >>=
  10. return . map (T.toLower . T.filter (/=' ')) . T.lines
  11.  
  12. compareAndPrint :: S.Set Char -> String
  13. compareAndPrint sset = if S.null result
  14. then "NULL"
  15. else S.toList result
  16. where result = S.difference (S.fromList ['a'..'z']) sset
  17.  
  18. main = do
  19. args <- getArgs
  20. sentences <- buildList $ head args
  21. mapM_ (putStrLn . compareAndPrint) $ map( S.fromList . T.unpack) sentences

I came up with the solution pretty quickly by using Sets. Having a set of the alphabet and finding the difference of the letters used in the sentence makes the problem almost trivial. The hard part for me was figuring out how to filter out the spaces and change all characters to lower case in the buildList function. I eventually figured it out, but it took some head against wall action to get it right.

This is going to be my last post for this year. I would like to wish you all Happy Holidays and a Happy New Year. Thank for reading and see you again in 2012. I would also like to thank everyone from planet.haskell.org who decided to read this. Welcome!

by Bryce at December 29, 2011 05:30 PM

Kevin Reid (kpreid)

Compatibility testing help wanted for Cubes

I upgraded to Mac OS X 10.7 Lion this month, and ever since then Cubes has crashed or otherwise misbehaved on my machine. However, I have only the one to test on, so I would like your help in figuring out what the extent of the incompatibility is — whether this is a GPU driver bug or something Cubes is doing wrong.

If you have a WebGL-capable browser (such as Chrome or Firefox) and GPU, and especially if you're running Lion or are otherwise similar to my configuration, please run Cubes (no installation is required) and tell me whether it works and what your system configuration is, including these properties:

PropertyI have
Hardware modelMacBookPro5,1
GPU modelNVIDIA GeForce 9600M GT
OS versionMac OS X 10.7.2
GPU driver version?
Browser versionChrome 17.0.963.12, Firefox 8.0

(For some MacBook Pro models such as mine, the GPU used (9600M or 9600M GT) depends on the “Graphics” setting in Energy Saver preferences.)

The problems I observe are:

  • All blocks (terrain) being flat white except for lighting; the sky and particle effects still have color. (For the interested, this appears to be a mis-execution of the over-unity-to-white code introduced in commit 48674f….)
  • After some amount of usage, a crash (in Firefox, the browser exits; in Chrome, the display goes gray and the Web Inspector console says “WARNING: WebGL content on the page might have caused the graphics card to reset” while the in-page text says “Previous GL errors: INVALID_OPERATION”).

I will appreciate any help as this is making it quite impractical to work on Cubes.

by Kevin Reid (kpreid) (kpreid@switchb.org) at December 29, 2011 01:57 PM

December 26, 2011

Magnus Therning

Adjusting to Sweden with XKB

Having lived outside of Sweden for about a decade I’ve grown accustomed to non-Swedish keyboard layouts, first the US (as it’s widely used in The Netherlands) and later on the UK layout. Moving back to Sweden had me swearing over the layout used here within only a few days. The placement of “{[]}” is especially painful. Clearly the Swedish layout wasn’t designed for developers! Rather than go on muscle memory I decided to first attempt a small change to the X key mappings.

I found a good description of per-user XKB configuration after a bit of searching. Then I modified it slightly to fit better in my Arch-based LXDE system.

The XKB config

I started with removing all the configuration I’d previously put into /etc/X11/xorg.conf.d — if I’m to use per-user configuration then there should be no system-wide settings at all. Then I put the output of setxkbmap -print into ~/.xkb/maps/$(hostname) as a starting point. The main goal is to move the characters that requires awkward single-hand combinations with AltGr to slightly more comfortable mappings. After a bit of experimentation I settled on the following (which I put in ~/.xkb/symbols/sedev)

1
2
3
4
5
6
7
8
9
10
partial alphanumeric_keys
xkb_symbols "devkeys" {
    key <AD01> { [ q, Q, backslash ] };
    key <AD02> { [ w, W, asciitilde ] };
 
    key <AC01> { [ a, A, braceleft ] };
    key <AC02> { [ s, S, bracketleft ] };
    key <AC03> { [ d, D, bracketright ] };
    key <AC04> { [ f, F, braceright ] };
};

After setting it manually and verifying that the new mappings work I added it to my keymap, which ended up looking like this

1
2
3
4
5
6
7
xkb_keymap {
    xkb_keycodes  { include "evdev+aliases(qwerty)" };
    xkb_types     { include "complete" };
    xkb_compat    { include "complete" };
    xkb_symbols   { include "pc+se(nodeadkeys)+inet(evdev)+capslock(swapescape)+compose(paus)+sedev(devkeys)" };
    xkb_geometry  { include "pc(pc104)" };
};

Tying it together

Now all the remains is to load the new configuration on login. Based on madduck’s example I put the following into ~/.xprofile

1
2
3
4
5
6
# load XKB, if there is one
XKBDIR=${HOME}/.xkb
XKBMAPFILE=${XKBDIR}/keymap/$(hostname)
if [[ -f ${XKBMAPFILE} ]]; then
    xkbcomp -I${XKBDIR} ${XKBMAPFILE} ${DISPLAY}
fi

Now I just have to get used to using the new mappings.

Share/Bookmark

by Magnus at December 26, 2011 08:12 AM

December 25, 2011

Edward Kmett

Searching Infinity Parametrically

Andrej Bauer recently gave a really nice talk on how you can exploit side-effects to make a faster version of Martin Escardo's pseudo-paradoxical combinators.

A video of his talk is available over on his blog, and his presentation is remarkably clear, and would serve as a good preamble to the code I'm going to present below.

Andrej gave a related invited talk back at MSFP 2008 in Iceland, and afterwards over lunch I cornered him (with Dan Piponi) and explained how you could use parametricity to close over the side-effects of monads (or arrows, etc) but I think that trick was lost in the chaos of the weekend, so I've chosen to resurrect it here, and improve it to handle some of his more recent performance enhancements, and show that you don't need side-effects to speed up the search after all!

First, we'll need to import a few things:

 
{-# LANGUAGE RankNTypes #-}
 
import Data.Maybe (fromMaybe)
import Control.Applicative
import Data.IntMap (IntMap)
import qualified Data.IntMap as IntMap
import Control.Monad
import Control.Monad.Trans.Class
import Data.Functor.Identity
 

What are looking for is an implementation of Hilbert's epsilon.

This is a formal mechanism for eliminating existentials over some non-empty set X by defining a function

 
ε: (X -> Prop) -> X
 

such that if there exists an x in X such that p(X) holds then p(ε(p)) holds.

As noted by Andrej, we could reify this constructively as a function "epsilon :: (X -> Bool) -> X" for some X.

Now, for some sets, Hilbert's epsilon is really easy to define. If X is a finite set, you can just exhaustively enumerate all of the options returning a member of X such that the property holds if any of them do, otherwise since X is non-empty, just return one of the elements that you tested.

This would be a pretty boring article and I'd be back to eating Christmas dinner with my family if that was all there was to it. However, certain infinite spaces can also be searched.

Last year, Luke Palmer wrote a post on "Searchable Data Types" that might also serve as a good introduction. In that article he led off with the easiest infinite space to search, the lazy naturals, or the 'one point compactification of the naturals'. That is to say the natural numbers extended with infinity.

 
data LazyNat = Zero | Succ LazyNat
infinity :: LazyNat
infinity = Succ infinity
 

Now we can implement Palmer's epsilon (called lyingSearch in his article).

 
palmer :: (LazyNat -> Bool) -> LazyNat
palmer p
  | p Zero = Zero
  | otherwise = Succ $ palmer $ p . Succ
 

The trick to making this work is that we place a requirement that the predicate that you pass has to terminate in a bounded amount of time no matter what input you give it, and since we're working with the naturals extended with infinity, if no natural satisfies the predicate, we'll just keep returning a longer and longer chain of Succ's, effectively yielding infinity.

To check to see if the returned number satisfies the predicate you can always use p (palmer p). The predicate is required to terminate in finite time, even when given infinity, so this will yield a Bool and not bottom out unless the user supplied predicate takes an unbounded amount of time.

I posted a reply to Luke's article when it came up on reddit which included a Hinze-style generic implementation of his lyingSearch predicate, which you can see now is just Hilbert's epsilon for arbitrary recursive polynomial data types.

http://www.reddit.com/r/haskell/comments/e7nij/searchable_data_types/c15zs6l

Another space we can search is the Cantor space 2^N.

 
type Cantor = Integer -> Bool
 

With that we jump clear from countable infinity to uncountable infinity, but it can still be searched in finite time!

This is the space we'll be paying attention to for the rest of this article.

First we'll define how to "book a room in Hilbert's Hotel."

 
infixr 0 #
(#) :: Bool -> Cantor -> Cantor
(x # a) 0 = x
(x # a) i = a (i - 1)
 

Then this can be used to obtain the following implementation of Hilbert's epsilon for the Cantor space, attributed by Andrej to Ulrich Berger.

 
berger :: (Cantor -> Bool) -> Cantor
berger p =
  if ex $ \a -> p $ False # a
  then False # berger $ \a -> p $ False # a
  else True  # berger $ \a -> p $ True # a
  where ex q = q (berger q)
 

This version is particularly close in structure to the one for searching the LazyNats, but it is dreadfully slow!

It would be nice to be able to search the space faster and that is just what Martin Escardo's improved version does, through a more sophisticated divide and conquer technique.

 
escardo :: (Cantor -> Bool) -> Cantor
escardo p = go x l r where
  go x l r n =  case divMod n 2 of
    (0, 0) -> x
    (q, 1) -> l q
    (q, 0) -> r $ q-1
  x = ex $ \l -> ex $ \r -> p $ go True l r
  l = escardo $ \l -> ex $ \r -> p $ go x l r
  r = escardo $ \r -> p $ go x l r
  ex q = q (escardo q)
 

To proceed from here I'll need a State monad:

 
newtype S s a = S { runS :: s -> (a, s) }
 
instance Functor (S s) where
  fmap f (S m) = S $ \s -> case m s of
    (a, s') -> (f a, s')
 
instance Applicative (S s) where
  pure = return
  (< *>) = ap
 
instance Monad (S m) where
  return a = S $ \s -> (a, s)
  S m >>= k = S $ \s -> case m s of
    (a, s') -> runS (k a) s'
 

And now we've reached the point. From here, Andrej's pure code ends, and his side-effecting ocaml and custom programming language start. The first thing he does is compute the modulus of continuity by using a side-effect that writes to a reference cell which he very carefully ensures doesn't leak out of scope, so he doesn't have to concern himself with the proposition code editing the value of the reference.

 
let mu f a =
  let r = ref 0 in
  let b n = (r := max n ! r; a n) in
    ignore (f b);
    !r
 

To obtain the same effect we'll instead make a predicate using the state monad to model the single reference cell.

modulus phi alpha = snd $ runS (phi beta) 0 where
  beta n = S $ \ i -> (alpha n, max i n)

But now, we've lost the safety that was implied by the local lexical scope. If we let the type checker give us a type we obtain:

 
-- bad
modulus :: (Num b, Ord s) =>
  ((s -> S s a) -> S b c) -> (s -> a) -> b
 

We can mash b and s together, and try to make the ordering and number agree by claiming that it is instead Real and we'd get the slightly more reasonable looking type:

 
-- still bad
modulus :: Real a =>
  ((a -> S n b) -> S n c) -> (a -> b) -> a
 

In the imperative code, lexical scoping had ensured that no other code could edit the reference cell, but with this type we don't have that. The predicate is allowed to use arbitrary state actions to muck with the modulus of convergence even though the only thing that should be editing it is the wrapper beta that we placed around alpha.

But how can we ensure that the end user couldn't gain access to any of the additional functionality from the monad? Parametricity!

 
-- getting better
modulus :: Real a =>
  (forall f. Monad f => (a -> f b) -> f c) ->
  (a -> b) ->
  a
 

Here the only thing you are allowed to assume about f is that it forms a monad. This gives you access to return and >>=, but the predicate can't do anything interesting with them. All it can do is work with what is effectively the identity monad, since it knows no additional properties!

We can have mercy on the end user and give them a little bit more syntactic sugar, since it doesn't cost us anything to let them also have access to the Applicative instance.

 
-- good
modulus :: Real a =>
  (forall f. (Monad f, Applicative f) => (a -> f b) -> f c) ->
  (a -> b) ->
  a
 

With that we can show Andrej's version of the modulus of convergence calculation does not need side-effects!

>
> modulus (\a -> a 10 >>= a) (\n -> n * n)
100
 

Admittedly plumbing around the monadic values in our proposition is a bit inconvenient.

His next example was written in a custom ocaml-like programming language. For translating his effect type into Haskell using parametricity, we'll need a CPS'd state monad, so we can retry from the current continuation while we track a map of assigned values.

 
newtype K r s a = K { runK :: (a -> s -> r) -> s -> r }
 
instance Functor (K r s) where
  fmap = liftM
 
instance Applicative (K r s) where
  pure = return
  (< *>) = ap
 
instance Monad (K r s) where
  return a = K $ \k -> k a
  K m >>= f = K $ \k -> m $ \a -> runK (f a) k
 

For those of you who have been paying attention to my previous posts, K r s is just a Codensity monad!

 
neighborhood ::
  (forall f. (Applicative f, Monad f) => (Int -> f Bool) -> f Bool) ->
  IntMap Bool
neighborhood phi = snd $ runK (phi beta) (,) IntMap.empty where
  beta n = K $ \k s -> case IntMap.lookup n s of
    Just b -> k b s
    Nothing -> case k True (IntMap.insert n True s) of
      (False, _) -> k False (IntMap.insert n False s)
      r -> r
 

With that we can adapt the final version of Hilbert's epsilon for the Cantor space that Andrej provided to run in pure Haskell.

 
bauer ::
  (forall f. (Applicative f, Monad f) => (Int -> f Bool) -> f Bool) ->
  Cantor
bauer p = \n -> fromMaybe True $ IntMap.lookup n m where
  m = neighborhood p
 

With a little work you can implement a version of an exists and forAll predicate on top of that by running them through the identity monad.

 
exists ::
  (forall f. (Applicative f, Monad f) => (Int -> f Bool) -> f Bool) ->
  Bool
forAll ::
  (forall f. (Applicative f, Monad f) => (Int -> f Bool) -> f Bool) ->
  Bool
 

I've gone further in playing with this idea, using monad homomorphisms rather than simply relying on the canonical homomorphism from the identity monad. You can get the gist of it here:

https://gist.github.com/1518767

This permit the predicates themselves to embed some limited monadic side-effects, but then you get more extensional vs. intensional issues.

An obvious direction from here is to fiddle with a version of Martin Escardo's search monad that takes advantage of these techniques, but I'll leave the exploration of these ideas to the reader for now and go enjoy Christmas dinner.

Happy Holidays,
Edward Kmett

by Edward Kmett at December 25, 2011 06:19 AM

December 24, 2011

Ken T Takusagawa

[ueneqbhw] Window shading without compositing

A minimalist window manager without title bars or borders needs a way to indicate which window has focus.

If you have compositing, one way to do it is, if you press the "Tell Me Which Window is Focused" key combination, all the non-focused windows get shaded darker (or blurred, etc.).  The same can happen when cycling or switching between windows.

If you don't have compositing, you can get an extreme version of darker shading by painting over the non-focused windows black (or showing desktop background), temporarily completely hiding the non-focused windows.

An idea for XMonad, so I can recover that 1 pixel wide red border of screen real estate.  Perhaps someone has already implemented this.

by Ken (noreply@blogger.com) at December 24, 2011 08:48 PM

Well-Typed.Com

Parallel Haskell Digest 7

GHC 7.4 is coming! There is loads to look forward to, but sometimes, it's the little things that count. For example, do you hate the fact that you can't just flip on an +RTS -N without having to first recompile your program, this time remembering to throw an -rtsopts on it? Duncan Coutts has relaxed the requirement so that commonly used RTS options can be used without it. This flag was originally implemented to counter security problems for CGI or setuid programs; however, it was also a hassle for regular users because it got in the way of common options like -eventlog, -N, or -prof. The GHC 7.4 RTS will make a better tradeoff between security and convenience, allowing a common set of benign flags without needing -rtsopts.

That's the sort of thing that the Parallel GHC Project is about. We want to push parallel Haskell out into the real world, first by helping real users (our guinea pigs industrial partners) to apply it to their work, second by making it easier to use (tools, libraries), and finally communicating more about it (this digest).

In this month's digest, we'll be catching up on news from the community. After the holidays, we'll be back with some new words of the month exploring a bit of concurrent Haskell. In the meantime, happy hacking and Merry Christmas!

News

Job Opportunity at Parallel Scientific

Peter Braam wants you, parallel Haskeller!

Parallel Scientific, LLC is a Boulder, CO based early stage, but funded startup company working in the area of scalable parallelization for scientific and large data computing. We are implementing radically new software tools for the creation and optimization of parallel programs benefiting applications and leveraging modern systems architecture. We build on our mathematical knowledge, cutting edge programming languages and our understanding of systems software and hardware. We are currently working with the Haskell development team and major HPC laboratories world wide on libraries and compiler extensions for parallel programming.

Note the mandatory Haskell experience and the desirability of “in depth knowledge of core Haskell libraries for parallel programming (NDP, REPA etc)”.

Parallel GHC Project Update

The Parallel GHC Project is an MSR-funded project, run by Well-Typed, with the aim of demonstrating that parallel Haskell can be employed successfully in real world projects.

Our most recent work has been in polishing the upcoming ThreadScope release that we previewed this September at the Haskell Implementor's Workshop. This new release comes with goodies for users of Strategies or the basic par/pseq parallelism: spark creation/conversion graphs, visualisations showing your spark pools filling and emptying, and histograms displaying the distribution of spark sizes. All this with the aim of helping you gain deeper insight, not just what your program is doing but why.

We've also done backend work to make ThreadScope even more useful further down the road. First, we have improved the ghc-events package by encoding the meanings of events in state machines. This makes it possible to validate eventlogs, and doubles as an always up-to-date source of code as documentation. Second, we have extended the GHC RTS to emit the startup wall-clock time and Haskell threads labels to the eventlog. The wall-clock time event allows us to synchronise logs for simultaneous processes, brining us a step closer to using ThreadScope on distributed programs. Named Haskell thread make it easier to distinguish threads from each other.

Finally, we have been exploring the use of Cloud Haskell for high performance computing on clusters. To do this we would need to abstract Cloud Haskell over different transport mechanisms, that is to develop a robust Cloud Haskell implementation sitting on top of a swappable transport layer. We have posted an initial design for this layer on the parallel-haskell list. We have taken the substantial feedback into consideration and will be sending a revised design and recording it in a page on the GHC wiki. Meanwhile, we are working to further validate our design on simple models of both the transport layer and a cloud Haskell layer on top. Longer term, we aim to implement some transports, an IP transport in particular and perhaps a single-node multi-process transport using forks and pipes.

Tutorials and Papers

  • Tutorial: Deterministic Parallel Programming in Haskell (7 Oct)

    Well-Typed's Andres Löh presented a parallel programming tutorial at the recent Haskell in Leipzig meeting. The tutorial comes with slides, exercises, sample code. It paints a picture of the parallel Haskell landscape, and then focuses on one of the many possible approaches (namely, strategies). One nice feature of the tutorial is an emphasis on practicalities, for example, on using ThreadScope to figure out where performance goes wrong in a program. So if you're looking for a way to get started using on parallelism to speed up your Haskell code, give Andres' tutorial a try!

  • Parallel Genome Assembly with Software Transactional Memory (27 Oct)

    Ketil Malde wrote up some of his experiences using STM to parallelise an inherently complicated program best solved with multiple interacting threads. His article demonstrates that a program using STM is able to successfully parallelize the genome scaffolding process with a near linear speedup. Ketil would be interested in any feedback the community may have.

Blogs and Packages

Actors, actors everywhere

  • remote: Cloud Haskell is here! (27 Oct)

    You may have been hearing a lot about Cloud Haskell lately, the new Erlang-ish distributed programming library for Haskell. Now's your chance to see what all the fuss is about! Jeff Epstein has uploaded the remote package to Hackage, so take it for a spin by doing

    cabal update
    cabal install remote
    

    Library documentation is on the Hackage page, and more details are available in the paper Towards Haskell in the Cloud

  • Distributed storage in Haskell (30 Oct)

    So what are people doing with Cloud Haskell? Julian Porter for one has been working on a distributed monadic MapReduce implementation. Along the way he's produced a general proof of concept for distributed storage. Have a look at Julian's page for a short paper and GitHub page.

  • simple-actors 0.1.0 released (11 Oct)

    Brandon Simmons accounced simple-actors, an EDSL-style library for writing more structured concurrent programs, based on the Actor Model. It was designed for local concurrency, as an alternative to ad-hoc use of Chans, but could be extended to a distributed system by defining appropriate SplitChan instances for some network "channel".

  • Haskell Actors (28 Oct)

    Martin Sulzmann wishes he'd named his actor package “multi-headed-actor”. With the recent interest in actor style concurrency in Haskell, there may be some confusion about the various packages that are out there. The point in Martin's library is being to pattern match over multiple events in the message queue, which makes it easier elegantly express ideas like a marketplace actor which matchmakes buyer/seller messages. While Martin's library is built on concurrent channels, it could be adapted to use distributed channels provided by haskell-mpi or Cloud Haskell. See the paper for more information Actors with Multi-Headed Message Receive Patterns.

More concurrency

  • stm-stats: Retry statistics for STM transaction (9 Oct)

    Joachim Breitner blogged about the stm-stats package, which provides wrappers around atomically to track how often a transaction was initiated and how often it was retried. The stm-stats library is used interally by Factis research, but recently released to the wider Haskell community. In fact, Factis have recently hired Joachim to help them contribute back to the Free Software community where possible. So, thanks, Factis and congratulations, Joachim!

  • How to deal with concurrent external events? (11 Oct)

    Apfelmus has been scratching his head over a design problem for event-based frameworks such as GUI libraries: how do you deal with events that occur while you are currently handling another event? Apfelmus gave a simple wxHaskell demonstrator illustrating the problem, (A) reacting to an event while handling another one may expose internal invariants but (B) reacting to an event after finishing another one may render it “impossible”, i.e. it should not have happened in the first place. Any thoughts on the dilema?

  • Concurrency And Foreign Functions In The Glasgow Haskell Compiler (24 Oct)

    Leon P. Smith posted an overview of the interaction between Haskell concurrency and FFI calls in GHC. Leon's post walks us through some the basic concepts: capabilities, Haskell threads, OS threads, and bound threads. This could be good place to start before delving into papers or library documentation.

  • iteratee-stm (4 Nov)

    John Lato announced the new iteratee-stm library recently uploaded to Hackage. Iteratee-stm provides an iteratee interface that uses bounded TChans for communication. This makes it simple to run IO in a separate thread from processing.

Parallelism

  • Automatic deparallelization (17 Nov)

    Ken Takusagawa explored a different perspective on parallelism. Instead of adding parallelism to programs, what if we started with too much parallelism and stripped it away to fit reality?

    Consider always writing code in a style using egregious fine grained parallelism: assume lots of cores with no communication latency and no overhead. It is the compiler's job to deparallelize (unparallelize, serialize) the program to run on the actual number of cores available, taking into account communication latency and the overhead of parallelization

    Oh, and [qkhsskbg]

  • Introducing Speculation (22 Jul 2010)

    Recently, I got a chance to catch up with Edward Kmett, getting my mind twisted into delightful funny shapes in the process. Edward mentioned his speculation library, yet more parallelism in Haskell! The library is based on the paper Safe Programmable Speculative Parallelism by Prakash Prabhu et al. It provides a way to parallelise inherently sequential algorithms (eg. lexing, Huffman decoding) by guessing the value of intermediate results. You start working in parallel to build work off the guess, only discarding it if the guess turns out to be wrong later on. Check out Edward's blog and slides for more details.

  • Quasicrystals as sums of waves in the plane (24 Oct)

    Keegan McAllister posted an somewhat hypnotic animation of quasicrystals. His post comes with complete source code for his program using the Repa parallel arrays library. Repa was useful to Keegan because it provides

    • Immutable arrays, supporting clean, expressive code
    • A fast implementation, including automatic parallelization
    • Easy output to image files, via repa-devil
  • Simple library for CAS posted (7 Dec)

    Ryan Newton released IORefCAS, which provides a drop-in replacement for atomicModifyIORef that takes advantage of the new casMutVar# primop from GHC 7.2. Ryan says that “[b]ecause it's an easy change it might be worth trying that for hot IOrefs in your parallel app.”

  • OpenCL 10.2.2 (23 Nov)

    Luis Cabellos has updated the Haskell OpenCL package with better documentation and improved error handling using Control.Exception instead of Either error.

Mailing list discussions

Help wanted

  • Parallel Matrix Multiplication (10 Dec)

    Mukesh Tiwari is trying to teach himself parallel Haskell (welcome!). He's gone through Real World Haskell and the tutorial by Simon Peyton-Jones and Satnam Singh, but now trying to implement a parallel matrix multiplication function, he finds himself with no sparks converted. Can anybody give Mukesh a hand?

    Mukesh also asked about resources for Parallel Haskell, which would be where I come in. Mukesh, have a look at the parallel Haskell portal: http://www.haskell.org/haskellwiki/Parallel

Cloud Haskell

  • Cloud Haskell now on Hackage (27 Oct)

    Jeff Epstein's announcement that he had uploaded “remote” to Hackage was greeted with joy and a somewhat lengthy discussion on package/module naming. It looks like the modules will be moved from 'Remote' to 'Control.Distributed.Actor' or 'Control.Distributed.Process' to match the approach used for the concurrency packages. The final package name seems to be distributed-process.

    Anybody got a paintbrush?
  • Haskell Cloud and Closures (1 Oct)

    Fred Smith gave Cloud Haskell a try, using it to remotely compute the plus function. Now he wants to be able to send a function to a remote host, no matter if the function is locally declared or at the top level. Erik de Castro Lopo replied that this was a known limitation with the only known workaround being to move the required function to the top-level. Chris Smith pointed out that while the current restrictions may be too tight, there is good reason to have them. As for alternatives approaches to serialising functions, David Barbour suggested maybe looking at the tangible values work by Conal Elliot.

  • Feedback on Cloud Haskell transport layer interface (2 Nov)

    As I mentioned in the Parallel GHC Project update, we've been looking quite a bit into Cloud Haskell lately. Duncan Coutts posted a request for feedback on the design for a Cloud Haskell transport layer interface. We're hoping one day to make use of Cloud Haskell on for high performance computing on clusters. To do this, we hope to develop a robust Cloud Haskell implementation sitting on top of a swappable transport layer, for example, an IP transport, or a single-node multi-process transport using forks and pipes.

    One issues that emerged from the discussion is how to deal with potentially a plethora of paramaters (eg. buffered vs eager? ordered? reliable?) associated with connection/endpoint creation. It doesn't help that each connection type may have its own set of parameters. Is it enough to be able to set and forget them during transport session initialisation, or is it essential for Cloud Haskell be able to set these parameters differently for different connections in the same session?

  • Parallel Haskell in industry (7 Nov)

    Sébastien Lannez also got a chance to try out Cloud Haskell. The remote package uploaded by Jeff seems to work well and — dabblers take note — the examples shipped with the code are very easy to adapt. Before digging deeper, Sébastien wanted to know more about

    1. performance limitations
    2. communication requirements/overheads
    3. stability
    4. already developed applications

    Jeff cautioned that while he thinks Cloud Haskell could be a good platform to develop distributed applications, it's still very much research software and a work in progress. Don't stake your company on Cloud Haskell just yet.

    That said, Duncan Coutts added, we are pretty happy with the design and optimistic about developing a robust implementation, because we can build it as an ordinary Haskell library without requiring tricky extensions to the runtime system. As for Sébastien's fourth question, a couple of Parallel GHC Project partners are rather keen on Cloud Haskell. We are working on the implementation and will hopefully have more to report on performance, overheads and other issues we encounter.

Multicore performance

  • SMP parallelism increasing GC time dramatically (5 Oct)

    It takes a village to tune a program. Tom Thorne has a program with a function does some fairly intensive calculations on with hmatrix. When Tom tries to get some simple parallelism on his 12 core machine, replacing a map with a parMap rdeepseq, he finds GC time going through the roof, from 1s (1.7%) to 248s (40.5%). Is the big scary number just an artefact of how GC time is reported, or is something really wrong?

    ThreadScope is a good first response here and Tom was duly nagged by the community. Tom promises to give it a go, although the last time he tried, the event log output produced about 1.8GB, and then crashed. The ThreadScope team would love to get hold of any hints about reproducing the crash.

    Ryan Newton observed that GC aside, the program does not appear to be scaling; the mutator time itself isn't going down with parallelism. Tom improved the parallelism a bit, breaking the work into chunks and spreading it around more evenly, and provided he disables the parallel GC, it turns much faster and outperforms the sequential version. Having loads of RAM to play and code that doesn't use much memory, Tom then tried telling the RTS to perform GC less often. This worked. Increasing the minimum allocation area size from its default 512K with +RTS -A32M allows Tom to get performance with the parallel GC comparable to that without. Hooray! But there's still this little problem… now Tom's program intermittently segfaults. Getting a bug report out of this may take a while though as Tom attempts to boil it down.

    Meanwhile, Oliver Batchelor offered his experience that enabling more threads than he has cores makes his program get drastically slower. Alexander Kjeldaas and Austin Seipp commented that this is due to GC needing to co-ordinate with blocked threads, and that the problem of oversaturating is well known. There's also the "dreaded last core slowdown" bug which once affected Linux users but seems to have gone away in recent Linux/GHC.

  • AMD Bulldozer modules and Haskell parallelism (13 Oct)

    Herbert Valerio Riedel has been eyeing the AMD FX-8120 Bulldozer processor. Bulldozer cores are not independent from each other, but grouped into pairs. So Herbert wanted to know how this might affect Haskell parallelism; would 8 cores really mean 8 or just 4 with slightly better SMT capability? Simon Marlow does not know (benchmarks). Duncan Coutts believes that it should be all fine as the pairing is not at all like hyperthreading.

  • Estimating contention on an IORef hammered with atomicModifyIORef (27 Oct)

    Ryan Newton starts us off with a hypothetical scaling bottleneck: all threads frequently accessing a single IORef using atomicModifyIORef (Data.IORef). This is commonly understood to be likely a bad idea, but how do we go about measuring just how bad it is? This sort of design appears in monad-par, as pointed out by Johan Tibbell, in the GHC IO manager, so it would be good to know how much it really hurts. (See also Ryan's IORefCAS package which seems to be partly a result of this discussion)

    One approach is to use GHC events to count operations on particular IORefs, then put that through a model that reports whether if the IORef is being used acceptably, or is "hot". Duncan Coutts suggests a simple way to get partway there: stick something like a traceEvent "IORef #3" on each use of atomicModifyIORef and do something like a ghc-events show | grep IORef to at least get an idea which IORefs are hotter than others and some orders of magnitude. We'll hear back from Ryan when he's had a chance to try it.

    Also for the interested, it's worth mentioning that GHC 7.4 will be sporting a new and improved traceEvent, this time exported through Debug.Trace and offering versions for use in pure code and IO both.

  • Way to expose BLACKHOLES through an API? (7 Nov)

    A BLACKHOLE in GHC acts as a placeholder for a thunk that is currently being evaluated. When the thunk is forced, GHC replaces it with a BLACKHOLE object, which it later replaces when it has the evaluation result. In a parallel/concurrent setting, it may happen that two threads are trying to evaluate the same thunk at the same time. In that case, the first thread creates the blackhole, which the second thread notices and blocks on until the evaluation result is available.

    Ryan Newton observes that this blocking is implicit, whereas “[w]hen implementing certain concurrent systems-level software in Haskell it is good to be aware of all potentially blocking operations”. He proposes a mechanism to expose blackholes, for example with a evaluateNonblocking :: a -> IO (Maybe a) that returns Nothing if the value is blackholed. Simon Marlow points out that this may be slightly problematic as thunks depend on each other and “you might be a long way into evaluating the argument and have accumulated a deep stack before you encounter the BLACKHOLE” See the discussion for a counter-proposal.

Data structures and concurrency

  • Efficient mutable arrays in STM (25 Oct)

    Ben Franksen has large arrays (millions of elements) with mostly small elements (Int or Double) and largely chunk-wise access patterns. The current implementation of Control.Concurrent.STM.TArray as Array ix (TVar e) is not nearly efficient enough for his use case. A more efficient implementation would be most welcome, but for now Ben is eyeing Data.Vector.Unboxed from the vector package instead. The idea is to use unsafeIOToSTM to provide shared transactional access to his arrays. Ben thinks he can live with the consequences: IO code being rerun, aborting, and inconsistent views.

    But does the STM transaction actually "see" that he changed part of the underlying array so that the transaction is retried? If not, how does he go about manually implementing this behaviour? Antoine Latter reports that no unsafeIOToSTM is not transactional - IO actions will be performed immediately and are not rolled back, and are then re-performed on retry. David Barbour and Ketil Malde suggested possible implementations, either keeping an extra TVar Int for every chunk in the array, or (B) cleaner and safer: create a “chunked” TArray that works with fixed-width immutable chunks in a spine.

    Another issue that came up is that transactions scale quadratically with the number of TVars touched. Bryan O'Sullivan and Ryan Ingram explained that this is due to choice of data structure (a list) for the STM transaction log, and should be easy to fix.

  • High performance threadsafe mutable data structures in Haskell? (27 Oct)

    Ryan Newton wanted to know if anybody else was working on threadsafe mutable data structures in Haskell. He and the monad-par team were planning to replace their work stealing deques with something more efficient. If anybody else is working in the same general area, teaming up would be great!

    Ryan will be exploring both a pure Haskell approach and one based on wrapping foreign data structures with the FFI. Ultimately, Ryan is aiming for an "abstract-deque" parameterizable interface that abstracts over many variants (bounded/unbounded, concurrent/non-concurrent, single/1.5/double-ended, etc). His current prototype makes use of phantom types and the type families extension to handle all this abstraction, with the intended end result being that someone can create a new queue by setting all the switches on the type (eg. q :: Deque NT T SingleEnd SingleEnd Bound Safe Int <- newQ), but this brings up a set of Haskell language and type system questions. More details in the thread!

  • Persistent Concurrent Data Structures (1 Nov)

    Like Ryan, Dmitri Kondratiev is interested in concurrent mutable data structures, but this time with persistence to boot. His goal is to program at a higher level of abstraction, avoiding the detail bloat that would result from directly using some data storage API (eg. SimpleDB). Dmitri's idea: a module tree of data structures mirroring Data.List, Data.Map, etc but with concurrency and persistence. One would be able to configure through the type interfaces:

    1. media to persist data (file? DBMS?)
    2. caching policy
    3. concurrency configuration (optimistic/pessimistic locking?).

    Dmitri's post prompted some suggestions for packages to look into:

    • safecopy: addresses both the issues of serializing the data and migrating it when the datastructure changes
    • acid-state: builds on top of safecopy to add a notion of transactions to any Haskell data structure
    • TCache: a transactional cache with configurable persistence
    • Haskell web server frameworks (eg. Yesod, Happstack [acid-state was formerly happstack-state]), as some come with persistence support

    Jeremy Shaw and David Barbour had reservations about what Dmitri had in mind when he said "concurrent". How would he deal with transaction boundaries, and would a concurrently modified Data.List variant still be a list? Evan Laforge also expressed skepticism about the viability of abstracting over data stores with potentially very different needs.

Threads, blocking

  • Waiting on input with hWaitForInput' orthreadWaitRead' (17 Oct)

    Jason Dusek would like to use evented I/O for a proxying application, in particular, to fork a thread for each new connection and then to wait for data on either socket in this thread, writing to one or the other socket as needed. He's found two functions which could help, System.IO.hWaitForInput and Control.Concurrent.threadWaitRead but each comes with some difficulties. Is there something like select() that works with handles rather than file descriptors?

    Ertugrul Soeylemez suggested an alternative approach, just plain Concurrent Haskell because “[a] hundred Haskell threads reading from Handles are translated to one or more OS threads using whatever polling mechanism (select(), poll(), epoll) your operating system supports”. He pasted a small echo server to demonstrate the idea. It wasn't entirely clear for Jason how to apply this to a proxy server. Jason has a lazyBridge :: Handle -> Handle -> IO () which writes everything it reads from one handle into the other and vice-versa, but it blocks and does not allow packets to go back and forth. Gregory Collins sketched out a possible solution: how about forkIOing two threads (one for the read end, one for the write end), with a loop over lazy I/O? This works, but is still somewhat surprising.

  • System calls and Haskell threads (3 Nov)

    Andreas Voellmy noticed this in Kazu Yamamoto's Monad Reader article on a high performance web server.

    When a user thread issues a system call, a context switch occurs. This means that all Haskell user threads stop, and instead the kernel is given the CPU time.

    Can that be right? Andreas thought, and Johan Tibell confirms, that when a Haskell thread is blocking a particular OS threads, other Haskell threads can continue run concurrently on other OS threads on other CPUs (see Extending the Haskell Foreign Function Interface with Concurrency).

    Further clarification comes from David Barbour, who points out why Kazu's original statement was correct in the context of the article. While Mighttpd uses Haskell threads for concurrency; it does not go the traditional route of using the RTS -Nx argument to generate OS threads. Instead it gets its parallelism from a "prefork" model that creates separate processes to balance user invocations (each process may itself be running multiple Haskell threads). This unusual approach is chosen to avoid issues with garbage collection.

  • Where threadSleep is defined? (6 Dec)

    Dmitri Kondratiev was looking for a function to make the current process (executing thread) go to sleep for a given time. Felipe Almeida Lessa pointed to the threadDelay function in Control.Concurrent.

Stack Overflow and Reddit

Help and Feedback

If you'd like to make an announcement in the next Haskell Parallel Digest, then get in touch with me, Eric Kow, at parallel@well-typed.com. Please feel free to leave any comments and feedback!

Bikeshed image by banlon1964 available under a CC-NC-ND-2.0 license.

by eric at December 24, 2011 08:32 PM

Joachim Breitner

First (academic) publication

Nice christmas present: I just got message that the paper “Loop subgroups of Fr and the image of their stabilizer subgroups in GLr(ℤ)” that I extracted from my Diploma thesis in Mathematics, and which I submitted to the Israel Journal of Mathematics in November 2010 was finally published today, in “Online First“ form and under the doi 10.1007/s11856-011-0213-3.


Flattr this post

by nomeata (mail@joachim-breitner.de) at December 24, 2011 10:47 AM

December 17, 2011

Alex Mason

OpenCL From Haskell – Hello World!

It’s been a very long time since I’ve even looked at this blog, so I thought I should do something about that. For the past two days, I’ve been working on making the OpenCLWrappers nee OpenCLRaw package more usable, while fixing some bugs while I’m at it.

The main change I wanted to make was to move from everything returning IO (Either ErrorCode a) or IO (Maybe ErrorCode) to a more useable OpenCL monad. The obvious way to do this is to use ErrorT:

> type OpenCL a = ErrorT ErrorCode IO a

(Be sure to comment out the previous line if you decide to use this is a literate haskell file.)

This involved first converting all the IO (Maybe ErrorCode) functions to IO (Either ErrorCode ()) first, and then implementing the OpenCL monad wrapper on top of that. This has resulted in a new set of modules under System.OpenCL.Monad.

To demonstrate how to make use of this initial work, I’ll use a slightly modified version of the canonical CUDA/OpenCL example which takes two vectors of floats, an adds them. My slight modification is to make the kernel compute the hypotenuse between the two vectors. First let’s start with the OpenCL kernel, which should make more clear what we’re trying to do:

__kernel void vectorHypot(
    __global const float * a,
    __global const float * b,
    __global       float * c)
{
    int nIndex = get_global_id(0);
    c[nIndex] = sqrt(a[nIndex] * a[nIndex] + b[nIndex] * b[nIndex]);
}

Next comes the Haskell code. To make use of this code, you’ll need my latest version of OpenCLWrappers from github.

We start, as with any decent literate haskell document, with various imports to break the flow of the document (note to self, investigate using anansi in the future to see if it makes this easier).

> {-# LANGUAGE BangPatterns #-}
> module Main where
> import System.OpenCL.Monad
> import System.OpenCL.Wrappers.Types
> import System.Random (randoms, mkStdGen)
> import Foreign.Marshal.Array (newArray, peekArray)
> import Foreign.Marshal.Alloc (free)
> import Foreign.Ptr (castPtr, nullPtr, Ptr)
> import Control.Monad (forM, forM_)
> import Data.Bits ((.|.))
> import Data.Time (getCurrentTime, diffUTCTime)

Next, we have a function to time execution times. I’m pretty sure it doesn’t work, so I’d love some suggestions for a better way to do this!

> time :: IO a -> IO a
> time x = do
>     !before <- getCurrentTime
>     !a <- x
>     !after <- getCurrentTime
>     print $ diffUTCTime after before
>     return a
And finally on to the guts of the program.We start by reading in the source for the file. Then we create two lists of len‘ random Float values. I’m sure there are better ways to do this too, but I was after a quick (ha!) and dirty result.

The lists are then written to arrays, which are cast to pointers of () (equivalent to void *), so that it matches the types of required by clCreateBuffer later. Then we run the computation (via runHypot), the arrays are read and freed, and we check to see whether the results differ by much, compared to what we expect.

> len = 2^22 :: Int
> main = do
>     str <- readFile "kernel.cl"
>
>     let a = take len $ randoms (mkStdGen 1) :: [Float]
>         b = take len $ randoms (mkStdGen 2) :: [Float]
>
>     pa' <- newArray a
>     pb' <- newArray b
>     pc' <- newArray (replicate len (0.0 :: Float))
>     psize' <- newArray [len]
>     let pa = castPtr pa' :: Ptr ()
>         pb = castPtr pb' :: Ptr ()
>         pc = castPtr pc' :: Ptr ()
>         psize = castPtr psize' :: Ptr ()
>
>     time $ runHypot str pa pb pc
>
>     cres <- peekArray len pc'
>     free pa'
>     free pb'
>     free pc'
>
>     time $ print
>          $ take 100
>          $ map (\(a,b) -> a-b)
>          $ dropWhile (\(a,b) -> abs (a-b) < 10e-7)
>          $ zipWith3 (\a b c -> (sqrt (aa + bb), c)) a b cres

Now we get to the uh… fun part. It turns out that OpenCL is amazingly tedious for such a simple task. The process of running a kernel is as follows:

  1. Find out about the platforms available
  2. Find out about all the devices you have access to. In my case, on my MacBook Pro I have access to one CPU, and one GPU. This gets printed on the following line.
  3. Select a device to run the computation on. I chose the GPU, mainly because choosing the CPU didn’t work for some reason. I may investigate this in the future
  4. Create an OpenCL context, which is used for all sorts of stuff…
  5. Create a command queue for the device. Each action you wish to perform on the device will be queued here, which including moving data to the device’s memory, running the kernels themselves, and moving data back to the host’s memory
  6. Next the program is created from the course passed in (originally from kernel.cl remember?)
  7. Next we compile the program. You can see I’ve had to jump through some hoops to make this work. I technically could have just run clBuildProgram, but the way I’ve done it allows me to get some info about what went wrong with the compilation. Here I print out the compile/error log returned from the compiler if something goes wrong.
  8. Buffers are created, which will have the contents of the host pointers we allocated and passed as arguments copied into them. This step is what moved the data onto the device.
  9. Finally we get to running the kernel. You may be wondering why I’m using the magic number maxWISize `div` 4 here… I’m using it because it worked. I was hoping that just setting the work item size to maxWISize would work, but for some reason it doesn. I might investigate this later…
  10. Now all that’s left is to read the data back from the device, and then free the memory used on the device also. Once this is done, the pointer pc should contain our results.

> runHypot :: String -> Ptr () -> Ptr () -> Ptr () -> IO (Either ErrorCode ())
> runHypot str pa pb pc = runOpenCL $ do > pids <- clGetPlatformIDs -- 1 > dids <- fmap concat $ forM pids $ \pid -> > clGetDeviceIDs pid clDeviceTypeAll -- 2 > infos <- forM dids $ \did -> > clGetDeviceInfo did clDeviceType > liftIO $ print infos > let devid = dids !! 1 -- 3 > ctx <- clCreateContext [] [devid] Nothing nullPtr -- 4 > queue <- clCreateCommandQueue ctx (dids !! 1) [] -- 5 > > prog <- clCreateProgramWithSource ctx str -- 6 > err <- liftIO $ runOpenCL $ clBuildProgram prog [devid] "" Nothing nullPtr > case err of -- ^ 7 > Left err -> do > x <- clGetProgramBuildInfo prog devid clProgramBuildLog > liftIO $ print x > Right x -> return x > kern <- clCreateKernel prog "vectorHypot" -- 8 > > let bytes = fromIntegral len * 4 -- 9 > pad' <- clCreateBuffer ctx (clMemReadOnly .|. clMemCopyHostPtr) bytes pa > pbd' <- clCreateBuffer ctx (clMemReadOnly .|. clMemCopyHostPtr) bytes pb > pcd' <- clCreateBuffer ctx clMemWriteOnly bytes nullPtr > pad <- liftIO $ newArray [pad'] > pbd <- liftIO $ newArray [pbd'] > pcd <- liftIO $ newArray [pcd'] > clSetKernelArg kern 0 8 $ castPtr pad > clSetKernelArg kern 1 8 $ castPtr pbd > clSetKernelArg kern 2 8 $ castPtr pcd > > (DeviceInfoRetvalCLsizeiList (n:_)) <- > clGetDeviceInfo devid clDeviceMaxWorkItemSizes -- ^ 10 > let maxWISize = fromIntegral n > liftIO $ print maxWISize > eventRun <- > clEnqueueNDRangeKernel queue kern -- 11 > [fromIntegral len] > [fromIntegral maxWISize div 4] [] > > eventRead <- clEnqueueReadBuffer pcd' True 0 bytes -- 12 > pc queue [eventRun] > > clEnqueueWaitForEvents queue [eventRun, eventRead] -- 13 > clReleaseMemObject pad' > clReleaseMemObject pbd' > clReleaseMemObject pcd'

To compile, make sure you call ghc with -lopencl or -framework OpenCL on OS X: ghc -framework OpenCL main.lhs

As you can see, this is a hell of a lot of work to go through for such a simple task, and this is why I hope to make a higher level set of wrappers in the nearish future. I would love to be able to do everything using either Vectors or Repa arrays (the latter would be more ideal). It would also be nice to create a DSL for creating OpenCL kernels, but that’s a long way away at the moment.

I think I’ll focus first on making a cleaner interface to things like attaining a context, and allocating data.

Anyway, that’s it for now, let me know if you have any questions, or is anything doesn’t make sense.

by axman6 at December 17, 2011 07:21 AM

December 14, 2011

Alessandro Vermeulen

A transition to static site generation

Today I’ve launched my new blog. It is based on Octopress and works by statically generating the pages and then syncing them with the server.

If you are for example on OS X Lion and installed XCode 4.2 and you run into weird errors like a missing gcc-4.2, and Homebrew throws errors like this:

Error: The linking step did not complete successfully The formula built, but is not symlinked into /usr/local

Please install the gcc package from this nice fellow: osx-gcc-installer

And if you are getting nagged by rb-fsevent. Change

<figure class="code"><figcaption>Gemfile</figcaption>
1
  gem 'rb-fsevent'
</figure>

to

<figure class="code"><figcaption>Gemfile</figcaption>
1
  gem 'rb-fsevent', "0.9.0.pre4"
</figure>

Update The comments have been exported with the Wordpress plugin to Disqus. I’m currently looking at how to highlight code within Disqus comments.

December 14, 2011 09:00 PM

Tom Schrijvers

OCaml @ Incubaid

Nicolas Trangez and Romain Slootmaekers were so kind to give a guest lecture on real-world functional programming in my Functional and Logic Programming course. They gave an overview of their experience (and success!) with OCaml in building distributed storage systems at Incubaid.

Their slides are available here:
http://www.slideshare.net/eikke/realworld-functional-programming-incubaid

by Tom Schrijvers (noreply@blogger.com) at December 14, 2011 08:02 PM

December 13, 2011

Ken T Takusagawa

[dgokhmib] Case statements thwart point-free

Case expressions in Haskell force binding values to names in two places, suggesting future improvements to syntax to encourage point-free programming style.

The first is the expression being examined.  We would like to do this: f . \e -> case e of { ... } . g but without having to name it e.  Here is a simple addition to syntax to fix this: f . LAMBDACASE { ... } . g .  It might make sense to deprecate plain case and always use LAMBDACASE.

The second is capturing values in pattern matching.  Haskell lacks a syntax to take the value from a pattern binding and pass it along to a function without ever having to assign the value to a bound variable name.  We want f $ g $ case e of { Foo x -> x } without ever having to temporarily name x.  Here is one possible weird syntax: FUNCTIONCASE e of { Foo \(f . g) }.   If the constructor has many fields, then the other captured variables are available, for example to a curried function : FUNCTIONCASE e of { Foo x _ _ \(f x y) _ _ y }

This effect is somewhat possible with named fields (record syntax).

We probably want LAMBDAFUNCTIONCASE to combine these.

by Ken (noreply@blogger.com) at December 13, 2011 12:36 AM

December 06, 2011

mightybyte

What's in a name? A lot.

One letter identifier names have been getting bad press recently. The other day a post entitled Name Your Type Variables! got some discussion on the Haskell reddit. I agree with the author's value of good names, but I disagree with the specific examples he uses as well as his sweeping condemnation of one letter names. Since I am the primary author of one of his examples and have contributed code


by mightybyte (noreply@blogger.com) at December 06, 2011 05:08 PM

December 02, 2011

Snap Framework

Announcing: Snap Framework v0.7

The Snap team would like to announce the release of version 0.7 of the Snap Framework. There are some major changes in this release that are not backwards compatible, so be sure to read these release notes before upgrading.

New features / improvements

  • Heist: We realized that the $() syntax Heist uses for accessing splices inside of HTML attributes conflicts with the jQuery ‘$’ function. We changed the syntax to ${} to fix this. This is a major breaking change, so we added a temporary compatibility mode to Heist that you can enable by using the useOldAttributeSyntax function to modify your TemplateState. We will remove this compatibility mode in the next major release, so update your templates ASAP!

  • Snaplets: added getRoutePattern (and setRoutePattern) which allows you to find out the actual route string that matched the currently running handler.

  • Heist Snaplet: added heistInit' to make it possible to use Heist’s onLoad hooks.

  • Added the convenience functions getsRequest and getsResponse to Snap.Core.

  • Fixed a bug in how the route function handled url-encoded path segments.

  • snap-server: added primitives escape the http session in order to support websockets.

API Changes

  • heist: changed the callTemplate function to take general splices rather than constant text splices, and simplified the return type. The old functionality now exists with a simpler return type as callTemplateWithText.

  • heist: Since we are making major breaking changes, we decided to change the name of TemplateState to the more appropriate HeistState. The old name still exists as a type alias, but is marked as deprecated. Update your code now because we will remove all the deprecated functionality in the next major release.

Dependency changes

  • Increased the upper version bound on the time package from 1.4 to 1.5.

  • Increased the upper version bound on the regex-posix package from 0.94.4 to 0.95.2.

  • Increased the upper version bound on the case-insensitive package from 0.4 to 0.5.

  • Increased the lower version bound on attoparsec to 0.10.

  • Increased the version constraints on attoparsec-enumerator to 0.3.*.

by Doug Beardsley (mightybyte@gmail.com) at December 02, 2011 01:35 AM

December 01, 2011

Robert Harper

Of Course ML Has Monads!

A popular meme in the world of PL’s is that “Haskell has monads”, with the implication that this is a distinctive feature of the language, separate from all others.  While it is true that Haskell has popularized the use of monads as a program structuring device, the idea of a monad is not so much an issue of language design (apart from the ad hoc syntactic support provided by Haskell), but rather one of library design.  After all, a monad is just one of a zillion signatures (type classes) with which to structure programs, and there is no particular reason why this one cannot be used in any language that supports even a modicum of modularity.  There is a particular reason why monads had to arise in Haskell, though, which is to defeat the scourge of laziness.  But even in the presence of monads, one still needs things like seq to sequentialize evaluation, because the lazy cost model is so disadvantageous.  In an ironic twist the emphasis on monads in Haskell means that programming in Haskell is rather like programming in an updated dialect of Algol with a richer type structure than the original, but the same overall structure.

Examined from the point of view of ML, monads are but a particular of use of modules.  The signature of monads is given by the definition

signature MONAD = sig
  type 'a monad
  val ret : 'a -> 'a monad
  val bnd : 'a monad -> ('a -> 'b monad) -> 'b monad
end

There are many, many, many structures that satisfy this signature; I needn’t (and, in any case, can’t) rehearse them all here.  One particularly simple example should suffice to give the general idea:

structure Option : MONAD = struct
  type 'a monad = 'a option
  fun ret x = SOME x
  fun bnd (SOME x) k = k x
    | bnd NONE k = NONE
end

This is of course the option monad, which is sometimes used to model the data flow aspects of exceptions, perhaps with some elaboration of the NONE case to associate an exceptional value with a non-result.  (The control flow aspects are not properly modeled this way, however.  For that one needs, in addition, access to some sort of jump mechanism.)

Examples like this one proliferate.  A monad is represented by a structure.  Any structure that provides the facilities specified by the MONAD signature gives rise to the characteristic sequentialization mechanisms codified by it.  Monad transformers are functors that transform one monad into another, with no fuss or bother, and no ad hoc mechanisms required.  Standard modular programming techniques suffice to represent monads; moreover, the techniques involved are fully general, and are equally applicable to other signatures of interest (arrows, or quivers, or bows, or what have you).  Moreover, it is shown in my paper with Chakravarty and Dreyer how to integrate modules into the type inference mechanism of ML so that one can get automatic functor instantiation in those limited cases where it is self-evident what is intended.  This has been implemented by Karl Crary in a prototype compiler for an extension of Standard ML, and it would be good to see this supported in more broadly available compilers for the language.

The bulk of the mania about monads is therefore accounted for by modules.  I have no doubt, however, that you are wondering about the infamous IO monad in Haskell (and it’s associated work-around, unsafePerformIO).  Isn’t that a fundamental feature of the language that cannot be replicated in ML?  Hardly!  It’s entirely a matter of designing the signatures of the standard basis library modules, and nothing more.  The default basis library does not attempt to segregate effects into a monad, but it is perfectly straightforward to do this yourself, by providing your own layer over the standard basis, or to reorganize the standard basis to enforce the separation.  For example, the signature of reference cells might look like this:

signature REF = sig
  type 'a ref
  val ref : 'a -> 'a ref IO.monad
  val ! : 'a ref -> 'a IO.monad
  val := : 'a ref -> 'a -> unit IO.monad
end

Here we are presuming that we have a fixed declaration

structure IO : MONAD = ...

that packages up the basic IO primitives that are already implemented in the run-time system of ML, more or less like in Haskell.  The other signatures, such as those for mutable arrays or for performing input and output, would be modified in a similar manner to push effects into the IO monad.  Et voila, you have monadic effects, just like in Haskell.

There’s really nothing to it.  In fact, the whole exercise was carried out by a Carnegie Mellon student, Phillippe Ajoux, a couple of years ago.  He also wrote a number of programs in this style just to see how it all goes: swimmingly.  He also devised syntactic extensions to the Moscow ML compiler that provide a nicer notation for programming with monads, much as in Haskell, but better aligned with ML’s conventions.  (Ideally it should be possible to provide syntactic support for any signature, not just monads, but I’m not aware of a worked-out design for the general case, involving as it would an intermixing of parsing and elaboration.)

My point is that the ML module system can be deployed by you to impose the sorts of effect segregation imposed on you by default in Haskell.  There is nothing special about Haskell that makes this possible, and nothing special about ML that inhibits it.  It’s all a mode of use of modules.

So why don’t we do this by default?  Because it’s not such a great idea.  Yes, I know it sounds wonderful at first, but then you realize that it’s pretty horrible.  Once you’re in the IO monad, you’re stuck there forever, and are reduced to Algol-style imperative programming.  You cannot easily convert between functional and monadic style without a radical restructuring of code.  And you inevitably need unsafePerformIO to get anything serious done.  In practical terms, you are deprived of the useful concept of a benign effect, and that just stinks!

The moral of the story is that of course ML “has monads”, just like Haskell.  Whether you want to use them is up to you; they are just as useful, and just as annoying, in ML as they are in Haskell.  But they are not forced on you by the language designers!

Update: This post should’ve been called “ML Has Monads, Why Not?”, or “Of Course ML Has Comonads!”, but then no one was wondering about that.

Update: I now think that the last sentence is excessive.  My main point is simply that it’s very simple to go one way or the other with effects, if you have modules to structure things; it’s all a matter of library design.  A variant of ML that enforced the separation of effects is very easily constructed; the question is whether it is useful or not.  I’ve suggested that the monadic separation is beguiling, but not clearly a great idea.  Alternatively, one can say that we’re not that far away from eliminating laziness from Haskell, at least in this respect: just re-do the standard basis library in ML, and you’re a good ways there.  Plus you have modules, and we understand how to integrate type classes with modules, so the gap is rather small.


Filed under: Programming

by Abstract Type at December 01, 2011 02:33 PM

Bryan O'Sullivan

aeson 0.4: easier, faster, better

After months of work, and a number of great contributions from other developers, I just released version 0.4 of aeson, the de facto standard Haskell JSON library. This is a major release, with a number of improvements. Enjoy!

Ease of use

The new decode function complements the longstanding encode function, and makes the API simpler.

New examples make it easier to learn to use the package.

Generics support

aeson’s support for data-type generic programming makes it possible to use JSON encodings of most data types without writing any boilerplate instances.

Thanks to Bas Van Dijk, aeson now supports the two major schemes for doing datatype-generic programming:

  • the modern mechanism, built into GHC itself

  • the older mechanism, based on SYB (aka "scrap your boilerplate")

The modern GHC-based generic mechanism is fast and terse: in fact, its performance is generally comparable in performance to hand-written and TH-derived ToJSON and FromJSON instances. To see how to use GHC generics, refer to examples/Generic.hs.

The SYB-based generics support lives in Data.Aeson.Generic, and is provided mainly for users of GHC older than 7.2. SYB is far slower (by about 10x) than the more modern generic mechanism. To see how to use SYB generics, refer to examples/GenericSYB.hs.

Improved performance

  • We switched the intermediate representation of JSON objects from Data.Map to Data.HashMap, which has improved type conversion performance.

  • Instances of ToJSON and FromJSON for tuples are between 45% and 70% faster than in 0.3.

Evaluation control

This version of aeson makes explicit the decoupling between identifying an element of a JSON document and converting it to Haskell. See the Data.Aeson.Parser documentation for details.

The normal aeson decode function performs identification strictly, but defers conversion until needed. This can result in improved performance (e.g. if the results of some conversions are never needed), but at a cost in increased memory consumption.

The new decode' function performs identification and conversion immediately. This incurs an up-front cost in CPU cycles, but reduces reduce memory consumption.

by Bryan O'Sullivan at December 01, 2011 06:20 AM

November 29, 2011

TypLAB

Launch of the first public Silk site: The Next Web Index

Today the first publicly available Silk site has launched: The Next Web Index!

The Next Web Index is a technology companies database that runs on the Silk platform. The content has been created by The Next Web and is partially based on data from Wikipedia and TechCrunch’s CrunchBase. It contains profiles for over 73,000 companies, 88,000 persons and 6,000 related organizations. Think companies like Groupon, Netflix and Tumblr, persons such as Steve Ballmer, Elon Musk and Jerry Yang, related organizations such as Atomico and Y Combinator, and products such as Google TV.

The Next Web Index is a powerful resource for finding information about technology companies. Silk’s added value becomes particularly clear as you enter explore mode, which lets you search, sort and view the information in a breeze. Use the controls on the left to find the information you’re looking for. Generate sophisticated overviews such as CEOs under 30 who have raised over $1 Million in early funding rounds.

The Index is still work in progress. The editors from The Next Web are updating the content continuously, but as of today it’s out there to be enjoyed by everyone. The pages contain feedback links and all contributions are greatly appreciated by the TNW team.

This is a great day for us here at Silk. Finally everyone can see what we’ve been working on the past two years. The next step is the launch of the Silk editor, which lets everyone create Silk sites like The Next Web Index.

If you didn’t request an invite yet, make sure to request it on our homepage.

by sander at November 29, 2011 04:21 PM

November 26, 2011

Brent Yorgey

Typeclassopedia v2: moving my efforts to the Haskell wiki

I said back in April that I was going to be working on a second edition of the Typeclassopedia. Since then I have worked on it some, but mostly I’ve dithered trying to work out a toolchain for publishing it in multiple formats including PDF and (commentable) HTML. But it’s become increasingly clear to me that this is really just getting in the way of the important work of actually updating the content, and I ought to just choose a simple solution, imperfect though it may be, and get on with it.

In the meantime, it turns out that Henk Krauss (geheimdienst) has been doing the tedious work of porting the Typeclassopedia to the Haskell wiki (thanks!). It looks much nicer than I thought possible for a wiki page, allaying one particular misgiving I had about publishing in wiki format. And this format obviously makes it extremely easy for others to contribute.

The path forward is clear: I hereby declare that the Haskell wiki version of the Typeclassopedia is now the official version, and from now on I will be focusing my efforts there. You can expect a bunch of updates, improvements, and expansions over the next few months. If you have any improvements to suggest, you can make them directly, or leave a comment on the talk page.


by Brent at November 26, 2011 04:39 AM

November 24, 2011

Paul Johnson

Enough with the blue LEDs!

The first blue LED was created by Shuji Nakamura in 1993. It was a technical triumph, and Nakamura eventually managed to win a $9M bonus (after suing his employer). Before then LEDs ranged from red to a slightly yellowish green, and nobody could figure out how to get the wavelengths any shorter. Nakamura's LED was not merely slightly blue; it could go right up into the UV, and was amazingly bright. Today I am typing this in a room illuminated by white LEDs that are made using this technology. Slim, low powered LED-illuminated screens have been created, and blu-ray players use lasers with the same basic semiconductor combination to give us high definition movies on a disk. Science and technology march on, hand in hand.

But that's not what I want to write about.

On Wednesday I plugged in a new cable modem, enabling faster broadband. On the front panel are two LEDs. One is green and shows that the modem has locked on to a signal. The other is blue and flashes. So now I have this bright blue flashing light near me. The coax cable comes into my office near my desk, so I can't move it far. Fortunately I've been able to tuck it into a cubby-hole next to the bookshelf, so its not in my line of sight. Without that I'd probably have had to put it inside a box or something just to hide the LED.

I was not too surprised by the appearance of blue LEDs as an alternative to the old red and green back at the turn of the century; they were novel and eye-catching, which is what manufacturers need if they are to sell stuff. But the trouble is that they are too eye catching; the piercing blue light is actually unpleasant to look at directly, and is far more intrusive than the quiet red and green.

Some manufacturers seem to have got the idea. I have a couple of USB hubs with blue LEDs, but they don't flash and they are sufficiently buried inside that you don't have the piercing gleam. But this cable modem (its a Netgear SuperHub, by the way) doesn't follow that rule. We also bought a moderately expensive audio system a few months ago, and that also has a piercingly bright blue LED on the front. So when we watch TV we have this shining in our eyes just below. We've got some relief by sticking some white tape over the LED, but we shouldn't have to do that.

So could anyone reading this who works for a consumer electronics company please point the design department at this posting. Remember guys; shining bright flashing lights in your customers' eyes may get their attention, but it won't get you repeat business.

I wrote about this before, in 2008. I'm disappointed to have the same problem three years later.

by Paul Johnson (noreply@blogger.com) at November 24, 2011 09:19 PM

November 23, 2011

André Pang (ozone)

Two new mixes

I've been pretty dormant in my music for the past few years, but I have been working on two two mixes in my sparse spare time: "Tes Lyric":/music/tes_lyric, a weird blend of electronica, classical and rock, and "Stage Superior":/music/ss, a progressive house mix. They're up on my "music":/music page now; enjoy!

by André Pang (ozone) (ozone@algorithm.com.au) at November 23, 2011 04:10 PM

November 22, 2011

Sean Seefried

Haskell GADTs in Scala

This is an updated version of an earlier post. Owing to a comment by Jed Wesley-Smith I restructured this post somewhat to introduce two techniques for programming with GADTs in Scala. Thanks also go to Tony Morris.

First we’ll start with a fairly canonical example of why GADTs are useful in Haskell.

{-# LANGUAGE GADTs #-}
module Exp where

data Exp a where
LitInt :: Int -> Exp Int
LitBool :: Bool -> Exp Bool
Add :: Exp Int -> Exp Int -> Exp Int
Mul :: Exp Int -> Exp Int -> Exp Int
Cond :: Exp Bool -> Exp a -> Exp a -> Exp a
EqE :: Eq a => Exp a -> Exp a -> Exp Bool

eval :: Exp a -> a
eval e = case e of
LitInt i -> i
LitBool b -> b
Add e e' -> eval e + eval e'
Mul e e' -> eval e * eval e'
Cond b thn els -> if eval b then eval thn else eval els
EqE e e' -> eval e == eval e'

Here we have defined a data structure that represents the abstract syntax tree (AST) of a very simple arithmetic language. Notice that it ensures terms are well-typed. For instance something like the following just doesn’t type check.

LitInt 1 `Add` LitBool True -- this expression does not type check

I have also provided a function eval that evaluates terms in this language.

In Scala it is quite possible to define data structures which have the same properties as a GADT declaration in Haskell. You can do this with case classes as follows.

abstract class Exp[A] 

case class LitInt(i: Int) extends Exp[Int]
case class LitBool(b: Boolean) extends Exp[Boolean]
case class Add(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int]
case class Mul(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int]
case class Cond[A](b: Exp[Boolean], thn: Exp[A], els: Exp[A]) extends Exp[A]
case class Eq[A](e1: Exp[A], e2: Exp[A]) extends Exp[Boolean]

But how do we implement eval. You might think that the following code would work. I mean, it looks like the Haskell version, right?

abstract class Exp[A] {
def eval = this match {
case LitInt(i) => i
case LitBool(b) => b
case Add(e1, e2) => e1.eval + e2.eval
case Mul(e1, e2) => e1.eval * e2.eval
case Cond(b,thn,els) => if ( b.eval ) { thn.eval } else { els.eval }
case Eq(e1,e2) => e1.eval == e2.eval
}

}

case class LitInt(i: Int) extends Exp[Int]
case class LitBool(b: Boolean) extends Exp[Boolean]
case class Add(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int]
case class Mul(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int]
case class Cond[A](b: Exp[Boolean], thn: Exp[A], els: Exp[A]) extends Exp[A]
case class Eq[A](e1: Exp[A], e2: Exp[A]) extends Exp[Boolean]

Unfortunately for us, this doesn’t work. The Scala compiler is unable to instantiate the type Exp[A] to more specific ones (such as LitInt which extends Exp[Int])

3: constructor cannot be instantiated to expected type;
  found   : FailedExp.LitInt
  required: FailedExp.Exp[A]
    case LitInt(i)       => i
        ^

There are two solutions to this problem.

Solution 1: The object-oriented way

You must write eval the object-oriented way. The definition of eval gets spread over each of the sub-classes of Exp[A].

abstract class Exp[A] {
def eval: A
}

case class LitInt(i: Int) extends Exp[Int] {
def eval = i
}

case class LitBool(b: Boolean) extends Exp[Boolean] {
def eval = b
}

case class Add(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int] {
def eval = e1.eval + e2.eval
}
case class Mul(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int] {
def eval = e1.eval * e2.eval
}
case class Cond[A](b: Exp[Boolean], thn: Exp[A], els: Exp[A]) extends Exp[A] {
def eval = if ( b.eval ) { thn.eval } else { els.eval }
}
case class Eq[A](e1: Exp[A], e2: Exp[A]) extends Exp[Boolean] {
def eval = e1.eval == e2.eval
}

Solution 2: The functional Haskell-like way

Personally I don’t like the OO style as much as the Haskell-like style. However, it turns out that you can program in that style by using a companion object.

object Exp {
def evalAny[A](e: Exp[A]): A = e match {
case LitInt(i) => i
case LitBool(b) => b
case Add(e1, e2) => e1.eval + e2.eval
case Mul(e1, e2) => e1.eval * e2.eval
case Cond(b, thn, els) => if (b.eval) { thn.eval } else { els.eval }
case Eq(e1, e2) => e1.eval == e2.eval
}
}

abstract class Exp[A] {
def eval: A = Exp.evalAny(this)
}

case class LitInt(i: Int) extends Exp[Int]
case class LitBool(b: Boolean) extends Exp[Boolean]
case class Add(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int]
case class Mul(e1: Exp[Int], e2: Exp[Int]) extends Exp[Int]
case class Cond[A](b: Exp[Boolean], thn: Exp[A], els: Exp[A]) extends Exp[A]
case class Eq[A](e1: Exp[A], e2: Exp[A]) extends Exp[Boolean]

Ah, much better. But why does this work when the previous style doesn’t? The problem is that the constructors are not polymorphic. In Haskell-speak the type is:

LitInt :: Int -> Exp Int

not

LitInt :: Int -> Exp a

The second solution is subtly different. Method evalAny is polymorphic but its type is instantiated to that of the value of whatever it is called on. For instance evalAny when called on LitInt(42) equates type variable A with Int. It can then correctly deduce that it does indeed take a value of Exp[Int] and produce a value of Int.

November 22, 2011 12:00 AM

November 21, 2011

Shin-Cheng Mu

Proving the Church-Rosser Theorem Using a Locally Nameless Representation

Around 2 years ago, for an earlier project of mine (which has not seen its light yet!) in which I had to build a language with variables and prove its properties, I surveyed a number of ways to handle binders. For some background, people have noticed that, when proving properties about a language with bound variables, one often ends up spending most of the effort proving “uninteresting” things about names, substitution, weakening, etc. For formal theorem proving to be practical, the effort has to be reduced. A number of approaches were proposed. Among them, the “locally nameless” style appeared to be interesting and promising.

With the only reference I used, Engineering Formal Metatheory by Aydemir et al., which outlines the principle ideas of the approach, I imagined how it works and tried to implement my own version in Agda. My first few implementations, however, all ended up in a mess. It appeared that there was an endless number of properties to prove. Besides the complexity of the language I was implementing, there must be something I got wrong about the locally nameless representation. Realising that I could not finish the project this way, I eventually decided to learn from the basics.

I started with the tutorial by Chargueraud, with complete code in Coq. I would then follow his footprints using Agda. The task is a classical one: define untyped λ-calculus and its reduction rules, and prove the Church-Rosser theorem.

Term

From an abstract level, there is nothing too surprising. We define a syntax for untyped λ-calculus that distinguishes between free and bound variables:

data Term : Set where
  bv : (i : BName) → Term
  fv : (x : FName) → Term
  ƛ : (e : Term) → Term
  _·_ : (e₁ : Term) → (e₂ : Term) → Term

where BName = ℕ represents bound variables by de Bruin indexes, while FName is the type of free variables. The latter can be any type that supports equality check and a method that generates a new variable not in a given set (in fact, a List) of variables. If one takes FName = String, the expression λ x → x y, where y occurs free, is represented by ƛ (bv 0 · fv "y"). For ease of implementation, one may takeFName = ℕ as well.

Not all terms you can build are valid. For example, ƛ (bv 1 · fv "y") is not a valid term since there is only one ƛ binder. How to distinguish the valid terms from invalid ones? I would (and did) switch to a dependent datatype Term n, indexed by the number of enclosing binders, and let BName = Fix n. The index is passed top-down and is incremented each time we encounter a ƛ. Closed terms are then represented by the type Term 0.

The representation above, if works, has the advantage that a term that can be build at all must be valid. Choosing such a representation was perhaps the first thing I did wrong, however. Chargueraud mentioned a similar predicate that also passes the “level” information top-down, and claimed that the predicate, on which we will have to perform induction on to prove property about terms, does not fit the usual pattern of induction. This was probably why I had so much trouble proving properties about terms.

The way to go, instead, is to use a predicate that assembles the information bottom up. The predicate LC (“locally-closed” — a term is valid if it is locally closed) is defined by:

data LC : Term → Set where
  fv : ∀ x → LC (fv x)
  ƛ : (L : FNames) → ∀ {e} →
       (fe : ∀ {x} → (x∉L : x ∉ L) → LC ([ 0 ↦ fv x ]  e)) → LC (ƛ e)
  _·_ : ∀ {e₁ e₂} → LC e₁ → LC e₂ → LC (e₁ · e₂)

A free variable alone is a valid term. A application f · e is valid if both f and e are. And an abstraction ƛ e is valid if e becomes a valid term after we substitute any free variable x for the first (0-th) bound variable. There can be an additional constraint on x, that it is not in L, a finite set of “protected” variables — such co-finite quantification is one of the features of the locally nameless style.

The “open” operator [ n ↦ t ] e substitutes the term t for the n-th bound variable in e. It is defined by

[_↦_] : ℕ → Term → Term → Term
[ n ↦ t ] (bv i) with n ≟ i
...        | yes _ = t
...        | no _ = bv i
[ n ↦ t ] (fv y) = fv y
[ n ↦ t ] (ƛ e) = ƛ ([ suc n ↦ t ] e)
[ n ↦ t ] (e₁ · e₂) = [ n ↦ t ] e₁ · [ n ↦ t ] e₂

note how n is incremented each time we go into a ƛ. A dual operator,

[_↤_] : ℕ → FName → Term → Term

instantiates the n-th bound variable to a term.

β Reduction and Church-Rosser Theorem

Small-step β reduction can be defined by:

data _β→_ : Term → Term → Set where
  β-red : ∀ {t₁ t₂} → Body t₁ → LC t₂
          → ((ƛ t₁) · t₂) β→ [ 0 ↦ t₂ ] t₁
  β-app₁ : ∀ {t₁ t₁' t₂} → LC t₂
           → t₁ β→ t₁'
           → (t₁ · t₂) β→ (t₁' · t₂)
  β-app₂ : ∀ {t₁ t₂ t₂'} → LC t₁
           → t₂ β→ t₂'
           → (t₁ · t₂) β→ (t₁ · t₂')
  β-ƛ : ∀ L {t₁ t₁'}
           → (∀ x → x ∉ L → ([ 0 ↦ fv x ] t₁) β→ ([ 0 ↦  fv x ] t₁'))
           → ƛ t₁ β→ ƛ t₁'

where β-red reduces a redux, β-app₁ and β-app₂ allows reduction respectively on the right and left hand sides of an application, and β-ƛ goes into a ƛ abstraction — again we use co-finite quantification.

Given _β→_ we can define its reflexive, transitive closure _β→*_, and the reflexive, transitive, symmetric closure _β≣_. The aim is to prove that _β→*_ is confluent:

β*-confluent :
  ∀ {m s t} → (m β→* s) → (m β→* t)
                  → ∃ (λ u → (s β→* u) × (t β→* u))

which leads to the Church-Rosser property:

β-Church-Russer : ∀ {t₁ t₂} → (t₁ β≣ t₂)
                    → ∃ (λ t → (t₁ β→* t) × (t₂ β→* t))

At an abstract level, the proof follows the classical route: it turns out that it is easier to prove the confluence of a “parallel reduction” relation _⇉_ which allows β reduction to happen in several places of a term in one step. We then prove that _β→*_ is equivalent to _⇉*_, thereby proving the confluence of _β→*_ as well. All these can be carried out relatively nice and clean.

The gory details, however, hides in proving the infrastructutal properties supporting the abstract view of the proofs.

Infrastructures

Confluence, Church-Rosser… these are the interesting stuffs we want to prove. However, we often end up spending most of the time proving those infrastructure properties we have to prove — which is why there have been so much recent research hoping to find better representations that simplify them. The locally nameless style is supposed to be such a representation. (Another orthogonal topic is to seek generic representations such that the proofs can be done once for all languages.)

In my code, most of these properties are piled in the file Infrastructure.agda. They range from stuffs you might expect to have:

open-term : ∀ k t {e} → LC e → e ≡ [ k ↦ t ] e
close-var-open-aux : ∀ k x e → LC e → e ≡ [ k ↦ fv x ] ([ k ↤ x ] e)

to stuffs not that obvious:

open-var-inj : ∀ k x t u → x ∉ fvars t → x ∉ fvars u
               → [ k ↦ fv x ] t ≡ [ k ↦ fv x ] u → t ≡ u
open-term-aux : ∀ j t i u e → ¬ (i ≡ j)
                → [ j ↦ t ] e ≡ [ i ↦ u ] ([ j ↦ t ] e)
                → e ≡ [ i ↦ u ] e

The lemma open-var-inj is one of the early lemmas that appeared in Chargueraud’s tutorial, which might give one the impression that it is an easy first lemma to prove. On the contrary, it is among the tedious ones — I needed a 40-line proof (most of the cases were simply eliminated by contraction, though).

It takes experience and intuition to know what lemmas are needed. Without promise that it will work, I would think something must have gone wrong when I found myself having to prove weird looking lemmas like:

close-var-rec-open :
  ∀ x y z t i j
  → ¬(i ≡ j) → ¬(y ≡ x) → y ∉ fvars t
  → [ i ↦ fv y ] ([ j ↦ fv z ] ([ j ↤ x ] t))
    ≡ [ j ↦ fv z ] ([ j ↤ x ] ([ i ↦ fv y ] t))

which is not easy to prove either.

The Bottom Line?

So, is the locally nameless representation what it claims to be — a way to represent binders that simplifies the infrastructural proofs and is easier to scale up? When I was struggling with some of the proofs in Infrastructure.agda I did wonder whether the claim is true only for Coq, with cleverly designed proof tactics, but not for Agda, where everything is done by hand (so far). Once the infrastructural proofs are done, however, the rest was carried out very pleasantly.

To make a fair comparison, I should re-implement everything again using de Bruin notation. That has to wait till some other time, though. (Any one want to give it a try?)

It could be the case that, while some proofs are easily dismissed in Coq using tactics, in Agda the programmer should develop some more abstractions. I did feel myself repeating some proof patterns, and found one or two lemmas that do not present in Chargueraud’s tutorial which, if used in Agda, simplifies the proofs a bit. There could be more, but at this moment I am perhaps too involved in the details to see the patterns from a higher viewpoint.

The exercise does pay off, though. Now I feel I am much more familiar with this style, and am perhaps more prepared to use it in my own project.

Code

A zip file containing all the code.

References

  1. Brian Aydemir, Arthur Chargueraud, Benjamin Pierce, Randy Pollack, and Stephanie Weirich. Engineering Formal Metatheory. POPL ’08.
  2. Arthur Chargueraud. The Locally Nameless Representation. Journal of Automated Reasoning, May 2011

by Shin at November 21, 2011 11:53 AM

Manuel M T Chakravarty

Seminar by John Hughes & Simon Peyton Jones @ UNSW

As part of YOW! 2011 (the Australian Software Developer Conference), Simon Peyton Jones and John Hughes will be in Sydney on the 7th and 8th of December 2011. On the evening of the 7th, they will appear at the YOW! Night Sydney.

On Thursday (8 December), John and Simon will give two presentations at the School of Computer Science and Engineering (CSE) of the University of New South Wales. The details are as follows.

Time: 8 December 2011, 10AM

Location: CSE Seminar room (K17_113), Level 1, CSE Building (K17)

 

Talk #1

Title: Accelerating race condition detection through procrastination

Speaker: John Hughes (Chalmers University & Quviq AB)

Abstract

Race conditions are notoriously frustrating to find, and good tools can help. The main difficulty is reliably provoking the race condition. In previous work we presented a randomising scheduler for Erlang that helps with this task.

In a language without pervasive shared mutable state, such as Erlang, performing scheduling decisions at random uncovers race conditions surprisingly well. However, it is not always enough. We describe a technique, procrastination, that aims to provoke race conditions more often than by random scheduling alone. It works by running the program and looking for pairs of events that might interfere, such as two message sends to the same process. Having found such a pair of events, we re-run the program but try to provoke a race condition by reversing the order of the two events.

We apply our technique to a piece of industrial Erlang code. Compared to random scheduling alone, procrastination allows us to find minimal failing test cases more reliably and more quickly.

John Hughes

John Hughes has worked with functional programming since 1980, was one of the designers of Haskell, and has been heavily involved with Erlang in recent years. In 2000 he and Koen Claessen published the first version of QuickCheck, a software testing tool which recently won the ACM SIGPLAN award for Most Influential Paper from ICFP in that year. He has focussed more and more on testing since then, co-founding Quviq AB in 2006 to market a commercial version of QuickCheck. He is currently both a Professor at Chalmers University, Sweden, and CEO of Quviq AB.

Talk #2

Title: Termination Combinators Forever

Speaker: Simon Peyton Jones (Microsoft Research)

Abstract

Nobody wants their compiler, or theorem prover, to go into an infinite loop, but making sure that never happens usually means applying some over-conservative heuristic like “never unfold a recursive function”. Approaches like that don’t work at all when you are doing partial evaluation or supercompilation, which fundamentally depend on inlining recursive functions. 

What we need is an online termination test. That is easier said than done; it is easy to make a mistake. In this talk I’ll describe a new combinator library that lets you build complex termination tests by combining simpler ones, while guaranteeing that that the result really is a termination test. The library elegantly encapsulates some clever mathematical ideas on well-quasi orders the cunning details are hidden from the customer.

I’ll use Haskell as the language of exposition, but you don’t need to be a Haskell guru to understand it, and the library would work equally well in other languages.

Simon Peyton Jones

Simon Peyton Jones, MA, MBCS, CEng, graduated from Trinity College Cambridge in 1980. After two years in industry, he spent seven years as a lecturer at University College London, and nine years as a professor at Glasgow University, before moving to Microsoft Research (Cambridge) in 1998.

His main research interest is in functional programming languages, their implementation, and their application. He has led a succession of research projects focused around the design and implementation of production-quality functional-language systems for both uniprocessors and parallel machines. He was a key contributor to the design of the now-standard functional language Haskell, and is the lead designer of the widely-used Glasgow Haskell Compiler (GHC). He has written two textbooks about the implementation of functional languages.

More generally, he is interested in language design, rich type systems, software component architectures, compiler technology, code generation, runtime systems, virtual machines, and garbage collection. He is particularly motivated by direct use of principled theory to practical language design and implementation — that’s one reason he loves functional programming so much.

Posted via email from PLS @ UNSW | Comment »

November 21, 2011 03:51 AM

Programming Languages and Systems research group at the University of New South Wales

Seminar by John Hughes & Simon Peyton Jones @ UNSW

As part of YOW! 2011 (the Australian Software Developer Conference), Simon Peyton Jones and John Hughes will be in Sydney on the 7th and 8th of December 2011. On the evening of the 7th, they will appear at the YOW! Night Sydney.

On Thursday (8 December), John and Simon will give two presentations at the School of Computer Science and Engineering (CSE) of the University of New South Wales. The details are as follows.

Time: 8 December 2011, 10AM

Location: CSE Seminar room (K17_113), Level 1, CSE Building (K17)

 

 

Talk #1

Title: Accelerating race condition detection through procrastination

Speaker: John Hughes (Chalmers University & Quviq AB)

Abstract

Race conditions are notoriously frustrating to find, and good tools can help. The main difficulty is reliably provoking the race condition. In previous work we presented a randomising scheduler for Erlang that helps with this task.

In a language without pervasive shared mutable state, such as Erlang, performing scheduling decisions at random uncovers race conditions surprisingly well. However, it is not always enough. We describe a technique, procrastination, that aims to provoke race conditions more often than by random scheduling alone. It works by running the program and looking for pairs of events that might interfere, such as two message sends to the same process. Having found such a pair of events, we re-run the program but try to provoke a race condition by reversing the order of the two events.

We apply our technique to a piece of industrial Erlang code. Compared to random scheduling alone, procrastination allows us to find minimal failing test cases more reliably and more quickly.

John Hughes

John Hughes has worked with functional programming since 1980, was one of the designers of Haskell, and has been heavily involved with Erlang in recent years. In 2000 he and Koen Claessen published the first version of QuickCheck, a software testing tool which recently won the ACM SIGPLAN award for Most Influential Paper from ICFP in that year. He has focussed more and more on testing since then, co-founding Quviq AB in 2006 to market a commercial version of QuickCheck. He is currently both a Professor at Chalmers University, Sweden, and CEO of Quviq AB.

 

Talk #2

Title: Termination Combinators Forever

Speaker: Simon Peyton Jones (Microsoft Research)

Abstract

Nobody wants their compiler, or theorem prover, to go into an infinite loop, but making sure that never happens usually means applying some over-conservative heuristic like “never unfold a recursive function”. Approaches like that don’t work at all when you are doing partial evaluation or supercompilation, which fundamentally depend on inlining recursive functions. 

What we need is an online termination test. That is easier said than done; it is easy to make a mistake. In this talk I’ll describe a new combinator library that lets you build complex termination tests by combining simpler ones, while guaranteeing that that the result really is a termination test. The library elegantly encapsulates some clever mathematical ideas on well-quasi orders the cunning details are hidden from the customer.

I’ll use Haskell as the language of exposition, but you don’t need to be a Haskell guru to understand it, and the library would work equally well in other languages.

Simon Peyton Jones

Simon Peyton Jones, MA, MBCS, CEng, graduated from Trinity College Cambridge in 1980. After two years in industry, he spent seven years as a lecturer at University College London, and nine years as a professor at Glasgow University, before moving to Microsoft Research (Cambridge) in 1998.

His main research interest is in functional programming languages, their implementation, and their application. He has led a succession of research projects focused around the design and implementation of production-quality functional-language systems for both uniprocessors and parallel machines. He was a key contributor to the design of the now-standard functional language Haskell, and is the lead designer of the widely-used Glasgow Haskell Compiler (GHC). He has written two textbooks about the implementation of functional languages.

More generally, he is interested in language design, rich type systems, software component architectures, compiler technology, code generation, runtime systems, virtual machines, and garbage collection. He is particularly motivated by direct use of principled theory to practical language design and implementation — that's one reason he loves functional programming so much.

 

Permalink | Leave a comment  »

November 21, 2011 03:27 AM

November 20, 2011

Alejandro Serrano Mena

Moving to Persistent

I haven't said much in a while, but of course, things are moving in the EclipseFP world. JPMoresmau announced the 2.20 release of EclipseFP, which in terms of interface gives you a way to manage your local installation of Cabal packages and a better outline of Cabal files. Internally, a lot has changed, as EclipseFP moved from using Scion as backend to BuildWrapper. This big change will hopefully solve most of the memory problems that EclipseFP was experiencing.

Also, I would like to announce that scion-browser is moving from a in-memory database to using SQLite to back up the data. In my tests performance is comparable to the old one (although for big tasks as browsing the entire list of local modules is a bit slower), but doesn't need a big pile of memory to hold the entire database, which I think it's a big win. If you're curious, you can browse the code, but be warned it doesn't work with the current version of EclipseFP.

The change has been possible thanks to the incredible Persistent package of Yesod fame. With that, I hadn't had to write a single line of code to create the database or to insert data. The only problem has been that Persistent doesn't have any way to encode queries with multiples joins. This lead me to write some of the queries by hand (and to refresh my ancient knowledge of SQL).

Hope you like it. As usual, every suggestion to improve EclipseFP or scion-browser is more than welcome :)

by Alejandro Serrano (noreply@blogger.com) at November 20, 2011 04:50 PM

November 19, 2011

Russell O'Connor

Bug in OpenSSL?

OpenSSL uses DER when encoding DSA signatures. DER is a standardized encoding of structured data that is widely implemented. Integers are among the primitives that DER specifics how to encode. In particular, DER specifies that encoded integers are signed. However, when OpenSSL decodes signatures, it always decodes integers as if they are unsigned. In particular, OpenSSL will decode negative numbers as positive numbers.

OpenSSL claims this is not a problem by noting that only improperly encoded signatures will use negative numbers.

/* Custom primitive type for BIGNUM handling. This reads in an ASN1_INTEGER as a
 * BIGNUM directly. Currently it ignores the sign which isn't a problem since all
 * BIGNUMs used are non negative and anything that looks negative is normally due
 * to an encoding error.
 */

openssl-1.0.0e/crypto/asn1/x_bignum.c

I am skeptical of their claim that this is not a problem. This means a user, with access to a properly encoded signature, can create an improper DER encoding of a signature using negative numbers that is still a proper DER encoding of a pair of integers and that OpenSSL will validate as a proper signature. This could wreak havoc on any system with two verification engines; one using OpenSSL and another using different software that, correctly, rejects such a signature for using a negative number.

Unfortunately people, even in security, are reactive instead of proactive; I presume that the OpenSSL people will not change anything unless there is an exploit. Your task, should you chose to accept it, is to use this knowledge to find an exploit that uses negative numbers in DSA signatures to exploit or create a denial of service on some system using OpenSSL. Oh, and then tell the OpenSSL people about it.

Good luck.

November 19, 2011 09:15 PM

November 18, 2011

Thomas M. DuBuisson

Clean and reinstall those GHC packages!

For those of you who are as lazy as me about fixing and reinstalling broken GHC packages, I’ve ripped off the well-known and loved ghc-pkg-clean script and made it recursive.  No promises of termination:
 
function ghc-pkg-supercleaner() {
    b="0"
    for p in `ghc-pkg check $* 2>&1  | grep problems | awk '{print $6}' | sed -e 's/:$//'`
    do
        echo unregistering $p; ghc-pkg $* unregister $p;
        cabal install $p; b="1"
    done
    if [ $b == "1" ];
      then ghc-pkg-supercleaner
    fi
}

by tommd at November 18, 2011 03:15 AM

November 15, 2011

Bryan O'Sullivan

The future of MailRank’s open source technologies

(Cross-posted from the MailRank engineering blog.)

You may have seen my exciting news about our upcoming move to Facebook.

It’s been a total blast working on our product, and of course as we did so we released a number of open source libraries and tools. It only added to our pleasure to see so much of that code used outside of our own domain. I will continue to develop and maintain the code that we have released.

Here is a quick rundown of the code we have released, roughly ordered by significance. Yep, we wrote all of these projects in Haskell, definitely a decision that in retrospect I’m very happy about.

  • pronk (not yet actually released) is an application for load testing web servers. Think of it as similar to httperf or ab, only more modern, simpler to deal with, and with vastly better analytic and reporting capabilities.

  • configurator is a library that allows fast, dynamic reconfiguration of a Haskell application or daemon.

  • aeson is a JSON encoding and decoding library optimized for high performance and ease of use.

  • text-format is a library for printf-like text formatting.

  • mysql-simple is an easy-to-use client library for the MySQL database. It is several times faster than its competitors, and easier to use. It is built on top of the low-level mysql library.

  • riak-haskell-client is a client for the Riak decentralized data store.

  • blaze-textual is a library for efficiently rendering Haskell data as text.

  • double-conversion is a very fast library for rendering double precision floating point numbers as text, based on the code from the V8 Javascript engine.

  • resource-pool is a fast resource pooling library.

  • snappy provides Haskell bindings to Google’s extremely fast snappy compression library.

  • base16-bytestring provides fast handling of base16-encoded data.

  • hdbc-mysql provides a MySQL transport for the HDBC database access library. (Yes, we recommend using mysql-simple instead!)

Thanks to all of you who have contributed patches and bug reports. It’s going to be an exciting future!

by Bryan O'Sullivan at November 15, 2011 10:23 PM

Bryn Keller

BookSleeve

I’ve been working with BookSleeve lately. It’s a .Net API for interacting with Redis, a fast key-value store / distributed cache / data structure server. Redis is very nice and very easy to work with, and the API that BookSleeve provides is both as simple as it could possibly be, and deeply performant – all the methods return their results as Tasks, so the API is as asynchronous as you want it to be – you only have to Task.Wait for the results you need. Also, Marc Gravell provided immediate and very helpful support when I ran into an issue (mostly of my own making).

One thing to keep in mind, though – the connections BookSleeve creates are multiplexers, so for a web app, you may as well just open one connection and leave it open for the life of the app. I initially made the mistake of opening and closing a connection for each request, but Marc explained that’s useless overhead for this scenario.

If you need to work with Redis and you’re in .Net, give BookSleeve a try!

by xoltar at November 15, 2011 05:55 PM

Bill Atkins

I should also say, there's really nothing wrong in just coding the
imperative algorithm in Scala. It's not a sacrilege to use imperative code,
in particular if it's inside a function. Sometimes it's the clearest way to
express things. Many other times it is not.

- Martin Oderksy

by More Indirection (noreply@blogger.com) at November 15, 2011 01:58 PM

November 12, 2011

Epilogue for Epigram

Hier Soir, an OTT Hierarchy

I’m still waiting for my computer to typecheck this file, which I concocted yesterday evening. I’m told by someone with a better computer (thanks, Darin) that it does typecheck, and moreover, satisfy the termination checker. It relies unexcitingly on the size-change principle and is easily rendered entirely structural. What I’ve managed to do (at last!) is [...]

by Conor at November 12, 2011 11:18 PM

David Amos

New release of HaskellForMaths


I've just uploaded a new version v0.4.1 of HaskellForMaths, containing three new modules and a couple of other improvements. The additions are as follows:

Math.Algebras.Quaternions

This module was already present: it defines the quaternion algebra on the basis {1,i,j,k}, where multiplication is defined by:
i^2 = j^2 = k^2 = ijk = -1

This is enough information to figure out the full multiplication table. For example:
ijk = -1
=> (ijk)k = -k
=> ij(kk) = -k (associativity of multiplication)
=> ij = k
It turns out that the basis elements i,j,k anti-commute in pairs, eg ij = -ji, etc.

In this release I've added a couple of new things.

First, the quaternions are a division algebra, so I've added a Fractional instance.

Specifically, we can define a conjugation operation on the quaternions (similar to complex conjugation) via
conj (w+xi+yj+zk) = w-xi-yj-zk
Then we can define a quadratic norm via
sqnorm q = q * conj q = w^2+x^2+y^2+z^2
Since the norm is always a scalar, we can define a multiplicative inverse by
q^-1 = conj q / sqnorm q

For example:

$ cabal install HaskellForMaths
$ ghci
> :m Math.Algebras.Quaternions
> (2*i+3*j)^-1 :: Quaternion Q
-2/13i-3/13j

(If you leave out the type annotation, you'll be working in Quaternion Double.)

Second, the quaternions have an interesting role in 3- and 4-dimensional geometry.

Given any non-zero quaternion q, the map x -> q^-1 x q turns out to be a rotation of the 3-dimensional space spanned by {i,j,k}. To multiply rotations together (ie do one then another), just multiply the quaternions. This turns out to be a better way to represent rotations than 3*3 matrices:
- It's more compact - four scalars rather than nine
- They're faster to multiply - 16 scalar multiplications versus 27
- It's more robust against rounding error - whatever quaternion you end up with will still represent a rotation, whereas a sequence of matrix multiplications of rotations might not be quite a rotation any more, due to rounding error.

If you're curious, the function reprSO3 converts a quaternion to the corresponding 3*3 matrix:

> reprSO3 (1+2*i) :: [[Q]]
[[1,0,0],[0,-3/5,-4/5],[0,4/5,-3/5]]

(Exercise: Figure out why we got this matrix.)

Quaternions can also be used to represent rotations of 4-dimensional space - see the documentation.


Math.Algebras.Octonions

This is a new module, providing an implementation of the 8-dimensional non-associative division algebra of octonions. I follow Conway's notation [1], so the octonions have basis {1,e0,e1,e2,e3,e4,e5,e6}, with multiplication defined by:
ei * ei = -1, for i in [0..6]
ei+1 * ei+2 = ei+4, where the indices are taken modulo 7.

The octonions are not associative, but they are an inverse loop, so they satisfy x-1(xy) = y = (yx)x-1. This is enough to enable us to deduce the full multiplication table from the relations above.

Like the quaternions, the octonions have conjugation and a norm, and multiplicative inverses:

> :l Math.Algebras.Octonions
> (2+i0+2*i3)^-1
2/9-1/9i0-2/9i3

The octonions are an exceptional object in mathematics: there's nothing else quite like them. They can be used to construct various other exceptional objects, such as the root lattice E8, or the Lie group G2. Hopefully I'll be able to cover some of that stuff in a future installment.

[1] Conway and Smith, On Quaternions and Octonions


Math.NumberTheory.Prime

The main function in this module is isPrime :: Integer -> Bool, which tells you whether a number is prime or not. It's implemented using the Miller-Rabin test.

The basic idea of the test is:
- If p is prime, then Zp is a field
- In a field, the equation x^2 = 1 has only two solutions, 1 and -1
- Given an arbitrary b coprime to p, we know from Fermat's little theorem that b^(p-1) = 1 (mod p)
- So if p-1 = q * 2^s, with q odd, then either b^q = 1 (mod p), or there is some r, 0 <= r < s with b^(q*2^r) = -1 (mod p)

The idea of the algorithm is to try to show that p isn't prime by trying to find a b where the above is not true. We take several different values of b at random, and repeatedly square b^q, to see whether we get -1 or not.

The advantage of the Miller-Rabin test, as compared to trial division say, is that it has a fast running time even for very large numbers. For example:

> :m Math.NumberTheory.Prime
> :set +s
> nextPrime $ 10^100
10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000267
(0.09 secs, 14904632 bytes)

The potential disadvantage of the Miller-Rabin test is that it is probabilistic: There is a very small chance (1 in 10^15 in this implementation) that it could just fail to hit on a b which disproves n's primeness, so that it would say n is prime when it isn't. In practice, at those odds it's not worth worrying about.


Math.NumberTheory.Factor

The main function in this module is pfactors :: Integer -> [Integer], which returns the prime factors of a number (with multiplicity). It uses trial division to try to find prime factors less than 10000. After that, it uses the elliptic curve method to try to split what remains. The elliptic curve method relies on some quite advanced maths, but the basic idea is this:
- If p is a prime, then Zp is a field
- Given a field, we can do "arithmetic on elliptic curves" over the field.
- So to factor n, pretend that n is prime, try doing arithmetic on elliptic curves, and wait till something goes wrong.
- It turns out that if we look at what went wrong, we can figure out a non-trivial factor of n

Here it is in action:

> :m Math.NumberTheory.Factor
> pfactors $ 10^30+5
[3,5,4723,1399606163,10085210079364883]
(0.55 secs, 210033624 bytes)
> pfactors $ 10^30+6
[2,7,3919,758405810021,24032284101871]
(2.31 secs, 883504748 bytes)

I love the way it can crunch through 12-digit prime factors with relative ease.

by DavidA (noreply@blogger.com) at November 12, 2011 09:36 PM

Mikael Vejdemo Johansson (Syzygy-)

A quick python hack for a mathematical puzzle

So, today I saw this in my Twitter feed:

«Phil Harvey wants us to partition {1,…,16} into two sets of equal size so each subset has the same sum, sum of squares and sum of cubes.» — posted by @MathsJam and retweeted @haggismaths.

Sounds implausible was my first though. My second thought was that there aren’t actually ALL that many of those: we can actually test this.

So, here’s a short collection of python lines to _do_ that testing.

import itertools
allIdx = range(1,17)
sets = itertools.combinations(allIdx,8)
setpairs = [(list(s), [i for i in allIdx if i not in s]) for s in sets]
def f((s1,s2)): return (sum(s1)-sum(s2), sum(map(lambda n: n**2, s1))-sum(map(lambda n: n**2, s2)), sum(map(lambda n: n**3, s1))-sum(map(lambda n: n**3, s2)))

goodsets = [ss for ss in setpairs if f(ss) == (0,0,0)]

And back comes one single response (actually, two, because the comparison is symmetric).

[([1, 4, 6, 7, 10, 11, 13, 16], [2, 3, 5, 8, 9, 12, 14, 15]),
([2, 3, 5, 8, 9, 12, 14, 15], [1, 4, 6, 7, 10, 11, 13, 16])]

by Michi at November 12, 2011 03:09 PM

November 09, 2011

Philippa Cowderoy

Really Only Relative

In Introduction to Constraint Programming I made something of a mistake. It was a remarkably easy one to make, because I stepped outside the bounds of the type system’s help – and especially that of those who’ve declared interfaces before me.

Finding and Fixing the Mistake

The centrepiece of the article is an embedded language, which I’d initially written as just an applicative functor and quickly decided needed more power so I added a bind-like construct as well. The functor at its heart only operates on a subcategory of that of Haskell values – defined by a typeclass called Embeddable:

class Embeddable a where
  subst :: Assignment -> a -> a

instance Embeddable () where
  subst _ v = v

instance Embeddable Term where
  subst = substTerm

instance (Embeddable a, Embeddable b) => Embeddable (a -> b) where
  subst = const id

So we’ve got two ground types in the category – () and Term – and arbitrary-order functions on them. What we haven’t got is the target of the functor, Embeddable a => Code a – which means that while the Bind constructor types fine and so does its implementations, we can’t write the join function that provides the difference between an applicative functor and a monad in Haskell. We can pass around higher-order Haskell functions, but our embedded language is strictly first-order!

Oops? We can fix this by adding a new instance:

instance Embeddable a => Embeddable (Code a) where
  subst _ c = c 

What made this mistake easy to make, of course, was not having a Monad-like typeclass with a defaulted join I was writing an instance for. That said, I did discover something new to me: I can have a "first-order-only monad" with the freedom of binding structure that comes with it. If I’m willing to invoke the relevant GHC flags, it can even play along with do notation.

What Did We Have Before?

This new structure is interesting – >>= still doesn’t admit self-analysis, but if I just want to know that there’s no passing around of computations happening in an EDSL then it’s a lot more pleasant to use than an Arrow. The first piece of good news is that there’s already somewhere for it to fit in on HackageDB – the RMonad package. Of course, this implies that not every RMonad is a monad.

For something with more theory developed, I believe it’s a Haskell encoding of a Relative Monad. A typeclass for them in Haskell using ConstraintKinds can be found in this draft paper: Subcategories in Haskell.


by Philippa Cowderoy at November 09, 2011 08:28 PM

Object and Meta, Typefully

In my previous post on constraint programming, I took the following term type as an object language and started doing meta-level work with it:

data Term = Term !Label ![Term]

I handled the meta level in a manner that isn’t particularly typeful:

data Term = Term !Label ![Term] | Metavar Integer

That’s okay for a quick and dirty solver, but there are better options if you want a bit more safety!

quick note: I leave the !s out in all these representations because the one in ![Term] doesn’t actually do that much good (it only achieves head-strictness) so I figured it’s easier to stick to the fast and loose approach

Just use two types

data ObjectTerm = ObjectTerm Label [ObjectTerm]
data MetaTerm = MetaTerm Label [MetaTerm] | Metavar Integer

liftToMeta :: ObjectTerm -> MetaTerm
liftToMeta (ObjectTerm l cs) = MetaTerm l (map liftToMeta cs)

concretise :: MetaTerm -> Maybe ObjectTerm
concretise (MetaTerm l cs) = return . ObjectTerm l =<< mapM concretise cs
concretise Metavar{} = Nothing

This one’s obvious enough. Given only Haskell 2010 type classes to abstract across the equivalents of the Term constructor this can be rather painful to work with because your polymorphic constructor is no good for pattern matching. With view patterns or the upcoming pattern synonyms things become more comfortable though.

data TermView children = Term Label [children] | NotTerm

class Term t where
  term :: Label -> [t] -> t
  matchTerm :: t -> TermView t

instance Term ObjectTerm where
  term = ObjectTerm
  matchTerm (ObjectTerm l cs) = Term l cs

instance Term MetaTerm where
  term = MetaTerm
  matchTerm (MetaTerm l cs) = Term l cs
  matchTerm _ = NotTerm

add :: (Term l, Term r) => l -> r -> r
add (matchTerm -> Term "Z" []) r = r
add (matchTerm -> Term "S" [n]) r = term "S" $ [add n r]

This example only uses view patterns, but pattern synonyms would allow patterns that’re indistinguishable from ordinary ones. Note the type of add though (equivalent to the inferred one) – it can lead to unexpected annotation requirements.

GADTs

One of the great things about GADTs is that you can use phantom types to mark out what might otherwise be subtypes of the GADT – and pattern-matching to perform the equivalent of a safe cast. The bad news is that they don’t play quite so nicely with the rose tree representation: you can’t prove anything without a separate leaf constructor, which makes a bit of a mess of the original code.

data Concrete
data Meta
data Term a where
  Leaf :: Label -> Term a
  Branch :: Label -> [Term a] -> Term a
  Metavar :: Integer -> Term Meta

concretise :: Term a -> Maybe (Term Concrete)
concretise (Leaf l) = return $ Leaf l
concretise (Branch l cs) = return . Branch l =<< mapM concretise cs
concretise Metavar{} = Nothing

The concretise function attempts to prove that a Term can be Concrete via pattern-matching – Metavar is always Term Meta, which effectively taints the entire term it appears in. The concretise function still ends up rebuilding all the Branch nodes though.

An alternative type for concretise is Term a -> Maybe (Term b), and if we’re comfortable with rank-n types we can use (forall a. Term a) as an equivalent to Term Concrete that’s readily usable as Term Meta as well. This means we already get liftToMeta for free if we keep types polymorphic until we care, but failing that it’s just a tree traversal rebuilding the same term at a different type.

Quick and Dirty Optimisation

The version of concretise above rebuilds an entire term just to change its type, but GHC already has the same underlying representations for them. I don’t know a good way of carrying coercions to Concrete or a type variable around, but we do know something the compiler doesn’t – and this means we can cheat.

import Unsafe.Coerce

concretise :: Term a -> Maybe (Term Concrete)
concretise l@Leaf{} = return $ unsafeCoerce l
concretise b@(Branch l cs) = return $ unsafeCoerce b =<< mapM_ concretise cs
concretise Metavar{} = Nothing

Two level types

There’s an alternative approach using two mutually recursive datatypes, which when generalised are parametric in each other. One type handles the per-node structure of a term, while the other handles the recursive structure – and via polymorphism, any additional structure that doesn’t belong to object terms.

data Term wrapper = Term Label [wrapper]

newtype Fix a = Fix { unfix :: (a (Fix a)) }
data Meta object = O (object (Meta object)) | Metavar Integer

liftToMeta :: Fix Term -> Meta Term
liftToMeta (Fix (Term l cs)) = O $ Term l (map liftToMeta cs) 

concretise :: Meta Term -> Maybe (Fix Term)
concretise (O (Term l cs)) = return . Fix . Term l =<< mapM concretise cs
concretise Metavar{} = Nothing

Fix is a type-level fixpoint operator – so it provides recursive structure in types polymorphic in their own children, and nothing else. The O constructor in Meta has much the same structure, but Meta also handles metavariables. As Fix and Meta have completely different internal representations, liftToMeta and concretise both require rebuilding the entire term again.

This approach has the major advantage that it starts to enable a degree of generic programming – we can reuse Fix and Meta for a variety of different object node types. The classic example is Tim Sheard‘s functional pearl on generic unification, which is implemented in the unification-fd package.

It may also be appropriate to use type classes etc much as in the initial example – possibly using one class generalising Term and a second one generalising Fix and Meta, such that an application of the Fix/Meta generalisation is also a generalisation of Term. I haven’t tried it so far though, and it’s possible that the increased annotation burden isn’t worth it.

Chaining Types and the Expression Problem

Having gone as far as two level types, we can try a refactoring:

data ObjectTermNode fix = Term Label [fix]
data MetaTermNode chain fix = O (chain fix) | Metavar Integer

newtype Fix a = Fix { unfix :: (a (Fix a)) }

type ObjectTerm = Fix ObjectTermNode
type MetaTerm = Fix (MetaTermNode ObjectTermNode)

Now the node info for metaterms is also separated from the recursive structure. This generalises to as many different node types as we like, though the collection of constructors used to pass up the chain will grow – clearly this cries out for a type class to match each node type. I’m hoping that with ConstraintKinds someone’ll figure out a good way to write instances that generate the chaining behaviour with only a linear instance count too, but as I’m sticking with the Haskell Platform at present I’m not able to have a go myself yet.

This solution has the advantange of generalising to the Expression Problem: we can add new node types (with corresponding type classes) and new functions reasonably orthogonally. Similarly, other solutions to the Expression Problem are solutions to the problem discussed in this post.


by Philippa Cowderoy at November 09, 2011 08:28 PM

Edwin Brady

Type Classes in Idris

My goal for Idris is to make it a practical dependently typed programming language – suitable for general purpose, systems programming, with precise types when you need them for increasing confidence in correctness of your program. One thing that a modern, practical language really needs, however, is a sane, safe, general way of overloading functions. Haskell‘s solution to this, type classes, is by far the neatest I’ve seen, so in the new implementation of Idris, that’s what I’ve done. They work much as you’d expect if you’re used to Haskell. For example, the Show class could look a bit like this:

class Show a where {
    show : a -> String;
}

This generates a function show : Show a => a -> String, i.e. a function which will display any a under the constraint that a is an instance of Show. We can declare instances…

instance Show Nat where {
    show O = "O";
    show (S k) = "s" ++ show k;
}

…and instances can themselves have constraints (and, indeed, be instances for dependent types):

instance Show a => Show (Vect a n) where {
    show xs = "[" ++ show' xs ++ "]" where {
        show' : Show a => Vect a n -> String;
        show' VNil = "";
        show' (x :: VNil) = show x;
        show' (x :: xs) = show x ++ ", " ++ show' xs;
    }
}

As you’d expect, classes can have constraints too:

class Monad (m : Set -> Set) where {
    return : a -> m a;
    (>>=)  : m a -> (a -> m b) -> m b;
}

class Monad m => MonadPlus (m : Set -> Set) where {
    mplus : m a -> m a -> m a;
    mzero : m a;
}

Note that, in this case, we have to give an explicit kind for the parameter m. If you don’t give this, Idris will assume that the parameter is of kind Set.

The prelude and builtins modules now have a couple of the more obviously useful type classes. However, I’m not going to rush into extending this – we should probably think carefully about what classes we need and how they relate to each other, rather than do things “just because Haskell did it that way”!

How it works

Type classes classify types according to which operations can be overloaded for them. Internally, therefore, we can represent type classes as a record of functions. For example, for show:

data Show : Set -> Set where
    instanceShow : (show : a -> String) -> Show a;

show : Show a -> a -> String;
show (instanceShow s) x = s x;

An instance for a type a is then a value of type Show a; for example for Nat:

showNat : Show Nat;
showNat = instanceShow show' where {
    show' O = "O";
    show' (S k) = "s" ++ show' k;
}

All the elaborator has to do on seeing a class declaration is to generate the appropriate data type, and on seeing an instance declaration generate the appropriate instance function.

Class constraints are a special kind of implicit argument (much like Agda’s instance arguments). On seeing a function call show x, the elaborator will attempt to resolve the implicit argument first by looking in the local context, then attempting to refine by any global class instances which match, recursively.

Still to do…

Default definitions of class methods will be very useful, as will a deriving mechanism. The former is easy, and so likely to happen soon. The latter is hard, and so isn’t.

Note also that this mechanism is intended entirely for a safe general overloading mechanism, and many of the type class tricks which are possible in Haskell are unlikely to work (though, at least, Idris classes can have more than one parameter). Such tricks, I believe, ought to be unnecessary anyway since the dependent type system is very flexible. It will, however, be interesting to see what can be done that I haven’t thought of…


by edwinb at November 09, 2011 12:46 AM

November 07, 2011

Tom Moertel

A flyweight mocking helper for Python

Recently, I needed a lightweight mocking solution when adding unit tests to some existing Python code. I could have used one of the many Python mocking libraries, but I only had to replace a few module functions during testing, so I just wrote a tiny mocking helper rather than add a dependency to the project.

The helper is simple and surprisingly versatile:

import contextlib
import functools

@contextlib.contextmanager
def mocked(func,
           expecting=None, returning=None, raising=None,  # specs
           replacement_logic=None, called=1):
    """Stub out and mock a function for a yield's duration.""" 

    if (returning, raising, replacement_logic).count(None) < 2:
        raise ValueError("returning, raising, and replacement_logic " 
                         "are incompatible with each other")

    # default logic for implementing mock fn: return or raise per specs
    def default_logic(*_args, **_kwds):
        if raising:
            raise raising
        return returning

    # prepare wrapper to replace mocked function for duration of yield
    invocations = [0]
    @functools.wraps(func)
    def replacement(*args, **kwds):
        if expecting is not None:
            assert expecting == (args, kwds)  # did we get expected args?
        invocations[0] += 1
        return (replacement_logic or default_logic)(*args, **kwds)

    # replace mocked function, yield to test, and then check & clean up
    module = sys.modules.get(func.__module__)
    setattr(module, func.__name__, replacement)
    try:
        yield  # give control back to test for a while
        assert invocations[0] == called  # was mock called enough?
    finally:
        setattr(module, func.__name__, func)

def uncalled(func):
    """Require that a function not be called for a yield's duration.""" 
    return mocked(func, called=0)

The idea is to wrap test code that requires mocked external functions with a mocking helper. Here’s an example:

def test_ml_errors_must_be_reported(self):
    """When an error occurs, it must be reported; nothing must be sent.""" 
    data = self._request_data()
    exception = mls.MlError()
    exception.exc = '[[[error description]]]'
    with mocked(mls.subscriber_count, raising=exception):
        with uncalled(sms.send):
            resp = self.app.request("/mailings", method='POST', data=data)
    self.assertEqual(resp.status, '200 OK')
    self.assertIn(exception.exc, resp.data)  # must be reported

In this test method, I’m making a simulated POST request to a web service that’s supposed to check how many users are in a mailing list and then, possibly, send some SMS messages. In this case, however, I want to simulate that an error occurs when talking to the mailing-list service. The test must verify that the error is reported and, crucially, that no messages are sent.

The library modules for talking to the mailing-list service and the SMS service are called mls and sms. So, for the duration of the simulated request, I’m replacing two functions in these modules with mock versions. The mock version of mls.subscriber_count, when called, will raise the exception I’m trying to simulate. The mock version of sms.send, however, must not be called; if it is, the uncalled mocking helper will alert me by raising an exception.

So the mocking helpers not only temporarily install mock implementations of functions but also assert that those mock implementations are (or are not) called as expected. In the following code, for example, I use this capability to make sure that the mailing-list service is asked to get the subscriber count for the mailing list having the right key. I also simulate the service returning a subscriber count of 123.

def test_unconfirmed_msgs_must_be_confirmed(self):
    """An unconfirmed msg must be confirmed and not sent.""" 
    mlkey = 'TEST_ML_KEY'
    body = 'Test message!'
    data = self._request_data(mlkey=mlkey, body=body)
    with mocked(mls.subscriber_count, expecting=((mlkey,),{}), returning=123):
        with uncalled(sms.send):
            resp = self.app.request("/mailings", method='POST', data=data)
    self.assertEqual(resp.status, '200 OK')
    self.assertIn('Reply CONFIRM', resp.data)    # we must ask for confirmation
    self.assertIn('123 subscribers', resp.data)  # and supply subscriber count
    self.assertIn(body, resp.data)               # and the msg to be sent

The mocked helper lets you do a few more things, too, but you get the idea: Sometimes a short helper function can take you a long way.

by Tom Moertel at November 07, 2011 05:39 AM

Computer Systems Design Laboratory at the University of Kansas

Kansas Lava

We are please to announce the first release of Kansas Lava, as well as the first release of Kansas Lava Cores.

Kansas Lava is Haskell library for simulating hardware and generating VHDL, in the spirit of Chalmers Lava and Xilinx Lava. We make extensive use of associated type functions, and other GHC extensions.

Kansas Cores are a set of libraries that use Kansas Lava to various build hardware components like UARTs and an LCD driver, as well as a monad and simulator for the Spartan3e.

Both are now available on hackage.

Kansas Lava has various online resources. The Kansas Lava webpage is

http://ittc.ku.edu/csdl/fpg/Tools/KansasLava

A video walkthrough of our tutorial has been started, and is available on youtube.

http://www.youtube.com/playlist?list=PL211F8711E3B3DF9C

The website for the functional programming group at KU is

http://ittc.ku.edu/csdl/fpg/Home

Kansas Lava Team @ KU

by Andy Gill at November 07, 2011 03:21 AM

November 04, 2011

Edward Kmett

What Constraints Entail: Part 2

Last time we derived an entailment relation for constraints, now let's get some use out of it.

Reflecting Classes and Instances

Most of the implications we use on a day to day basis come from our class and instance declarations, but last time we only really dealt with constraint products.

For example given:

 
#if 0
class Eq a => Ord a
instance Eq a => Eq [a]
#endif
 

we could provide the following witnesses

 
ordEq :: Ord a :- Eq a
ordEq = Sub Dict
 
eqList :: Eq a :- Eq [a]
eqList = Sub Dict
 

But this would require a lot of names and become remarkably tedious.

So lets define classes to reflect the entailment provided by class definitions and instance declarations and then use them to reflect themselves.

 
class Class b h | h -> b where
  cls :: h :- b
 
infixr 9 :=>
class b :=> h | h -> b where
  ins :: b :- h
 
instance Class () (Class b a) where cls = Sub Dict
instance Class () (b :=> a) where cls = Sub Dict
 

Now we can reflect classes and instances as instances of Class and (:=>) respectively with:

 
-- class Eq a => Ord a where ...
instance Class (Eq a) (Ord a) where cls = Sub Dict
-- instance Eq a => Eq [a] where ...
instance Eq a :=> Eq [a] where ins = Sub Dict
 

That said, instances of Class and Instance should never require a context themselves, because the modules that the class and instance declarations live in can't taken one, so we can define the following instances which bootstrap the instances of (:=>) for Class and (:=>) once and for all.

 
#ifdef UNDECIDABLE
instance Class b a => () :=> Class b a where ins = Sub Dict
instance (b :=> a) => () :=> b :=> a where ins = Sub Dict
#endif
 

These two instances are both decidable, and following a recent bug fix, the current version of GHC HEAD supports them, but my local version isn't that recent, hence the #ifdef.

We can also give admissable-if-not-ever-stated instances of Class and (:=>) for () as well.

 
instance Class () () where cls = Sub Dict
instance () :=> () where ins = Sub Dict
 

Reflecting the Prelude

So now that we've written a handful of instances, lets take the plunge and just reflect the entire Prelude, and (most of) the instances for the other modules we've loaded.

 
instance Class () (Eq a) where cls = Sub Dict
instance () :=> Eq () where ins = Sub Dict
instance () :=> Eq Int where ins = Sub Dict
instance () :=> Eq Bool where ins = Sub Dict
instance () :=> Eq Integer where ins = Sub Dict
instance () :=> Eq Float where ins = Sub Dict
instance () :=> Eq Double where ins = Sub Dict
instance Eq a :=> Eq [a] where ins = Sub Dict
instance Eq a :=> Eq (Maybe a) where ins = Sub Dict
instance Eq a :=> Eq (Complex a) where ins = Sub Dict
instance Eq a :=> Eq (Ratio a) where ins = Sub Dict
instance (Eq a, Eq b) :=> Eq (a, b) where ins = Sub Dict
instance (Eq a, Eq b) :=> Eq (Either a b) where ins = Sub Dict
instance () :=> Eq (Dict a) where ins = Sub Dict
instance () :=> Eq (a :- b) where ins = Sub Dict
 
 
instance Class (Eq a) (Ord a) where cls = Sub Dict
instance () :=> Ord () where ins = Sub Dict
instance () :=> Ord Bool where ins = Sub Dict
instance () :=> Ord Int where ins = Sub Dict
instance ():=> Ord Integer where ins = Sub Dict
instance () :=> Ord Float where ins = Sub Dict
instance ():=> Ord Double where ins = Sub Dict
instance () :=> Ord Char where ins = Sub Dict
instance Ord a :=> Ord (Maybe a) where ins = Sub Dict
instance Ord a :=> Ord [a] where ins = Sub Dict
instance (Ord a, Ord b) :=> Ord (a, b) where ins = Sub Dict
instance (Ord a, Ord b) :=> Ord (Either a b) where ins = Sub Dict
instance Integral a :=> Ord (Ratio a) where ins = Sub Dict
instance () :=> Ord (Dict a) where ins = Sub Dict
instance () :=> Ord (a :- b) where ins = Sub Dict
 
 
instance Class () (Show a) where cls = Sub Dict
instance () :=> Show () where ins = Sub Dict
instance () :=> Show Bool where ins = Sub Dict
instance () :=> Show Ordering where ins = Sub Dict
instance () :=> Show Char where ins = Sub Dict
instance Show a :=> Show (Complex a) where ins = Sub Dict
instance Show a :=> Show [a] where ins = Sub Dict
instance Show a :=> Show (Maybe a) where ins = Sub Dict
instance (Show a, Show b) :=> Show (a, b) where ins = Sub Dict
instance (Show a, Show b) :=> Show (Either a b) where ins = Sub Dict
instance (Integral a, Show a) :=> Show (Ratio a) where ins = Sub Dict
instance () :=> Show (Dict a) where ins = Sub Dict
instance () :=> Show (a :- b) where ins = Sub Dict
 
 
instance Class () (Read a) where cls = Sub Dict
instance () :=> Read () where ins = Sub Dict
instance () :=> Read Bool where ins = Sub Dict
instance () :=> Read Ordering where ins = Sub Dict
instance () :=> Read Char where ins = Sub Dict
instance Read a :=> Read (Complex a) where ins = Sub Dict
instance Read a :=> Read [a] where ins = Sub Dict
instance Read a :=> Read (Maybe a) where ins = Sub Dict
instance (Read a, Read b) :=> Read (a, b) where ins = Sub Dict
instance (Read a, Read b) :=> Read (Either a b) where ins = Sub Dict
instance (Integral a, Read a) :=> Read (Ratio a) where ins = Sub Dict
 
 
instance Class () (Enum a) where cls = Sub Dict
instance () :=> Enum () where ins = Sub Dict
instance () :=> Enum Bool where ins = Sub Dict
instance () :=> Enum Ordering where ins = Sub Dict
instance () :=> Enum Char where ins = Sub Dict
instance () :=> Enum Int where ins = Sub Dict
instance () :=> Enum Integer where ins = Sub Dict
instance () :=> Enum Float where ins = Sub Dict
instance () :=> Enum Double where ins = Sub Dict
instance Integral a :=> Enum (Ratio a) where ins = Sub Dict
 
 
instance Class () (Bounded a) where cls = Sub Dict
instance () :=> Bounded () where ins = Sub Dict
instance () :=> Bounded Ordering where ins = Sub Dict
instance () :=> Bounded Bool where ins = Sub Dict
instance () :=> Bounded Int where ins = Sub Dict
instance () :=> Bounded Char where ins = Sub Dict
instance (Bounded a, Bounded b) :=> Bounded (a,b) where ins = Sub Dict
 
 
instance Class () (Num a) where cls = Sub Dict
instance () :=> Num Int where ins = Sub Dict
instance () :=> Num Integer where ins = Sub Dict
instance () :=> Num Float where ins = Sub Dict
instance () :=> Num Double where ins = Sub Dict
instance RealFloat a :=> Num (Complex a) where ins = Sub Dict
instance Integral a :=> Num (Ratio a) where ins = Sub Dict
 
 
instance Class (Num a, Ord a) (Real a) where cls = Sub Dict
instance () :=> Real Int where ins = Sub Dict
instance () :=> Real Integer where ins = Sub Dict
instance () :=> Real Float where ins = Sub Dict
instance () :=> Real Double where ins = Sub Dict
instance Integral a :=> Real (Ratio a) where ins = Sub Dict
 
 
instance Class (Real a, Enum a) (Integral a) where cls = Sub Dict
instance () :=> Integral Int where ins = Sub Dict
instance () :=> Integral Integer where ins = Sub Dict
 
 
instance Class (Num a) (Fractional a) where cls = Sub Dict
instance () :=> Fractional Float where ins = Sub Dict
instance () :=> Fractional Double where ins = Sub Dict
instance RealFloat a :=> Fractional (Complex a) where ins = Sub Dict
instance Integral a :=> Fractional (Ratio a) where ins = Sub Dict
 
 
instance Class (Fractional a) (Floating a) where cls = Sub Dict
instance () :=> Floating Float where ins = Sub Dict
instance () :=> Floating Double where ins = Sub Dict
instance RealFloat a :=> Floating (Complex a) where ins = Sub Dict
 
 
instance Class (Real a, Fractional a) (RealFrac a) where cls = Sub Dict
instance () :=> RealFrac Float where ins = Sub Dict
instance () :=> RealFrac Double where ins = Sub Dict
instance Integral a :=> RealFrac (Ratio a) where ins = Sub Dict
 
 
instance Class (RealFrac a, Floating a) (RealFloat a) where cls = Sub Dict
instance () :=> RealFloat Float where ins = Sub Dict
instance () :=> RealFloat Double where ins = Sub Dict
 
 
instance Class () (Monoid a) where cls = Sub Dict
instance () :=> Monoid () where ins = Sub Dict
instance () :=> Monoid Ordering where ins = Sub Dict
instance () :=> Monoid [a] where ins = Sub Dict
instance Monoid a :=> Monoid (Maybe a) where ins = Sub Dict
instance (Monoid a, Monoid b) :=> Monoid (a, b) where ins = Sub Dict
 
 
instance Class () (Functor f) where cls = Sub Dict
instance () :=> Functor [] where ins = Sub Dict
instance () :=> Functor Maybe where ins = Sub Dict
instance () :=> Functor (Either a) where ins = Sub Dict
instance () :=> Functor ((->) a) where ins = Sub Dict
instance () :=> Functor ((,) a) where ins = Sub Dict
instance () :=> Functor IO where ins = Sub Dict
 
 
instance Class (Functor f) (Applicative f) where cls = Sub Dict
instance () :=> Applicative [] where ins = Sub Dict
instance () :=> Applicative Maybe where ins = Sub Dict
instance () :=> Applicative (Either a) where ins = Sub Dict
instance () :=> Applicative ((->)a) where ins = Sub Dict
instance () :=> Applicative IO where ins = Sub Dict
instance Monoid a :=> Applicative ((,)a) where ins = Sub Dict
 
 
instance Class (Applicative f) (Alternative f) where cls = Sub Dict
instance () :=> Alternative [] where ins = Sub Dict
instance () :=> Alternative Maybe where ins = Sub Dict
 
 
instance Class () (Monad f) where cls = Sub Dict
instance () :=> Monad [] where ins = Sub Dict
instance () :=> Monad ((->) a) where ins = Sub Dict
instance () :=> Monad (Either a) where ins = Sub Dict
instance () :=> Monad IO where ins = Sub Dict
 
 
instance Class (Monad f) (MonadPlus f) where cls = Sub Dict
instance () :=> MonadPlus [] where ins = Sub Dict
instance () :=> MonadPlus Maybe where ins = Sub Dict
 

Of course, the structure of these definitions is extremely formulaic, so when template-haskell builds against HEAD again, they should be able to be generated automatically using splicing and reify, which would reduce this from a wall of text to a handful of lines with better coverage!

An alternative using Default Signatures and Type Families

Many of the above definitions could have been streamlined by using default definitions. However, MPTCs do not currently support default signatures. We can however, define Class and (:=>) using type families rather than functional dependencies. This enables us to use defaulting, whenever the superclass or context was ().

 
#if 0
class Class h where
  type Sup h :: Constraint
  type Sup h = ()
  cls :: h :- Sup h
  default cls :: h :- ()
  cls = Sub Dict
 
class Instance h where
  type Ctx h :: Constraint
  type Ctx h = ()
  ins :: Ctx h :- h
  default ins :: h => Ctx h :- h
  ins = Sub Dict
 
instance Class (Class a)
instance Class (Instance a)
 
#ifdef UNDECIDABLE
instance Class a => Instance (Class a)
instance Instance a => Instance (Instance a)
#endif
 
instance Class ()
instance Instance ()
#endif
 

This seems at first to be a promising approach. Many instances are quite small:

 
#if 0
instance Class (Eq a)
instance Instance (Eq ())
instance Instance (Eq Int)
instance Instance (Eq Bool)
instance Instance (Eq Integer)
instance Instance (Eq Float)
instance Instance (Eq Double)
#endif
 

But those that aren't are considerably more verbose and are much harder to read off than the definitions using the MPTC based Class and (:=>).

 
#if 0
instance Instance (Eq [a]) where
  type Ctx (Eq [a]) = Eq a
  ins = Sub Dict
instance Instance (Eq (Maybe a)) where
  type Ctx (Eq (Maybe a)) = Eq a
  ins = Sub Dict
instance Instance (Eq (Complex a)) where
  type Ctx (Eq (Complex a)) = Eq a
  ins = Sub Dict
instance Instance (Eq (Ratio a)) where
  type Ctx (Eq (Ratio a)) = Eq a
  ins = Sub Dict
instance Instance (Eq (a, b)) where
  type Ctx (Eq (a,b)) = (Eq a, Eq b)
  ins = Sub Dict
instance Instance (Eq (Either a b)) where
  type Ctx (Eq (Either a b)) = (Eq a, Eq b)
  ins = Sub Dict
#endif
 

Having tested both approaches, the type family approach led to a ~10% larger file size, and was harder to read, so I remained with MPTCs even though it meant repeating "where ins = Sub Dict" over and over.

In a perfect world, we'd gain the ability to use default signatures with multiparameter type classes, and the result would be considerably shorter and easier to read!

Fake Superclasses

Now, that we have all this machinery, it'd be nice to get something useful out of it. Even if we could derive it by other means, it'd let us know we weren't completely wasting our time.

Let's define a rather horrid helper, which we'll only use where a and b are the same constraint being applied to a newtype wrapper of the same type, so we can rely on the fact that the dictionaries have the same representation.

 
evil :: a :- b
evil = unsafeCoerce refl
 

We often bemoan the fact that we can't use Applicative sugar given just a Monad, since Applicative wasn't made a superclass of Monad due to the inability of the Haskell 98 report to foresee the future invention of Applicative.

There are rather verbose options to get Applicative sugar for your Monad, or to pass it to something that expects an Applicative. For instance you can use WrappedMonad from Applicative. We reflect the relevant instance here.

 
instance Monad m :=> Applicative (WrappedMonad m) where ins = Sub Dict
 

Using that instance and the combinators defined previously, we can obtain the following

 
applicative :: forall m a. Monad m => (Applicative m => m a) -> m a
applicative m =
  m \\ trans (evil :: Applicative (WrappedMonad m) :- Applicative m) ins
 

Here ins is instantiated to the instance of (:=>) above, so we use trans to compose ins :: Monad m :- Applicative (WrappedMonad m) with evil :: Applicative (WrappedMonad m) :- Applicative m to obtain an entailment of type Monad m :- Applicative m in local scope, and then apply that transformation to discharge the Applicative obligation on m.

Now, we can use this to write definitions. [Note: Frustratingly, my blog software inserts spaces after <'s in code]

 
(< &>) :: Monad m => m a -> m b -> m (a, b)
m < &> n = applicative $ (,) < $> m < *> n
 

Which compares rather favorably to the more correct

 
(< &>) :: Monad m => m a -> m b -> m (a, b)
m < &> n = unwrapMonad $ (,) < $> WrapMonad m < *> WrapMonad n
 

especially considering you still have access to any other instances on m you might want to bring into scope without having to use deriving to lift them onto the newtype!

Similarly you can borrow < |> and empty locally for use by your MonadPlus with:

 
instance MonadPlus m :=> Alternative (WrappedMonad m) where ins = Sub Dict
 
alternative :: forall m a. MonadPlus m => (Alternative m => m a) -> m a
alternative m =
  m \\ trans (evil :: Alternative (WrappedMonad m) :- Alternative m) ins
 

The correctness of this of course relies upon the convention that any Applicative and Alternative your Monad may have should agree with its Monad instance, so even if you use Alternative or Applicative in a context where the actual Applicative or Alternative instance for your particular type m is in scope, it shouldn't matter beyond a little bit of efficiency which instance the compiler picks to discharge the Applicative or Alternative obligation.

Note: It isn't that the Constraint kind is invalid, but rather that using unsafeCoerce judiciously we can bring into scope instances that don't exist for a given type by substituting those from a different type which have the right representation.

[Source]

by Edward Kmett at November 04, 2011 03:01 AM

Epilogue for Epigram

Problem Types, Dub Types, Type Schemes

Peter Morris is visiting Glasgow, so we’re trying to get on with writing an Epigram Elaborator. Let’s walk through the layers of the Epigram system. Evidence There’s an underlying type theory, the Language Of Verifiable Evidence. We have a bidirectional typechecker which means it can be quite compact, even though it is fully explicit. Necessarily, we [...]

by Conor at November 04, 2011 12:09 AM

October 31, 2011

Functional Jobs

Open-source Scala/web developer at Northwestern University at Northwestern University (Full-time)

The CCL at Northwestern University is looking to hire a new software developer to work on NetLogo and our NetLogo-related grants. The job description is below. Please forward to anyone you think might be interested.

Job Summary:

The Software Developer position is based at Northwestern University’s Center for Connected Learning and Computer-Based Modeling (CCL). You'd be working in a small development team in a university research group that also includes professors, postdocs, graduate students, and undergraduates, supporting the needs of multiple research projects. A major focus would be on development of NetLogo, an open-source modeling environment for both education and scientific research. Our grants also involve development work on HubNet and other associated tools for NetLogo, especially our educational NSF grants involving building, delivering, and assessing NetLogo-based science curricula for secondary schools.

NetLogo is a programming language and agent-based modeling environment. The NetLogo language is a dialect of Logo/Lisp specialized for building agent-based simulations of natural and social phenomena. NetLogo has tens of thousands of users ranging from grade school students to advanced researchers. A collaborative extension of NetLogo, called HubNet, enables groups of participants to run participatory simulation activities in classrooms and distributed participatory simulations in social science research.

Specific Responsibilities:

Collaborates with the NetLogo development team in designing features for NetLogo, HubNet and web-based curriculum delivery environments; Writes code independently, but under direction of senior software engineer and principal investigator; Creates, updates and documents existing curricular activities and models using NetLogo, HubNet and web-based applications; Creates new such activities; Supports development of new devices to interact with HubNet; Interacts with commercial and academic partners to help determine design and functional requirements for NetLogo and HubNet; Interacts with user community including responding to bug reports, questions, and suggestions, and interacting with open-source contributors; Performs data collection, organization, and summarization for projects; Assists senior software engineer with coordination of team activities; Performs related duties as required or assigned.

Minimum Qualifications:

A bachelor's degree in computer science or a closely related field or the equivalent combination of education, training and experience from which comparable skills and abilities may be acquired; Enthusiasm for writing clean, modular, well-tested code.

Preferred Qualifications:

Experience with working effectively as part of a small software development team, including close collaboration, distributed version control, and automated testing; Experience with at least one JVM language such as Java; Experience with Scala programming, or enthusiasm for learning it; Experience with Haskell, Lisp, or other functional languages; Interest in and experience with programming language implementation, functional programming, and metaprogramming; Experience with GUI design; Interest in and experience with computer-based modeling and simulation, especially agent-based simulation; Interest in and experience with distributed, multiplayer, networked systems like HubNet; Experience with building web-based applications, both server-side and client-side components; Experience working on research projects in an academic environment; Experience with open-source software development and supporting the growth of an open-source community; Interest in education and an understanding of secondary school math and science content.

The Northwestern campus is in Evanston, Illinois on the Lake Michigan shore, adjacent to Chicago and easily reachable by public transportation.

--Uri

Uri Wilensky uri [at] northwestern [dot] edu

Get information on how to apply for this position.

October 31, 2011 12:47 AM

Open-source Scala/Web Developer at Northwestern University (Full-time)

The CCL at Northwestern University is looking to hire a new software developer to work on NetLogo and our NetLogo-related grants. The job description is below. Please forward to anyone you think might be interested.

Job Summary:

The Software Developer position is based at Northwestern University’s Center for Connected Learning and Computer-Based Modeling (CCL). You'd be working in a small development team in a university research group that also includes professors, postdocs, graduate students, and undergraduates, supporting the needs of multiple research projects. A major focus would be on development of NetLogo, an open-source modeling environment for both education and scientific research. Our grants also involve development work on HubNet and other associated tools for NetLogo, especially our educational NSF grants involving building, delivering, and assessing NetLogo-based science curricula for secondary schools.

NetLogo is a programming language and agent-based modeling environment. The NetLogo language is a dialect of Logo/Lisp specialized for building agent-based simulations of natural and social phenomena. NetLogo has tens of thousands of users ranging from grade school students to advanced researchers. A collaborative extension of NetLogo, called HubNet, enables groups of participants to run participatory simulation activities in classrooms and distributed participatory simulations in social science research.

Specific Responsibilities:

Collaborates with the NetLogo development team in designing features for NetLogo, HubNet and web-based curriculum delivery environments; Writes code independently, but under direction of senior software engineer and principal investigator; Creates, updates and documents existing curricular activities and models using NetLogo, HubNet and web-based applications; Creates new such activities; Supports development of new devices to interact with HubNet; Interacts with commercial and academic partners to help determine design and functional requirements for NetLogo and HubNet; Interacts with user community including responding to bug reports, questions, and suggestions, and interacting with open-source contributors; Performs data collection, organization, and summarization for projects; Assists senior software engineer with coordination of team activities; Performs related duties as required or assigned.

Minimum Qualifications:

A bachelor's degree in computer science or a closely related field or the equivalent combination of education, training and experience from which comparable skills and abilities may be acquired; Enthusiasm for writing clean, modular, well-tested code.

Preferred Qualifications:

Experience with working effectively as part of a small software development team, including close collaboration, distributed version control, and automated testing; Experience with at least one JVM language such as Java; Experience with Scala programming, or enthusiasm for learning it; Experience with Haskell, Lisp, or other functional languages; Interest in and experience with programming language implementation, functional programming, and metaprogramming; Experience with GUI design; Interest in and experience with computer-based modeling and simulation, especially agent-based simulation; Interest in and experience with distributed, multiplayer, networked systems like HubNet; Experience with building web-based applications, both server-side and client-side components; Experience working on research projects in an academic environment; Experience with open-source software development and supporting the growth of an open-source community; Interest in education and an understanding of secondary school math and science content.

The Northwestern campus is in Evanston, Illinois on the Lake Michigan shore, adjacent to Chicago and easily reachable by public transportation.

--Uri

Uri Wilensky uri [at] northwestern [dot] edu

Get information on how to apply for this position.

October 31, 2011 12:47 AM

October 28, 2011

Martin Sulzmann

Haskell Actors

There's some recent interest about Actor-style concurrency in the Haskell context, e.g. see

My own work in this context, see actor, is orthogonal to the above mentioned works.

The main point of actor is the ability to pattern match over multiple events (messages) in the message queue. Here's a simple example. Suppose, you write a market-place actor which receives requests from sellers and buyers. The task is to find matching sellers and buyers. In the actor extension this can be elegantly specified as follows (I'm using some pseudo-syntax here):


receive Seller item & Buyer item --> ...


Try to express the same using standard actors where we can pattern match over a single message only. There's more info about actors with multi-headed receive clauses in some earlier post. Please check out the COORDINATION'08 paper (online) for the details of the matching algorithm which is implemented in actor.

I retrospect, I should have named the package 'multi-headed-actor', what a freaky name :)

The actor approach is built upon concurrent channels but instead we could rely on distributed channels based on haskell-mpi or ErlangInHaskell.

This idea (of receive patterns with multiple heads) has been picked up by others, e.g. see the following works which basically adopt the matching algorithm from actor


  • Hubert Plociniczak and Susan Eisenbach. JErlang: Erlang with Joins

  • Scala Joins Library:
    Implementing Joins using Extensible Pattern Matching
    by Philipp Haller and Tom Van Cutsem

by Martin Sulzmann (noreply@blogger.com) at October 28, 2011 07:34 PM

Snap Framework

Announcing: Snap Framework v0.6

The Snap team is excited to announce the release of version 0.6 of the Snap Framework. This is a big release with a lot of new functionality. Here is a list of the major changes since 0.5.5:

New features / improvements

  • The most significant new feature in 0.6 is the snaplet API. It is a ground-up rewrite of our previous extensions system. Snaplets are self-contained portions of websites that can be easily composed to create larger sites. The snaplet infrastructure makes it simple to distribute snaplets for others to use, and provides a clear pathway for contributing to the community. State management, built-in config file infrastructure, and automatic snaplet resource installation are some of the notable things the snaplet infrastructure does for you. For more information on getting started with snaplets, check out the snaplet tutorial. Also be sure to take a look at the API documentation. It is written as prose and should function as a more comprehensive, api-oriented tutorial.

  • Added a session management interface and a built-in snaplet back end based on HTTP cookies.

  • Added a snaplet interface for user authentication. Out of the box, Snap 0.6 comes with a simple sample implementation of this user authentication interface that stores user information on the filesystem as an example. We expect, however, that users choose a more sophisticated implementation of this interface corresponding to their choice of data store. Jurriën Stutterheim will be releasing an implementation based on HDBC within a few days.

  • Added a new testing API to snap-core to make it easier to test Snap applications. See the Snap.Test module.

  • The Cookie data structure got two new fields for flagging secure or HTTP-only cookies.

  • Added a new convenience function expireCookie.

  • Made logging more flexible by allowing you to specify an arbitrary IO action.

  • heist: added getTemplateFilePath and addTemplatePathPrefix functions.

  • heist: markdownSplice no longer requires the template directory as a parameter. It uses the directory of the template currently being processed.

  • heist: added the templateNames and spliceNames functions to allow the user to see what templates and splices are being used by Heist.

API Changes

  • Headers from snap-core has been changed to an opaque data type, and the implementation has changed from Data.Map to the more efficient Data.HashMap from unordered-containers. This will break code that expects Headers to be a type alias for Map.

  • The Snap.Types module has been renamed to the more appropriate Snap.Core. Snap.Types has been deprecated and made an alias, but will be going away with the next few major versions.

  • heist: renamed TemplateMonad to the more appropriate HeistT. Again, TemplateMonad is still there, but deprecated and will be removed in a future release.

  • heist: deprecated the Text.Templating.Heist.Splices.Static module. The Cache module now binds both a <cache> and <static> tag.

  • Removed previously deprecated fileServe functions.

  • Deprecated the getRequestBody function from Snap.Core; it read request bodies into memory without specifying a maximum size, meaning that an attacker could DoS a server using this function by uploading an arbitrarily large request. The function has been replaced by readRequestBody, which forces you to choose a maximum length.

  • Removed the deprecated addCookie function.

  • Added a mime type for JSON.

Dependency changes

  • Increased the lower version bound on the case-insensitive package from 0.2 to 0.3.

by Doug Beardsley (mightybyte@gmail.com) at October 28, 2011 06:16 AM

October 24, 2011

Leon P Smith

Concurrency and Foreign Functions in the Glasgow Haskell Compiler

I had an informative chat with Simon Marlow and Jules Bean a little while ago about concurrency in the Glasgow Haskell Compiler and its interaction with foreign code, specifically the difference between safe and unsafe calls. It turns out that the implementation is both simpler and more powerful than my vague misconceptions about how it worked. And since there seems to be quite a few myths out there, I thought I should write up an overview of GHC’s implementation.

Haskell’s concurrency model is relatively simple. Conceptually, Haskell threads are OS threads, and in most cases the programmer can treat them as such. This is known as a 1-1 threading model. In actuality, GHC’s implementation uses an M-N model, but provides an illusion of a 1-1 model. As one would expect, there are a few caveats to this illusion, but they tend to be minor. First, we’ll cover three fundamental components of GHC’s concurrency:

Haskell Threads

Haskell threads are implemented in GHC via cooperative multitasking. They are scheduled by GHC’s run-time system, making them the M of the M-N model. Yield points are generated automatically by the compiler, which provides an illusion of preemptive multitasking to the programmer. Also, the stack for each thread starts small but grows as needed. Cooperative multitasking and growable stacks make Haskell threads cheap and efficient, typically scaling to millions of threads on contemporary computer systems.

The downside is that GHC’s threads cannot by themselves take advantage of multiple CPUs. Also, foreign functions don’t cooperatively multitask. Notably, since GHC implements its arbitrary-precision Integer datatype via the GNU Multiple Precision Library, the illusion of preemptive multitasking can be less than perfect when executing Integer-heavy code.

To make use of Haskell threads, all you have to do is create them by calling Control.Concurrent.forkIO. There are no special compile-time, link-time, or run-time options needed to enable them.

Operating System Threads

OS threads offer true preemptive multitasking and are scheduled by the kernel. They are also more expensive, typically scaling to thousands of threads on contemporary systems. However, they can execute on multiple CPUs, and they provide a way to deal with foreign calls that either block or take a long time to execute.

To use OS threads, all you do need to link your executable with GHC’s threaded runtime system using the -threaded option. There is nothing particularly special you need to do inside Haskell code to utilize operating system threads; GHC’s threaded runtime will endeavor to maintain the illusion that Haskell threads are OS threads as best it can.

Capabilities

In the context of GHC, a capability is an operating system thread that is able to execute Haskell code. GHC schedules Haskell threads among the capabilities, making them the N of the M-N model. Previous versions of GHC only supported one capability, but GHC now supports as many capabilities as you want. While the total number of capabilities remains fixed, the collection of OS threads that are capabilities changes over time.

You can set the number of capabilities “x” by using the -N[x] run-time option. This can be done either by passing the executable +RTS -N[x] on the command line, or by setting the default RTS options by linking the executable with -with-rtsopts="-N[x]". Note that you will need the -threaded and may need the -rtsopts link-time options.

The Foreign Function Interface

The FFI supports two kinds of calls: safe and unsafe. A “safe” call blocks the calling OS thread and Haskell thread until the call finishes. Blocking these is unavoidable. However, a safe call does not block the capability. When GHC performs a safe call, it performs the call inside the current OS thread, which is a capability. The capability then moves to another OS thread. If no other threads are available, GHC will transparently create a new OS thread. OS threads are pooled to try to avoid creating or destroying them too often.

An unsafe call blocks the capability in addition to blocking the OS and Haskell threads. This was a pretty big deal when GHC only supported a single capability. In effect, an unsafe call would block not only the Haskell thread that made the call, but every Haskell thread. This gives rise to the myth that unsafe calls block the entire Haskell runtime, which is no longer true. An unsafe foreign call that blocks is still undesirable, but depending on configuration, may not be as bad as it used to be.

The advantage of unsafe calls is that they entail significantly less overhead. Semantically, a foreign function that might call back into Haskell must be declared as a “safe” call, whereas foreign function that will never call back into Haskell may be declared as “unsafe”. As a result, safe calls must perform extra bookkeeping.

(As an aside, this vocabulary is not particularly consistent with Haskell’s other use of “safe” and “unsafe”. In this other sense, foreign calls are unsafe because they can be used to break the abstractions that GHC provides.)

In practice, most foreign functions will never call back into Haskell. Thus choosing whether to mark a foreign function as safe or unsafe usually revolves around the trade-off between concurrency and overhead. If needed, you can declare both a safe and an unsafe binding to the same foreign function.

Bound threads

Finally, the most significant caveat to GHC’s 1-1 threading illusion is that some foreign code (notably OpenGL) expects to be run in a single OS thread, due to the use of thread-local storage or the like.

GHC’s answer is “bound threads”. GHC provides forkOS, which creates a Haskell thread with a special property: every foreign call that the Haskell thread makes is guaranteed to happen inside the same OS thread. It is perhaps a bad choice of name, as there is no need to call forkOS to utilize operating system threads. The only time you need to call forkOS is when you need an honest-to-goodness 1-1 correspondence between a Haskell thread and a kernel thread.

Finishing up

These kinds of illusions are a common theme in GHC: what appears to be synchronous blocking IO is in fact asynchronous non-blocking IO, what appears to be preemptive multitasking is in fact cooperative multitasking, what appears to be 1-1 threading is actually M-N threading. GHC endeavors to present the user with a nice interface, while using more efficient techniques behind the scenes. And, for the most part, these illusions hold up pretty well.

I hope the community finds this overview helpful. Most of what I’ve talked about is covered in “Extending the Haskell Foreign Function Interface with Concurrency” by Simon Marlow, Simon Peyton Jones, and Wolfgang Thaller. I hope this post is a useful introduction to the paper.


by lpsmith at October 24, 2011 07:22 PM

David Peixotto

Haskell Programs Run Slower with llvm-gcc

In this article I discuss the issues with compiling GHC using the llvm-gcc C compiler and the implications for the performance of Haskell programs. I find that using llvm-gcc causes Haskell programs to run on average 10.54% slower which we can reduce to 3.55% slower by making invasive changes in the GHC garbage collector.

With the release of MacOSX 10.7 (Lion) Apple changed the default C compiler to llvm-gcc, which is a compiler that uses a GCC version 4.2 to pre-process and parse the source code, but uses LLVM to optimize and generate machine code. This article discusses the implications of this switch for the Glasgow Haskell Compiler. Although LLVM has received a lot of attention for its many optimizations we actually see a decrease in the performance of Haskell programs that are compiled with a GHC that has been built using the llvm-gcc compiler.

Why do we care about the C compiler for Haskell programs anyway?

Since we are talking about Haskell programs, it may sound a little odd to be discussing the impact a C compiler can have on performance. The reason the C compiler matters is that the GHC runtime (RTS) is primarily written in C. Any Haskell code that spends a significant amount of time in the runtime will be effected by performance regressions caused by the C compiler.

The major component of the RTS that concerns most Haskell programs is the garbage collector. The RTS is also deals with scheduling and synchronizing threads, but for the majority of programs I'd suspect that garbage collection is the main place where Haskell programs will spend time in the RTS.

So if we slow down the code that gets executed during a garbage collection we will slow down our Haskell programs. Of course before we can see the performance impact that llvm-gcc has on Haskell programs, we have to get GHC to compile using llvm-gcc.

What are the issues with compiling GHC using llvm-gcc?

GHC uses several GCC extensions in the RTS code to get better performance. The main difficulty we have to deal with to use llvm-gcc is the lack of support in LLVM for both __thread variables and global register variables. In GHC's multi-threaded runtime, the garbage collector keeps a piece of thread-local data for each thread that is participating in a GC. When compiling with GCC, GHC can keep this data in either a __thread variable, or by pinning it into a register using a global register declaration. Since LLVM does not support either of those extensions, we have to find a different solution.

I explored two approaches to accessing thread-local data in the garbage collector: using pthread_getspecific and passing the data as an extra parameter around in the garbage collector. Using pthread_getspecific is a fairly small change due to the nice way the thread-local access is factored in the code. On the other hand, adding an extra paramater to each function in the garbage collector is an invasive change, and even worse is that we only want to do it when compiling with llvm-gcc so there is some C pre-processor yuckiness that goes along with this change.

It would be nice if the pthread_getspecific solution worked well because it requires minimal changes to the source code. Unfortunately, the performance results show that we are much better off passing the data as a parameter in the GC.

What are the performance implications?

To measure the impact of the changes, I compiled a patched GHC with llvm-gcc and compared the two approaches described above. All measurements were done using the fibon benchmark suite on MacOSX 10.7 and compared against a baseline GHC head (7.3) that accesses thread-local data from a global register. The full results along with the raw data is available here.

The results show that using llvm-gcc with pthread_getspecific causes a 10.54% average increased walltime and passing the data as parameter causes a 3.55% average increased walltime compared to the programs compiled using a GHC built with a standard gcc. These results were collected using GHC's threaded RTS so they show the overheads from our changes as well as any overhead we get simply by moving to llvm-gcc. If we look at the results from running with the non-threaded runtime we see a 1.54% increased walltime, but a 7.51% increase in the time spent in the GC. This 7.51% is the overhead we get just by moving to llvm-gcc without any extra overhead from the changes needed to support thread-local data.

If we take a closer look at where the increase in execution time comes from, we can see that basically all of the increase comes from the garbage collector. The time spent in the mutator (i.e. the time spent executing Haskell code) is virtually unchanged from the baseline GHC while the time spent in the GC increased by 33.64% in the pthread version and 9.15% paramater passing version. The degree to which this increase translates into an increase in walltime depends on the efficiency of the program. Those programs that spend more of their overall execution time in the garbage collector see a greater increase in their wall clock time.

An obvious question to ask is "where do these overheads come from"? The only overhead I can speak confidently about is the one that comes from using pthread_getspecific. On MacOS it is defined using a snippet of assembly that loads from a specific offset from the thread-local memory segment. Unfortunately, the pthread library is dynamically linked so every time we call pthread_getspecific we pay the call of an extra jump and the compiler is unable to eliminate the repeated loads. The thread-local data is used very frequently inside the garbage collector and all those calls add up to a large overhead. I'm still not sure why the non-threaded runtime has a 7.5% increase in GC time when using llvm-gcc, but it seems there is some factor that allows gcc to generate better code than llvm in this case.

What about clang?

It looks fairly obvious that Apple is on a path to drop the gcc toolchain entirely and for a clang/llvm combination. It would be great to get GHC to build using clang, but my previous attempts ran into a number of problems. Some of those problems have since been fixed with my new patches and other upstream fixes, so it may be time to try again. I think the main hurdle to overcome will be the behavior of the C pre-processor because of GHC's use of the traditional mode for the pre-processor. If we can use a separate pre-processor and just use clang as the compiler then there is probably a good chance of success.

by David Peixotto (noreply@blogger.com) at October 24, 2011 03:10 PM

October 21, 2011

Gregory Collins

Announcing: "hashtables", a new Haskell library for fast mutable hash tables

<script language="javascript" src="http://gregorycollins.net/util.js" type="text/javascript"></script> <script language="javascript" src="http://gregorycollins.net/data.js" type="text/javascript"></script> <script language="javascript" src="http://gregorycollins.net/jquery.js" type="text/javascript"></script> <script language="javascript" src="http://gregorycollins.net/jquery.flot.js" type="text/javascript"></script> <style type="text/css"> body { font-family: sans-serif; margin: 50px auto; max-width: 960px; } figure { font-size: 80%; text-align: center; color: #777; margin: 2em auto; } div.chart { width: 610px; height: 340px; } table.chartTable { margin: 1em auto; } table.chartTable td.yaxis { width: 60px; } table.chartTable td { font-size: 70%; color: #888; } </style>

I’m very pleased to announce today the release of the first version of hashtables, a Haskell library for fast mutable hash tables. The hashtables library contains three different mutable hash table implementations in the ST monad, as well as a type class abstracting out the functions common to each and a set of wrapper functions to use the hash tables in the IO monad.

What’s included?

The hashtables library contains implementations of the following data structures:

  • Data.HashTable.ST.Basic: a basic open addressing hash table using linear probing as the collision resolution strategy. On a pure speed basis, this should be the fastest currently-available Haskell hash table implementation for lookups, although it has a higher memory overhead than the other tables. Like many hash table implementations, it can also suffer from long delays when the table is grown due to the rehashing of all of the elements in the table.

  • Data.HashTable.ST.Cuckoo: an implementation of Cuckoo hashing, as introduced by Pagh and Rodler in 2001. Cuckoo hashing features worst-case O(1) lookups and can reach a high “load factor”, meaning that the table can perform acceptably well even when more than 90% full. Randomized testing shows this implementation of cuckoo hashing to be slightly faster on insert and slightly slower on lookup than Data.HashTable.ST.Basic, while being more space-efficient by about a half word per key-value mapping. Cuckoo hashing, like open-addressing hash tables, can suffer from long delays when the table is forced to grow.

  • Data.HashTable.ST.Linear: a linear hash table, which trades some insert and lookup performance for higher space efficiency and much shorter delays during table expansion. In most cases, randomized testing shows this table to be slightly faster than Data.HashTable from the Haskell base library.

Why Data.HashTable is slow

People often remark that the hash table implementation from the Haskell base library is slow. Historically, there have been a couple of reasons why: firstly, Haskell people tend to prefer persistent data structures to ephemeral ones. Secondly, until GHC 6.12.2, GHC had unacceptably large overhead when using mutable arrays due to a lack of card marking in the garbage collector.

However, performance testing on newer versions of GHC still shows the hash table implementation from the Haskell base library to be slower than it ought to be. To explain why, let’s examine the data type definition for Data.HashTable:

data HashTable key val = HashTable {
cmp :: !(key -> key -> Bool),
hash_fn :: !(key -> Int32),
tab :: !(IORef (HT key val))
}

data HT key val
= HT {
kcount :: !Int32, -- Total number of keys.
bmask :: !Int32,
buckets :: !(HTArray [(key,val)])
}

For now, let’s ignore the HashTable type, as it is essentially just an IORef wrapper around the HT type, which contains the actual table. The hash table from Data.HashTable uses separate chaining, in which keys are hashed to buckets, each of which contains a linked list of (key,value) tuples. To explain why this is not an especially smart strategy for hash tables in Haskell, let’s examine what the memory layout of this data structure looks like at runtime.

Memory layout of Data.HashTable

Memory layout of Data.HashTable

Each arrow in the above diagram represents a pointer which, when dereferenced, can (and probably does) cause a CPU cache miss. During lookup, each time we test an entry in one of the buckets against the lookup key, we cause three cache lines to be loaded:

  • one for the cons cell itself
  • one for the tuple
  • one for the key

If the average bucket has b elements in it, the average successful lookup causes 1 + 3b/2 cache line loads: one to dereference the buckets array to get the pointer to the first cons cell, and 3b/2 to find the matching key in the buckets array. An unsuccessful lookup is worse, causing a full 1 + 3b cache line loads, because we need to examine every key in the bucket.

Why this new library is faster

The datatype inside Data.HashTable.ST.Basic looks like this:

data HashTable_ s k v = HashTable
{ _size :: {-# UNPACK #-} !Int
, _load :: !(U.IntArray s)
, _hashes :: !(U.IntArray s)
, _keys :: {-# UNPACK #-} !(MutableArray s k)
, _values :: {-# UNPACK #-} !(MutableArray s v)
}

Here, to avoid pointer indirections, I’ve flattened the keys and values into parallel arrays, and stored the hash code for every cell in an unboxed array to speed lookups. Lookup in these three parallel arrays looks like this:

Memory layout of Data.HashTable.ST.Basic

Memory layout of Data.HashTable.ST.Basic

The coloured region represents the location of the key we are looking for. Counting the cache line loads for a typical successful lookup in this structure:

  • one, maybe two, to find the correct hash code in the hash codes array. Note that since this is a contiguous unboxed integer array, a cache line load causes eight (sixteen on 32-bit machines) hash codes to be loaded into cache at once.
  • one to dereference the key pointer in the keys array.
  • one to dereference the key.
  • one to dereference the value pointer in the values array.

Unsuccessful lookups are even faster here, since we don’t touch the keys or values arrays except on hash collisions. Astute readers will wonder why we store the hash codes in the table, because this isn’t always done. There are a couple of reasons:

  • we can use the hash code slot to mark the cell as empty or deleted by writing a privileged value (zero or one). If we didn’t do this, we would either have to use a datatype something like Maybe in the keys array to distinguish empty or deleted entries, causing an extra indirection, or we would have to play tricks with unsafeCoerce and/or reallyUnsafePtrEquality# (the linear hash table actually uses these tricks to save indirections, but they’re marked “unsafe” for a reason!)
  • For the types we really care about (specifically ByteString, Text, and SmallString), the Eq instance is O(n) in the size of the key, as compared to the O(1) machine instruction required to compare two hash codes.
  • Keeping the hash codes in an unboxed array allows us to do super-efficient branchless cache line lookups in C. Moreover, we can take advantage of 128-bit SSE4.1 instructions on processors which support them (Intel chips, Core 2 and above) to make searching cache lines for hash codes even faster.
  • I tested it both ways, and storing the hash codes was consistently faster.

Performance measurements

I ran benchmarks for lookup and insert performance of the three hash table implementations included in the hashtables library against:

Unfortunately I cannot release the benchmark code at this time, as it relies on an unfinished data-structure benchmarking library based on criterion, for which I have not yet sought permission to open-source.

The methodology for lookups is:

  • create a vector of N random key-value pairs, where the key is a random ByteString consisting of ASCII hexadecimal characters, between 8 and 32 bytes long, and the value is an Int
  • load all of the key-value pairs into the given datastructure
  • create a vector of N/2 random successful lookups out of the original set
  • perform all of the lookups, and divide the total time taken by the number of lookups to get a per-operation timing
  • repeat the above procedure using criterion enough times to be statistically confident about the timings

The above procedure is repeated for a doubling set of values of N from 250 through to 2,048,000. Note that, to be fair and to ensure that random fluctuations in the input distribution don’t influence the timings for the different data structures, each trial uses the same input set for each data structure. We also force a garbage collection run between trials to try to isolate the unpredictable impact of garbage collection runs as much as possible.

The methodology for inserts is similar:

  • create a vector of N random key-value pairs as described above
  • time how long it takes to load the N key-value pairs into the given data structure. Where applicable, the data structure is pre-sized to fit the data set (i.e. for the hash tables). Note here, though, that I’m not being 100% fair to Data.HashTable in this test, as the newHint function wasn’t called — when I tried to use it, the benchmark took forever. I’m ashamed to say that I didn’t dig too deeply into why.

The benchmarks were run on a MacBook Pro running Snow Leopard with an Intel Core i5 processor, running GHC 7.0.3 in 64-bit mode. The RTS options passed into the benchmark function were +RTS -N -A4M.

Lookup performance

Avg time per lookup (seconds)
Input Size

Lookup performance, log-log plot

Avg time per lookup (seconds)
Input Size

Insert performance

Avg time per insert (seconds)
Input Size

Insert performance, log-log plot

Avg time per insert (seconds)
Input Size

Performance measurements, round 2

My first thought upon seeing these graphs was: “what’s with the asymptotic behaviour?” I had expected lookups and inserts for most of the hash tables to be close to flat, especially for cuckoo hash, which is guaranteed O(1) for lookups in the worst case. I had some suspicions, and re-running the tests with the RTS flags set to +RTS -N -A4M -H1G (specifying a 1GB suggested heap size) seems to confirm them:

Lookup performance

Avg time per lookup (seconds)
Input Size

Lookup performance, Log-log plot

Avg time per lookup (seconds)
Input Size

Insert performance

Avg time per insert (seconds)
Input Size

Insert Performance, Log-log plot

Avg time per insert (seconds)
Input Size

These are more or less the graphs I had been expecting to see. The main impact of setting -H1G is to reduce the frequency of major garbage collections, and the difference here would suggest that garbage collection overhead is what’s causing the poorer-than-expected asymptotic performance. The linear probing and cuckoo hash tables included in this library use very large boxed mutable arrays. The GHC garbage collector uses a card marking strategy in which mutable arrays carry a “card table” containing one byte per k entries in the array (I think here k is 128). When the array is written to, the corresponding entry in the card table is marked “dirty” by writing a “1” into it. It would seem that the card table is scanned during garbage collection no matter whether the array has been dirtied or not; this would account for the small linear factor in the asymptotic time complexity that we’re seeing here.

Conclusion

While Haskell people prefer to use immutable/persistent data structures in their programs most of the time, there are definitely instances in which you want a mutable hash table: no immutable data structure that I know of supports O(1) inserts and lookups, nor is there an immutable data structure that I know of which can match a good mutable hash table for space efficiency. These factors are very important in some problem domains, especially for things like machine learning on very large data sets.

The lack of a really good Haskell hash table implementation has been a sticking point for quite some time for people who want to work in these problem domains. While the situation is still not as good as it might eventually be due to continuing concerns about how the GHC garbage collector deals with mutable arrays, it’s my hope that the release of the hashtables library will go a long way towards closing the gap.

The source repository for the hashtables library can be found on github. Although I’ve made substantial efforts to test this code prior to release, it is a “version 1.0”. Please send bug reports to the hashtables github issues page.

Update (Oct 21, 2011)

A fellow by the name of Albert Ward has translated this blog post into Bulgarian.

<script type="text/javascript"> var axisFont = { family: 'arial', size: 11, style: 'normal', weight: 'normal', variant: 'normal' }; function showTooltip(x, y, contents) { $('
' + contents + '
').css( { position: 'absolute', display: 'none', top: y + 5, left: x + 5, border: '1px solid #fdd', padding: '6px', 'background-color': '#fee', opacity: 0.95 }).appendTo("body").fadeIn(200); } var previousDataIndex = null; var previousSeriesIndex = null; function hoverText(item) { var i = item.seriesIndex; var j = item.dataIndex; var series = item.series; var row = series.data[j]; var label = series.label; var obj = row[2]; var isz = obj['size']; var mean = humanSeconds(obj['mean'], 3); var sd = humanSeconds(obj['stddev'], 3); var n95 = humanSeconds(obj['n95'], 3); return ("" + label + "" + "
input size: " + isz + "
mean: " + mean + "
stddev: " + sd + "
95th percentile: " + n95); } function hoverFunc(event, pos, item) { if (item) { if (previousDataIndex != item.dataIndex || previousSeriesIndex != item.seriesIndex) { previousDataIndex = item.dataIndex; previousSeriesIndex = item.seriesIndex; $("#tooltip").remove(); showTooltip(item.pageX, item.pageY, hoverText(item)); } } else { $("#tooltip").remove(); previousDataIndex = null; previousSeriesIndex = null; } } function logSeries(seriesList) { var outSeriesList = []; for (var i in seriesList) { var series = seriesList[i]; var outseries = {}; outseries['label'] = series.label; var outdata = []; var data = series.data; for (var j in data) { var row = data[j]; var x = row[0]; var y = row[1]; var z = row[2]; outdata.push([Math.log(x), Math.log(y), z]); } outseries['data'] = outdata; outSeriesList.push(outseries); } return outSeriesList; } function mkChart(elem, series, lbl, logPlot) { if (logPlot) series = logSeries(series); var opts = { legend: { position: 'nw' }, series: { lines: { show: true }, points: { show: true } }, grid: { hoverable: true }, xaxis: { autoscaleMargin: 0.02, font: axisFont, label: 'Input Size' }, yaxis: { tickFormatter: function(x) { return humanSeconds(x,1); }, font: axisFont, label: lbl } }; if (logPlot) { var l = function (x) { return Math.log(x); }; var il = function (x) { return Math.exp(x); }; opts['xaxis']['tickFormatter'] = function (x) { return Math.exp(x).toFixed(0); } opts['yaxis']['tickFormatter'] = function (x) { return humanSeconds(Math.exp(x), 1); } /* opts['xaxis']['transform'] = l; opts['xaxis']['inverseTransform'] = il; opts['yaxis']['transform'] = l; opts['yaxis']['inverseTransform'] = il; */ } var plot = $.plot(elem, series, opts); elem.bind("plothover", hoverFunc); return plot; } $(function () { mkChart( $("#lookupChart"), lookupSeries, 'Avg time per\nlookup (s)', false); mkChart( $("#insertChart"), insertSeries, 'Avg time per\ninsert (s)', false); mkChart( $("#lookupChartLog"), lookupSeries, 'Avg time per\nlookup (s)', true); mkChart( $("#insertChartLog"), insertSeries, 'Avg time per\ninsert (s)', true); mkChart( $("#lookupChartH1G"), lookupSeriesH1G, 'Avg time per\nlookup (s)', false); mkChart( $("#insertChartH1G"), insertSeriesH1G, 'Avg time per\ninsert (s)', false); mkChart( $("#lookupChartLogH1G"), lookupSeriesH1G, 'Avg time per\nlookup (s)', true); mkChart( $("#insertChartLogH1G"), insertSeriesH1G, 'Avg time per\ninsert (s)', true); }); </script>

by Gregory Collins (greg@gregorycollins.net) at October 21, 2011 02:27 PM

October 18, 2011

David Terei

Safe Haskell slides from Haskell Implementors Workshop talk

I presented a talk this year at the Haskell Implementors Workshop over in Tokyo. It was a great time, attending some very interesting talks and meeting up with old and new friends in the Haskell and Functional Programming community. My talk was on Safe Haskell, an extension recently added to GHC that I've been working on with David Mazières, Simon Marlow and Simon Peyton Jones. I want to write a blog post about it at some point, but for now here are the slides from the talk:



Safe Haskell <iframe frameborder="0" height="355" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/9755401" width="425"></iframe>

View more presentations from dterei


You can also download the slides in PDF form here.

by David Terei (noreply@blogger.com) at October 18, 2011 06:47 PM

Dominic Orchard

Subcategories & “Exofunctors” in Haskell

In my previous post I discussed the new constraint kinds extension to GHC, which provides a way to get type-indexed constraint families in GHC/Haskell. The extension provides some very useful expressivity. In this post I’m going to explain a possible use of the extension.

In Haskell the Functor class is misleading named as it actually captures the notion of an endofunctor, not functors in general. This post shows a use of constraint kinds to define a type class of exofunctors; that is, functors that are not necessarily endofunctors. I will explain what all of this means.

This example is just one from a draft note explaining the use of constraint families, via the constraint kinds extension, for describing abstract structures from category theory that are parameterised by subcategories, including non-endofunctors, relative monads, and relative comonads.

I will try to concisely describe any relevant concepts from category theory, through the lens of functional programming, although I’ll elide some details.

The Hask category

The starting point of the idea is that programs in Haskell can be understood as providing definitions within some category, which we will call Hask. Categories comprise a collection of objects and a collection of morphisms which are mappings between objects. Categories come equipped with identity morphisms for every object and an associative composition operation for morphisms (see Wikipedia for a more complete, formal definition). For Hask, the objects are Haskell types, morphisms are functions in Haskell, identity morphisms are provided by the identity function, and composition is the usual function composition operation. For the purpose of this discussion we are not really concerned about the exact properties of Hask, just that Haskell acts as a kind of internal language for category theory, within some arbitrary category Hask (Dan Piponi provides some discussion on this topic).

Subcategories

Given some category C, a subcategory of C comprises a subcollection of the objects of C and a subcollection of the morphisms of C which map only between objects in the subcollection of this subcategory.

We can define for Hask a singleton subcategory for each type, which has just that one type as an object and functions from that type to itself as morphisms e.g. the Int-subcategory of Hask has one object, the Int type, and has functions of type Int → Int as morphisms. If this subcategory has all the morphisms Int → Int it is called a full subcategory. Is there a way to describe “larger” subcategories with more than just one object?

Via universal quantification we could define the trivial (“non-proper”) subcategory of Hask with objects of type a (implicitly universally quantified) and morphisms a -> b, which is just Hask again. Is there a way to describe “smaller” subcategories with fewer than all the objects, but more than one object? Yes. For this we use type classes.

Subcategories as type classes

The instances of a single parameter type class can be interpreted as describing the members of a set of types (or a relation on types for multi-parameter type classes). In a type signature, a universally quantified type variable constrained by a type class constraint represents a collection of types that are members of the class. E.g. for the Eq class, the following type signature describes a collection of types for which there are instances of Eq:

      Eq a => a

The members of Eq are a subcollection of the objects of Hask. Similarly, the type:

      (Eq a, Eq b) => (a -> b)

represents a subcollection of the morphisms of Hask mapping between objects in the subcollection of objects which are members of Eq. Thus, the Eq class defines an Eq-subcategory of Hask with the above subcollections of objects and morphisms.

Type classes can thus be interpreted as describing subcategories in Haskell. In a type signature, a type class constraint on a type variable thus specifies the subcategory which the type variable ranges over the objects of. We will go on to use the constraint kinds extension to define constraint-kinded type families, allowing structures from category theory to be parameterised by subcategories, encoded as type class constraints. We will use functors as the example in this post (more examples here).

Functors in Haskell

In category theory, a functor provides a mapping between categories e.g. F : C → D, mapping the objects and morphisms of C to objects and morphisms of D. Functors preserves identities and composition between the source and target category (see Wikipedia for more). An endofunctor is a functor where C and D are the same category.

The type constructor of a parametric data type in Haskell provides an object mapping from Hask to Hask e.g. given a data type data F a = ... the type constructor F maps objects (types) of Hask to other objects in Hask. A functor in Haskell is defined by a parametric data type, providing an object mapping, and an instance of the well-known Functor class for that data type:

class Functor f where
   fmap :: (a -> b) -> f a -> f b

which provides a mapping on morphisms, called fmap. There are many examples of functors in Haskell, for examples lists, where the fmap operation is the usual map operation, or the Maybe type. However, not all parametric data types are functors.

It is well-known that the Set data type in Haskell cannot be made an instance of the Functor class. The Data.Set library provides a map operation of type:

Set.map :: (Ord a, Ord b) => (a -> b) -> Set a -> Set b

The Ord constraint on the element types is due to the implementation of Set using balanced binary trees, thus elements must be comparable. Whilst the data type is declared polymorphic, the constructors and transformers of Set allow only elements of a type that is an instance of Ord.

Using Set.map to define an instance of the Functor class for Set causes a type error:

instance Functor Set where
   fmap = Data.Set.map

...

foo.lhs:4:14:
    No instances for (Ord b, Ord a)
      arising from a use of `Data.Set.map'
    In the expression: Data.Set.map
    In an equation for `fmap': fmap = Data.Set.map
    In the instance declaration for `Functor Set'

The type error occurs as the signature for fmap has no constraints, or the empty (always true) constraint, whereas Set.map has Ord constraints. A mismatch occurs and a type error is produced.

The type error is however well justified from a mathematical perspective.

Haskell functors are not functors, but endofunctors

First of all, the name Functor is a misnomer; the class actually describes endofunctors, that is functors which have the same category for their source and target. If we understand type class constraints as specifying a subcategory, then the lack of constraints on fmap means that Functor describes endofunctors HaskHask.

The Set data type is not an endofunctor; it is a functor which maps from the Ord-subcategory of Hask to Hask. Thus Set :: OrdHask. The class constraints on the element types in Set.map declare the subcategory of Set functor to which the morphisms belong.

Type class of exofunctors

Can we define a type class which captures functors that are not necessarily endofunctors, but may have distinct source and target categories? Yes, using an associated type family of kind Constraint.

The following ExoFunctor type class describes a functor from a subcategory of Hask to Hask:

{-# LANGUAGE ConstraintKinds #-}
{-# LANGUAGE TypeFamilies #-}

class ExoFunctor f where
   type SubCat f x :: Constraint
   fmap :: (SubCat f a, SubCat f b) => (a -> b) -> f a -> f b

The SubCat family defines the source subcategory for the functor, which depends on f. The target subcategory is just Hask, since f a and f b do not have any constraints.

We can now define the following instance for Set:

instance ExoFunctor Set where
   type SubCat Set x = Ord x
   fmap = Set.map

Endofunctors can also be made an instance of ExoFunctor using the empty constraint e.g.:

instance ExoFunctor [] where
    type SubCat [] a = ()
    fmap = map

(Aside: one might be wondering whether we should also have a way to restrict the target subcategory to something other than Hask here. By covariance we can always “cast” a functor C → D, where D is a subcategory of some other category E, to C → E without any problems. Thus, there is nothing to be gained from restricting the target to a subcategory, as it can always be reinterpreted as Hask.)

Conclusion (implementational restrictions = subcategories)

Subcategory constraints are needed when a data type is restricted in its polymorphism by its operations, usually because of some hidden implementational details that have permeated to the surface. These implementational details have until now been painful for Haskell programmers, and have threatened abstractions such as functors, monads, and comonads. Categorically, these implementational restrictions can be formulated succinctly with subcategories, for which there are corresponding structures of non-endofunctors, relative monads, and relative comonads. Until now there has been no succinct way to describe such structures in Haskell.

Using constraint kinds we can define associated type families, of kind Constraint, which allow abstract categorical structures, described via their operations as a type class, to be parameterised by subcategories on a per-instance basis. We can thus define a class of exofunctors, i.e. functors that are not necessarily endofunctors, which we showed here. The other related structures which are difficult to describe in Haskell without constraint kinds: relative monads and relative comonads, are discussed further in this draft note. The note includes examples of a Set monad and an unboxed array comonad, both of which expose their implementational restrictions as type class constraints which can be described as subcategory constraints.
Any feedback on this post or the draft note is greatly appreciated. Thanks.


by dorchard at October 18, 2011 09:07 AM

October 17, 2011

Ivan Lazar Miljenovic

graphviz in vacuum

During the past week, Conrad Parker announced on Google+ (though it wasn’t a public post so I can’t seem to link to it) that he had decided to take over maintainership (at least until someone else says they want to do it) of vacuum since Matt Morrow hasn’t been seen for a while.

I decided to take the opportunity to replace the current explicit String-based (well, actually Doc-based) mangling used to create Dot graphs for use with Graphviz from vacuum with usage of my graphviz library. I’ve just sent Conrad a pull request on his GitHub repo, and I decided that this would make a suitable “intro” tutorial on how to use graphviz.

First of all, have a look at the current implementation of the GHC.Vacuum.Pretty.Dot module. If you read through it, it’s pretty straight-forward: convert a graph in an adjacency-list format into Dot code by mapping a transformation function over it, then attach the required header and footer.

Note though that this way, the layout/printing aspects are mixed in with the actual conversion part: rather than separating the creation of the Dot code from how it actually appears, it’s all done together.

There’s also a mistake in there that probably isn’t obvious: the “arrowname=onormal” part of the definition of gStyle is completely useless: there is no such attribute as “arrowname”; what is probably meant there is “arrowhead” or “arrowtail”.

Let’s now consider the implementation using version 2999.12.* of graphviz (the version is important because even whilst doing this I spotted some changes I’m likely to make in the next version for usability purposes):

{-# LANGUAGE OverloadedStrings #-}

module GHC.Vacuum.Pretty.Dot (
   graphToDot
  ,graphToDotParams
  ,vacuumParams
) where

import Data.GraphViz hiding (graphToDot)
import Data.GraphViz.Attributes.Complete( Attribute(RankDir, Splines, FontName)
                                        , RankDir(FromLeft), EdgeType(SplineEdges))

import Control.Arrow(second)

------------------------------------------------

graphToDot :: (Ord a) => [(a, [a])] -> DotGraph a
graphToDot = graphToDotParams vacuumParams

graphToDotParams :: (Ord a, Ord cl) => GraphvizParams a () () cl l -> [(a, [a])] -> DotGraph a
graphToDotParams params nes = graphElemsToDot params ns es
  where
    ns = map (second $ const ()) nes

    es = concatMap mkEs nes
    mkEs (f,ts) = map (\t -> (f,t,())) ts

------------------------------------------------

vacuumParams :: GraphvizParams a () () () ()
vacuumParams = defaultParams { globalAttributes = gStyle }

gStyle :: [GlobalAttributes]
gStyle = [ GraphAttrs [RankDir FromLeft, Splines SplineEdges, FontName "courier"]
         , NodeAttrs  [textLabel "\\N", shape PlainText, fontColor Blue]
         , EdgeAttrs  [color Black, style dotted]
         ]

(The OverloadedStrings extension is needed for the FontName attribute.)

First of all, note that there is no mention or concept of the overall printing/structure of the Dot code: this is all done behind the scenes. It’s also simpler this way to choose custom attributes: Don Stewart’s vacuum-cairo package ends up copying all of this and extra functions from vacuum just to have different attributes; here, you merely need to provide a custom GraphvizParams value!

So let’s have a look more at what’s being done here. In graphToDotParams, the provided adjacency list representation [(a,[a])] is converted to explicit node and edge lists; the addition of () to each node/edge is because in many cases you would have some additional label attached to each node/edge, but for vacuum we don’t. There is a slight possible error in this, in that there may be nodes present in an edge list but not specified directly (e.g. [(1,[2])] doesn’t specify the “2″ node). However, Graphviz doesn’t require explicit listing of every node if it’s also present in an edge, and we’re not specifying custom attributes for each node, so it doesn’t matter. The actual grunt work of converting these node and edge lists is then done by graphElemsToDot in graphviz.

The type signature of graphToDotParams has been left loose enough so that if someone wants to specify clusters, it is possible to do so. However, by default, graphToDot uses the specified vacuumParams which have no clusters, no specific attributes for each node or edge but does have top-level global attributes. Rather than using Strings, we have a list of GlobalAttributes, with one entry for each of top-level graph, node and edge attributes (the latter two applying to every node/edge respectively). I’ve just converted over the attributes specified in the original (though dropping off the useless “arrowname” one). Some of these attributes have more user-friendly wrappers that are re-exported by Data.GraphViz; the other three need to be explicitly imported from the complete list of attributes (for these cases I prefer to do explicit named imports rather than importing the entire module so I know which actual attributes I’m using). I am adding more attributes to the “user-friendly” module all the time; RankDir will probably make it’s way over there for the next release, with a better name and documentation (and thus not requiring any more imports).

Now, you might be wondering how I’ve managed to avoid a (a -> String) or similar function like the original implementation had. That’s because the actual conversion uses the PrintDot class (which is going to have a nicer export location in the next version of graphviz). As such, as long as a type has an instance – and ones like String, Int, etc. all do – then it will be printed when the actual Dot code is created from the DotGraph a value.

So how to actually use this? In the original source, there’s a commented out function to produce a png image file. This is achieved by saving the Dot code to a file, then explicitly calling the dot command and saving the output as an image. Here’s the version using graphviz:

{-# LANGUAGE ScopedTypeVariables #-}

import GHC.Vacuum.Pretty.Dot
import Data.GraphViz.Exception

graphToDotPng :: FilePath -> [(String,[String])] -> IO Bool
graphToDotPng fpre g = handle (\(e::GraphvizException) -> return False)
                       $ addExtension (runGraphviz (graphToDot g)) Png fpre >> return True

(Note: The exception-handling stuff is just used to provide the same IO Bool result as the original.)

I hope you’ve seen how convenient graphviz can be rather than manually trying to produce Dot code and calling the Graphviz tools to visualise it. There are still some cludgy spots in the API (e.g. I would be more tempted now to have the graph to visualise be the last parameter; at the time I was considering more about using different image outputs for the same graph), so I appreciate people telling me how the API can be improved (including which attributes are commonly used).


Filed under: Graphs, Haskell

by Ivan Miljenovic at October 17, 2011 02:49 AM

October 16, 2011

David Peixotto

Building GHC with Clang

Intro

I recently tried compiling GHC using clang as a replacement for GCC. Clang is a new C/C++/Objective C front end that integrates well with the LLVM tool chain. It will probably eventually replace GCC as the goto compiler for most projects at Apple. It was designed for fast compilation time, good error messages, and freedom from the GPL.
I wanted to compile GHC with clang because it can generate LLVM bitcode directly from all of the C files in the GHC runtime. These files can be collected together at link time and optimized using LLVM’s link time optimization. Combined with the new LLVM backend for GHC this offers some interesting opportunities for optimizing Haskell source programs together with the runtime. The apple supplied linker has built-in support for link time optimization of LLVM bitcode files, so it might be possible to get an improved optimization level for very little effort.
Note that we could also use the llvm-gcc front end to generate bitcode from c source files. I wanted to give clang a try because it is better integrated with LLVM and looked easier to add custom optimization passes into the compilation pipeline.

Compiling GHC with clang

The compilation process was quite a bit more difficult than I had suspected. GHC uses a number of GCC specific extensions, and these happen to be some of the very extensions that clang does not support. I was eventually able to get a successful compilation, but I had to give up global registers, SMP, and a threaded runtime to make it work.
  1. Compile LLVM and clang.
  2. This was an easy step. Following the instructions on the respective websites worked just fine.
  3. Let the linker find the libLTO.dylib that corresponds to the LLVM version you used to compile clang.
  4. The latest version of clang generates bitcode that the apple shipped version of LLVM does not understand. The linker uses libLTO.dylib to turn the bitcode into object code and so we have to point the linker to the version corresponding to the latest LLVM. The easiest way to do this is to set the DYLD_LIBRARY_PATH environment variable.
    $ export DYLD_LIBRARY_PATH=$PATH_TO_LLVM/lib
  5. Configure GHC to use clang as the compiler and emit LLVM bitcode.
  6. This step is easily done using the configure script:
    $ CC=$HOME/llvm/bin/clang CFLAGS=-emit-llvm CPP="gcc -E" ./configure
    Notice that we still need to use GCC as the preprocessor. Clang does not support the traditional mode of the GCC preprocessor used by GHC. This mode is needed by GHC because some of the Haskell syntax clashes with ansi standard c preprocessor syntax. The preprocessor usage inside GHC turned out to be a fairly big impediment to compiling with clang for a variety of reasons.
  7. Change alocal.m4 to accept compiler version &gt;= 1.0
  8. There is a check in the configure script to make sure we are using GCC &gt;= 3.0. Since clang is only at 1.0, we have to modify this check to let us slip by.
    diff -rN old-ghc-DAVE/aclocal.m4 new-ghc-DAVE/aclocal.m4                        
    554c554
    FP_COMPARE_VERSIONS([$fp_cv_gcc_version], [-lt], [3.0],
    ---
    FP_COMPARE_VERSIONS([$fp_cv_gcc_version], [-lt], [1.0],
  9. Modify configure.ac for better detection of @progbits support.
  10. The configure script attempts to compile a bit of inline assembly that uses the progbits section. Mac OS X does not support this, so we want the configure script to detect that properly. The problem is that clang simply wraps the inline assembly in a string and sticks it in the bitcode. The compilation succeeds because it never gets beyond the bitcode stage. Changing the configure test to actually link the generated code allows the configure script to come to the right conclusion.
    diff -rN old-ghc-DAVE/configure.ac new-ghc-DAVE/configure.ac
    836c836
    AC_COMPILE_IFELSE(
    ---
    AC_LINK_IFELSE(
  11. Change Systools.lhs and DriverPipeline.hs to not rely on the C compiler for preprocessing.
  12. The current preprocessing assumes that it is safe to run $CC -E to do the preprocessing. This will work with clang as well, except in the case where the preprocessor needs to be used in traditional mode. I changed the code to have separate functions for those cases and explicitly use gcc -E for preprocessing in traditional mode.
  13. Patch clang to support the same macro expansion as GCC for macros containing ..

    There was a discrepancy in behavior between the clang preprocessor and gcc. GHC contains some macros that generate cmm code. The .. is a legal range operator in cmm and GCC leaves it after macro expansion, but clang was adding a space, which was causing the cmm parser in GHC to error out when parsing the code. See the example below.

    $ cat t.c
    #define X 0 .. 1

    0 .. 1 // Ok
    X // Bad: space between ..

    $ gcc -E t.c

    0 .. 1
    0 .. 1

    $ clang -E t.c

    0 .. 1
    0 . . 1


    Now, this is not really a bug in clang, since the preprocessor is meant for C code and .. is not a valid operator in C. However, it did not match the behavior of GCC and so it was causing problems for me. I submitted a patch to clang that makes the macro expansion behavior match GCC in this case.

  14. Patch the gmp configure script to work with clang.
  15. The configure script in gmp contains a test to see if the c compiler being used is ansi compliant. The test is faulty and causes configure to incorrectly report that clang is not an ansi compiler. The problem is the main function used in the compilation test looks like this:
    main (int argc, char *argv[])
    but the [] gets eaten by the m4 macro processor when expanding this function and clang barfs when it sees a main function that looks like this:
    main (int argc, char *argv)
    This bug has been reported elsewhere and fixed upstream, but not yet ported into the gmp version used by GHC.
  16. Do not use global registers or SMP.
  17. Clang does not support the global register extension of GCC so this must be disabled. Also, it is not possible to build the threaded runtime with SMP enabled if you can not keep the base register in a global register (see includes/stg/Regs.h). These extensions can be disabled by using a custom build.mk containing
    GhcRtsHcOpts +=-optc-DNO_GLOBAL_REG_DECLS -optc-DNOSMP
    GhcRtsCcOpts +=-DNO_GLOBAL_REG_DECLS -DNOSMP
    SRC_CC_OPTS +=-DNO_GLOBAL_REG_DECLS -DNOSMP
  18. Don’t build the threaded runtime.
  19. It turns out that clang does not support the __thread attribute used by GCC for thread local storage. This is normally not a problem (GCC does not support it on Mac OS X either), but for building the threaded runtime GHC either needs thread local storage (__thread) or the ability to pin global variables to registers. Since clang does not support either of those, we can not build the threaded runtime. This is easily disabled in build.mk using
    GhcRTSWays = v

Test Results

After getting GHC to compile using clang for the C compiler, it was time to run the tests.
OVERALL SUMMARY for test run started at Fri Apr  9 08:49:49 CDT 2010
2478 total tests, which gave rise to
7969 test cases, of which
0 caused framework failures
5792 were skipped

2054 expected passes
67 expected failures
0 unexpected passes
56 unexpected failures
The results are pretty decent for a first attempt with only 56 unexpected failures. In total 40 of the 56 failures occur when using the C back end. I didn’t spend a lot of time reviewing the failures, but of the two that I looked at, one failed because clang emitted an extra warning and the other failed using the C back end because the evil mangler failed to parse the assembly produced by clang. I suspect many of the failures in the C back end are caused by similar problems.

Conclusion

While this was an interesting exercise, I’d say using clang to compile GHC is not the best option at this point. I posted my detailed experience here in the hope that it may be of help to someone performing a similar exercise in the future.
In my view, the main impediment to using clang is the GCC specific code in the runtime. This code forces us to not be able to build the threaded runtime when compiling with clang because clang does not support global register declarations or the __thread attribute. Of these two, I would say GHC could probably get rid of using __thread in place of a more portable pthread solution for thread local data. That said, I’m not familiar enough with the runtime to say how difficult this would be and if it can be done without a loss of performance.
The other option for better integration with LLVM is to use the llvm-gcc front end. I’m not sure how it will deal with global register declarations, since I don’t think LLVM supports them either. Overall I’d say this is a step closer to enabling link time optimization in GHC through a combination of compiling the runtime to LLVM bitcode and using the LLVM back end to generate the bitcode for Haskell programs.

by David Peixotto (noreply@blogger.com) at October 16, 2011 06:47 PM

October 13, 2011

Bill Atkins

The Implicit Environment Pattern

In general, I'm pretty skeptical of the purported benefits of test-driven development. Nevertheless, there are cases where some unit tests can be extremely helpful. Recently, I wanted to write a simple command-line utility to analyze some sales data for me. The problem was simple and well-contained enough that I thought it would make sense to write a few unit tests.

One of the perennial problems with writing good unit tests is cleanly separating out the objects being tested from their dependencies and collaborators. Most discussions of this topic limit this to separating out references to global variables or singleton objects. They typically do this with dependency injection. Conventional dependency injection is a pretty good solution to this problem, but it's often an extralingual solution, relying on XML files that are kept separate from your source code and which fall outside the purview of the type system. Even if they avoid XML files (Guice comes to mind), they're fairly heavy systems, in that you have to bring in jar's, set up your configuration, and get familiar with the API.

And aside from simply avoiding hardcoded class references, there were some other kinds of dependencies that I wanted to isolate:

  • Environment information, such as where to find configuration files, server addresses and ports, or flags
  • Side effects, including I/O For example, println assumes the presence of a console and offers no way for you to compare what was printed to what you expected to print. Similarly, reading from the console assumes that someone is present to type in commands and provides no way for you to provide mock input.

I eventually hit on a solution I'm calling the Implicit Environment Pattern. It makes heavy use of Scala-specific features, like traits and implicits. Unlike most dependency injection solutions, this is simply a code pattern and requires no external jar's, dependencies or configuration languages.

The key principles of this pattern are:

  • Separate configuration, hardcoded class references and I/O operations into abstract traits
  • Define concrete implementations of these traits for different environments, e.g. production, testing, QA, development
  • Pass these implementations into their dependent classes via implicit parameters to the constructor
  • Dependencies are transitive - the implicit values passed into one class will be transparently passed into any other instances they create, as long as those classes declare their dependencies via implicit parameters

To start with, I'll create a number of traits to declare very fine-grained dependencies:

trait WritesOutput { 
def println(x: Any): Unit
}

trait ReadsSalesDirectory {
def salesDirectory: File
}

trait ReadsHistoricalRecords {
def historicalRecordsFile: File
}

trait NeedsConsoleInput {
def readLine(completionOptions: Iterable[String] = Nil): String
}

In this example, I've made the choice to treat the sales directory as a configuration parameter, so I can simply pass in a different directory in different environments. Depending on how much I/O is going on, I could also create a trait with a readFromFiles method that abstracts the I/O completely; in that case, my test implementations could just return an in-memory list from readFiles without touching the filesystem at all. Which of these paths you choose depends on how complicated the I/O is and on whether it's easier to provide test files or just create in-memory data structures. In this case, I have easy access to sample files, I'm pretty confident that basic line-reading code will work and I'm only reading data and not changing anything, so I've made this choice in the interest of pragmatism.

Now let's write a class that makes use of these traits via implicit parameters:

class SalesReportParser(val startingDay: LocalDate)(
implicit salesDir: ReadsSalesDirectory, writer: WritesOutput) {
...

def parseAll(): List[SalesRecord] = {
val result = withSource(Source.fromFile(salesDir.salesDirectory)) { source =>
...
}

writer.println("Parsed %d files.".format(result.count))
result
}
}

Clients of the class will pass in its dependencies transparently via implicits. So in my main method, I simply write:

object SalesReport {
class ProdEnv extends WritesToStdout
with ReadsSalesDirectory
with ReadsBooksAndRecords
with AcceptsInput {
def println(x: Any) = Predef.println(x)
val salesDirecotry = new File(...)
val historicalRecordsFile = new File(...)
def readLine(completionOptions: Iterable[String]) = {
...
}
}

def main(args: Array[String]) {
implicit val env: ProdEnv = new ProdEnv

val parser = new SalesReportParser(new LocalDate)
parser.parseAll()

...
}
}

(For convenience, one class implements all of the necessary traits. In tests, I might want to create separate implementations for individual dependencies.)

The implicit value env in main is transparently passed into SalesReportParser's constructor. Importantly, if SalesReportParser instantiates any classes that have their own dependencies declared as implicits, they'll be automatically passed in from the SalesReportParser instance.

If I fail to provide any of the parser's dependencies, the bug will be caught at compile time, since implicit resolution will fail when SalesReportParser's constructor is called. This is what I want: it's a beautiful thing when you're able to structure your code such that certain bugs are "inexpressible," i.e. the compiler will reject them outright before they can even become bugs.

So now I've isolated the console output dependency and sales directory path from my sales report parsing logic. Further, each class specifies exactly what dependencies it needs - the SalesReportParser can't read the historical records without explicitly adding ReadsHistoricalRecords as an implicit parameter, so readers of this code can immediately determine how a given class depends on and affects the world around it. In effect, the SalesReportParser class acts like a function in the functional programming sense - since I provide the class with the objects that will accept its input and emit its output, it need not affect the real world unless I provide it dependencies that do so.

But the other benefit is that my unit tests become extremely straightforward:

class TestTheParser extends TestCase {
class TestEnv extends ReadsSalesDirectory with WritesToStdout {
var writtenData: List[String] = Nil

def salesDirectory = new File("test/testSalesData")
def println(x: Any) = {
writtenData ::= x
}
}

val testDate = ...

def testMe = {
implicit val env = new TestEnv
val parser = new SalesReportParser(testDate)
parser.parseAll()

env.writtenData should_== List("Parsed 10 files.")
}
}

The Implicit Environment pattern also allows us to create a single TestEnv class to be used throughout all test cases and override its behavior only when necessary. For example, our TestEnv class will by default accumulate output into the writtenData array, but we may want to override this in certain cases:

class MyTest extends TestCase {
def testMe = {
implicit val testEnvCatchesPrinting = new TestEnv {
override def println(x) = fail("should not print to stdout")
}

val parser = new SalesReportParser
parser.parseAll() // any output here will cause the test to fail
}
}

Another advantage of this architecture - with strict separation of I/O into explicitly specified dependencies - is that it's less likely that test code will accidentally perform operations on production data. By isolating as much I/O as is practical into dependency traits, the only way the the two could intermingle would be by intentionally passing in a production implementation into test code, or vice-versa.

by More Indirection (noreply@blogger.com) at October 13, 2011 03:54 AM

October 12, 2011

Twan van Laarhoven

Finding rectangles, part 2: borders

In the previous post, we looked at finding axis aligned rectangles in a binary image. Today I am going to solve a variation of that problem:

Given a binary image, find the largest axis aligned rectangle with a 1 pixel wide border that consists entirely of foreground pixels.

Here is an example:
,
where white pixels are the background and blue is the foreground. The rectangle with the largest area is indicated in red.

Like the previous rectangle finding problem, this one also came up in my masters thesis. The application was to, given a scan of a book, find the part that is a page, cutting away clutter:
.

Specification

The types we are going to need are exactly the same as in my previous post:

-- An image is a 2D list of booleans, True is the foreground
type Image = [[Bool]]
-- An axis-aligned rectangle data Rect = Rect { left, top, width, height :: Int } deriving (Eq,Ord,Show)

The difference compared to last time is the contains function, which tells whether an image contains a given rectangle. We are now looking only at the borders of rectangles, or 'border rectangles' for short.

-- does an image contain a given border rectangle?
contains :: Image -> Rect -> Bool
contains im rect = isBorder (cropRect im rect)
-- crop an image to the pixels inside a given rectangle cropRect :: Image -> Rect -> Image cropRect im (Rect x y w h) = map cols (rows im) where rows = take h . drop y . (++repeat []) cols = take w . drop x . (++repeat False)
-- is the border of an image foreground? isBorder :: Image -> Bool isBorder im = and (head im) -- top border && and (last im) -- bottom border && and (map head im) -- left border && and (map last im) -- right border

Finding the largest border rectangle can again be done by enumerating all rectangles contained in the image, and picking the largest one:

largestRectspec :: Image -> Rect
largestRectspec = maximalRectBy area . allRects
allRects :: Image -> [Rect] allRects im = filter (im `contains`) rects where -- boring details omitted, see previous post

Just as before, this specification has runtime O(n6) for an n by n image, which is completely impractical.

An O(n4) algorithm

Unfortunately, the nice properties of maximal rectangles will not help us out this time. In particular, whenever a filled rectangle is contained in an image, then so are all smaller subrectangles So we could 'grow' filled rectangles one row or column at a time. This is no longer true for border rectangles.

We can, however, easily improve the above O(n6) algorithm to an O(n4) one by using the line endpoints. With those we can check if an image contains a rectangle in constant time. We just need to check all four of the sides:

-- pseudo code, not actually O(n^4) without O(1) array lookup
containsfast im (Rect x y w h)
    = r!!(x,y)     >= x+w -- top border
   && r!!(x,y+h-1) >= x+w -- bottom border
   && b!!(x,y)     >= y+h -- left border
   && b!!(x+w-1,y) >= y+h -- right border

Where r and b give the right and bottom endpoints of the horizontal and vertical lines through each pixel.
r = , b = .

An O(n3) algorithm

As the title of this section hints, a still more efficient algorithm is possible. The trick is to only look for rectangles with a specific height h. For any given height h, we will be able to find only maximal rectangles of that height.

For example, for h=6 we would expect to find these rectangles:
.
Notice how each of these rectangles consist of three parts: a left side, a middle and a right side:
.

The left and right parts both consist of a vertical line at least h pixels high. We can find those vertical lines by looking at the top (or bottom) line endpoints. The top endpoint for pixel (x,y+h-1) should be at most y,

let h = 6 -- for example
let av = zipWith2d (<=) (drop (h-1) t) y
-- in images:
av =  <= 
   = 

Each True pixel in av corresponds to a point where there is a h pixel high vertical line. So, a potential left or right side of a rectangle.

The middle part of each rectangle has both pixel (x,y) and (x,y+h-1) set,

let ah = zipWith2d (&&) a (drop (h-1) a)
-- in images:
ah =  && 
   = 

To find the rectangles of height h, we just need to find runs that start and end with a pixel in av, and where all pixels in between are in ah. First we find the left coordinates of the rectangles,

let leStep (av,ah,x) le
      | av = min le x  -- pixel in av ==> left part
      | ah = le        -- pixel in ah, but not av ==> continue middle part
      | otherwise = maxBound
let le = scanRightward leStep maxBound (zip2d3 av ah x)
le = 

Finally we need to look for right sides. These are again given by av. For each right side, le gives the leftmost left side, and h gives the height of the rectangles:

let mkRect x y av le
      | av = [Rect le y (x-le+1) h] -- pixel in av ==> right part
      | otherwise = []
let rects = zipWith2d4 mkRect x y av le
rects = 

Compare the resulting image to the one at the start of this section. We found the same rectangles.

Just like last time, all we need to do now is put the steps together in a function:

rectsWithHeight :: Int -> Image -> [Rect]
rectsWithHeight h a = concat . concat $ rects
  where
    x  = scanRightward (\_ x -> x + 1) (-1) a
    y  = scanDownward  (\_ y -> y + 1) (-1) a
    t  = scanDownward  (\(a,y) t -> if a then t else y+1) 0 (zip2d a y)
    ah = zipWith2d (&&) (drop (h-1) a) a
    av = zipWith2d (<=) (drop (h-1) t) y
    leStep (av,ah,x) le
      | av = min le x
      | ah = le
      | otherwise = maxBound
    le = scanRightward leStep maxBound (zip2d3 av ah x)
    mkRect x y av le
      | av = [Rect le y (x-le+1) h]
      | otherwise = []
    rects = zipWith2d4 mkRect x y av le

Of course, finding (a superset of) all maximal rectangles in an image is just a matter of calling rectsWithHeight for all possible heights.

findRectsfast :: Image -> [Rect]
findRectsfast im = concat [ rectsWithHeight h im | h <- [1..imHeight im] ]
largestRectfast :: Image -> Rect largestRectfast = maximalRectBy area . findRectsfast

Let's quickly check that this function does the same as the specification,

propfast_spec = forAll genImage $ \a -> largestRectspec a == largestRectfast a
> quickCheck propfast_spec
+++ OK, passed 100 tests.

Great.

Conclusions

The runtime of rectsWithHeight is linear in the number of pixels; and it is called n times for an n by n image. Therefore the total runtime of largestRectfast is O(n3). While this is much better than what we started with, it can still be quite slow. For example, the book page that motivated this problem is around 2000 pixels squared. Finding the largest rectangle takes on the order of 20003 = 8*109, or 8 giga-operations, which is still a pretty large number.

To make this algorithm faster in practice, I used a couple of tricks. Most importantly, if we know what function we are maximizing, say area, then we can stop as soon as we know that we can't possibly find a better rectangle. The idea is to start with h=imHeight im, and work downwards. Keep track of the area a of the largest rectangle. Then as soon as h * imWidth im < a, we can stop, because any rectangle we can find from then on will be smaller.

Is this the best we can do? No. I know an algorithm for finding all maximal border rectangles in O(n2*(log n)2) time. But it is rather complicated, and this post is long enough already. So I will save it for another time. If anyone thinks they can come up with such an algorithm themselves, I would love to read about it in the comments.

October 12, 2011 07:57 AM

October 11, 2011

Brandon Simmons

simple-actors 0.1.0 released

Just a quick announcement of the release of simple-actors, a DSL-style haskell library for more structured concurrent programs via the Actor Model. You can fork it on github, and install it via cabal with a

cabal install simple-actors

This version is significantly more coherent, powerful and simple than the original 0.0.1 version (which was something I just whipped together for a blog post I haven’t gotten around to writing yet).

Let me know if you have any comments or suggestions!

by jberryman at October 11, 2011 06:02 PM

apfelmus

FRP - How to deal with concurrent external events?

Kevin Cantu has kindly reported several bugs for my reactive-banana library. One of them is a particularly vexing issue about external events that occur concurrently. I’m a bit stumped on the question of choosing right semantics, so I’m writing this blog post to spark a small discussion; feedback of any kind is welcome.

UPDATE: Many thanks to everyone for discussion this issue, I have found a solution that I’m happy with!

Problem description

The problem itself is orthogonal to functional reactive programming, so let us imagine some generic, imperative event-based framework, for instance a GUI framework like wxHaskell or Gtk2Hs. You may also imagine Node.js, with the reservation that I know almost nothing about it.

The main question is: how do you deal with events that occur while you are currently handling another event? More concretely, consider the following minimal example program written using wxHaskell:

<script src="https://gist.github.com/1278515.js"> </script>

(Here a link to the source code in case it does not display.)

Screenshot

Screenshot

It’s quite long for a minimal example, so let me explain. The two main widgets on the screen are a button and a text entry box, which has the focus. When you click the button, the text entry box will be disabled. However, this will also cause it to lose focus, which will trigger a corresponding “lose focus” event.

The question is: when do you handle the “lose focus” event? If you handle it while the text entry is being disabled, you may violate internal invariants. This is exemplified by the variable invariant, which is set to False while the button click is handled. Clearly, this means that handling a button click is not atomic. (In the case of the reactive-banana library, this would screw up library internals; hence the bug report.)

In short: reacting to an event while handling another one may expose internal invariants.

But if you queue the “lose focus” event and handle it after the button click, you may violate external invariants. In this case, the external invariant is that a disabled text field cannot generate “lose focus” events; after all, that would be silly.

In short: reacting to an event after finishing another one may render it “impossible”, i.e. it should not have happened in the first place.

(A useful idea for reactive-banana would be to make the “lose focus” event happen simultaneously with the button click, but we’ve already performed too much IO for this to be viable. Once a single side effect is executed, there’s no way back.)

Solutions?

Do you, dear reader, have any ideas of how to resolve this problem?

I’m currently stumped on this. A small consolation is that the example demonstrates several interesting things:

  • This problem is about atomicity, an issue well-known in concurrent programming. Strangely, the code here is entirely single-threaded!
  • Once again, side effects prove to be nasty. If the whole GUI framework were somehow presented in FRP, then it would be no problem to make the two events happen simultaneously.
  • This is probably a surreptitiously subtle source of bugs in existing GUI applications.

Flattr this

October 11, 2011 04:30 PM

Chung-chieh Shan

Why not covariant generics?

Many people are riled up that Dart’s type system proudly features covariant generics and so “is unsound”. The basic problem with covariant generics is that it lets you take a bowl of apples and call it a bowl of fruit. That’s fine if you’re just going to eat from the bowl and you like all fruits, but what if you decide to contribute a banana to the fruit bowl? Someone taking from the bowl expecting an apple might choke. And someone else could call the same bowl a bowl of things then put a pencil in it, just to spite you.

… because they are unsound?

We all seem to agree that it’s bad for an apple eater to choke on a banana or a fruit eater to choke on a pencil. (Wittgenstein and Turing similarly agreed that it’s bad for a bridge to fall and kill people.) For example, Java seems to agree: it detects at run time when such choking is about to occur and throws an exception instead, so no immediate harm is done. There is less agreement as to whether a static type system must prevent such choking before run time. One argument for early prevention is that it’s the whole point of having a static type system—why even bother with a static type system if it’s unsound?

But plenty of static type systems are unsound yet respectable. For example, many dependent type systems are unsound, either because their term languages are blatantly non-terminating or because they include the Type:Type rule. (I faintly recall that Conor McBride calls Type:Type “radical impredicativity”.) So it doesn’t seem that unsoundness per se should rile people up about Dart in particular.

Perhaps it’s time to distinguish different sorts of unsoundness: a type system can be unsound because it fails normalization or because it fails subject reduction. In the former case, a program (or the type checker) might promise an apple but never produce one; in the latter case, the program might promise an apple then produce a banana. The unsoundness of a dependent type system tends to be of the former sort, whereas the unsoundness of covariant generics seems to be of the latter sort. And the former sort of unsoundness seems more benign than the latter sort (and more deserving of the term “radical impredicativity”). So, by calling for subject reduction but not normalization, it seems we can drive a wedge between the acceptable unsoundness of many dependent type systems and the unacceptable unsoundness of Dart.

But it is very easy to turn a failure of subject reduction detected at run time into a failure of normalization: just go into an infinite loop whenever choking is about to occur! In Java, we can write catch (Exception e) { for (;;) {} }. (In math, we can restore subject reduction by letting an uncaught exception take any type, as is standard.) Is that really better? Does this move really answer the critics? I don’t feel so. For one thing, this move makes it harder, not easier, to write better programs: I want my program to stop and tell me when it’s choking.

Just as it’s only useful for type systems to allow a program if it’s something good that a programmer might write, it’s only useful for type systems to disallow a program if it’s something bad that a programmer might write. In other words, both incompleteness and unsoundness “may appear to be a steep price to pay” yet “work well in practice”. Does Type:Type abet many bad programs that people produce? Perhaps not. Do covariant generics abet many bad programs that people produce? Perhaps not. Are untyped bowls and naive set theory terribly dangerous? Perhaps not. (Proof rules and programming facilities that are unsound may even encourage useful abstraction and reduce the proliferation of bugs by copy-and-paste.)

… because they are meaningless?

To summarize so far, I don’t think we should dislike covariant generics for its unsoundness in general or for its failure of subject reduction in particular. Why do covariant generics bother me, then? I think it’s because they dash my hope for types to have crisp meanings.

When I write down an expression, whether it’s a term or a type, I want it to refer to something else, whether in my mind or in the world. When I write that a certain term has a certain type (say, alice:person or 2+3:nat), I want to write about some entity, to mean that it has some property. That is not subject reduction, but to have a subject and a predicate in the first place, which is prior to subject reduction. I would be dissatisfied if there is no subject and predicate to be found outside the syntax and operational semantics of a formal language. After all, my desire to assert that 2+3:nat is not fueled by any interest in the curly symbols 2 and 3 or the binary node label +. In short, I want to denote.

Of course, it’s not entirely a formal language’s fault if I can’t use it to denote, if it’s hard to program in. Syntax only goes so far, and cognition must carry the rest of the way. But it does get easier to denote and program if how expressions combine inside the language matches how their referents tend to interact outside the language. For example, FORTRAN made it easier to denote numbers than in assembly language: having written 2+3 to denote a number, and knowing that numbers have absolute values, I can write abs(2+3) to denote the absolute value of the number denoted by 2+3. Numbers are outside FORTRAN, yet FORTRAN matches my knowledge that numbers have absolute values. In contrast, if you can hardly put a banana in a bowl of apples in reality but you can easily put(banana, bowl:Bowl<Apple>) in formality, or vice versa, then I have trouble seeing what banana and bowl mean.

This match I want is a kind of modular reasoning. Now, the word “modular” has been used to mean many things, but what I mean here is this. To combine two objects inside a language, I might need to worry about whether their types match, whether they live on the same machine, whether they use the same calling convention, etc. To combine two objects outside a language, I might need to worry about how big they are, whose kitchen it is, and whether fruit flies would come. Ideally, my worries inside the language should match my worries outside the language. Ideally, a combination should make sense inside a language just in case the referents of the combined expressions are compatible outside the language. If the language implementation could keep track of my worries and discharge them as early as possible for me, then so much the better, but that’s secondary and orthogonal—I’d be plenty happy if I can just talk about worrisome things without also worrying about the language at the same time.

I feel it’s hopeless to match a language with covariant generics against things we want to talk about. Why? I don’t have proof. For all I can show, the worries of a Dart programmer inside and outside the language will match perfectly, just as Java exceptions precisely simulate the material effects of putting a pencil in a bowl of apples. In the end, my feeling boils down to the unreasonable and unusual effectiveness of mathematics and logic.

Not only might this effectiveness be fallible, but it’s also possible that my hopelessness is misplaced. I said I want expressions to match their referents more closely so that denoting and programming becomes easier. But maybe the match we have today is the best we can do because the world has too many shades of gray to be modeled in the black-and-white terms provided by mathematics and logic. Maybe the world simply refuses to provide crisp referents for any expression to mean. Maybe, when lexical references slip and uncaught exceptions fire, it’s just collateral damage from the fact that life is messy. After all, it is one thing for a bridge engineer to leave gravity unquantified because calculus is hard; it is another thing for a bridge architect to leave beauty unquantified because psychology is hard. To avoid variance rules just because they “fly in the face of programmer intuition” feels like the former to me, but what do I know?

… because they are not even unsound?

Well, I do my best. I do my best to understand whether and how reference and truth can help programming. I do my best to enlist syntax and pragmatics. I’m not yet ready to give up just because so many people are confused by noun-noun compounds in English (what’s an “apple bowl”?) and by variance rules in Java today. To me, the notion of being sound or unsound only makes sense if expressions have meanings outside the language. I do my best so that Bowl<Apple> means something, so that when I claim “it is easy for tools to provide a sound type analysis if they choose” I know what I mean by “sound”. To not even worry about what expressions mean is not even unsound.

(Rob Simmons beat me to a blog post. He makes good, related points.)

October 11, 2011 02:56 AM

October 10, 2011

Chris Smith

Build Your Own Simple Random Numbers

Liam O’Connor got me thinking about the best way to explain the idea of a pseudo-random number generator to new programmers.  This post is my answer.  If you already understand them, there won’t be anything terribly new here.  That said, I enjoy clean examples even for easy ideas, so if you do too, then read on!

Note: The title may have caused some confusion.  I’m not suggesting you use the trivial algorithms provided here for any purpose.  Indeed, they are intentionally over-simplified to make them more understandable.  You should read this as an explanation of the idea of how generating random numbers works, and then use the random number generators offered by your operating system or your programming language, which are far better than what’s provided here.

The Problem

Suppose you’re writing a puzzle game, and you need to choose a correct answer.  Or suppose you are writing a role-playing game, and need to decide if the knight’s attack hits the dragon or deflects off of its scales.  Or you’re writing a tetris game, and you need to decide what shape is going to come next.  In all three of these situations, what you really want is a random number.  Random numbers aren’t the result of any formula or calculation; they are completely up to chance.

Well, here’s the sad truth of the matter: computers can’t do that.  Yes, that’s right.  Picking random numbers is one of those tasks that confound even the most powerful of computers.  Why?  Because computers are calculating machines, and we just said that random numbers aren’t the result of any calculation!

Of course, you’ve probably played games on a computer before that seem to pick numbers at random, so you may not believe me.  What you’re seeing, though, aren’t really random numbers at all, but rather pseudo-random numbers.  Pseudo-random numbers are actually the result of a mathematical formula, but one designed to be so complicated that it would be hard to recognize any pattern in its results!

Writing a Pseudo Random Number Generator

A lot of smart people actually spend a lot of time on good ways to pick pseudo-random numbers.  They try a bunch of different complicated formulas, and try to make sure that patterns don’t pop up.  But we can build a simple one pretty easily to pick pseudo-random numbers from 1 to 10.  Here it is, in the programming language Haskell:

random i = 7 * i `mod` 11

Since it’s a function, it needs to have an input.  It then multiplies that input by 7, and then finds the remainder when dividing by 11.  We’ll give it the previous number it picked as input, and it will give us back the next one.  Suppose we start at 1.  Then we get the following:

random  1 ->  7
random  7 ->  5
random  5 ->  2
random  2 ->  3
random  3 -> 10
random 10 ->  4
random  4 ->  6
random  6 ->  9
random  9 ->  8
random  8 ->  1

Let’s look at the range of answers.  Since the answer is always a remainder when dividing by 11, it’ll be somewhere between 0 and 10.  But it should be pretty easy to convince ourselves that if the number we give as input is between 1 and 10, then 0 isn’t a possibile answer: if it were, then we’d have found two numbers, both less than 11, that multiply together to give us a multiple of 11.  That’s impossible because…. 11 is prime.  So we’re guaranteed that this process picks numbers between 1 and 10.  It seems to pick them in a non-obvious order with no really obvious patterns, so that’s good.  We appear to have at least a good start on generating random numbers.

Notice a couple things:

  • We had to pick somewhere to start.  In this case, we started out by giving an input of 1.  That’s called the seed.  If you use the same seed, you’ll always get the exact same numbers back!  Why?  Because it’s really just a complicated math problem, so if you do the same calculation with the same numbers, you’ll get the same result.
  • To get the next number, we have to remember something (in our case, the last answer) from the previous time.  That’s called the state.  The state is important, because it’s what makes the process give you different answers each time!  If you didn’t remember something from the last time around, then you’d again be doing the same math problem with the same numbers, so you’d get the same answer.

Doing Better By Separating State

Unfortunately, our random number generator has a weakness: you can always predict what’s coming next, based on what came before.  If you write tetris using the random number generator from earlier, your player will soon discover that after a line, they always get an L shape, and so on.  What you really want is for your game to occasionally send them a line followed by a T, or even pick two lines in a row from time to time!

How do we do this?  Well, the next answer that’s coming depends on the state, so our mistake before was to use the previous answer as the state.  The solution is to use a state that’s bigger than the answer.   We’ll still be looking for random numbers from 1 to 10, but let’s modify the previous random number generator to remember a bigger state.  Now, since state and answer are different things, our random function will have two results: a new state, and an answer for this number.

random i = (j, ans)
    where j   = 7 * i  `mod` 101
          ans = (j - 1) `mod` 10 + 1  -- just the ones place, but 0 means 10

That says take the input, multiply by 7, and find the remainder mod 101.  Since 101 is still prime, this will always give answers from 1 to 100.  But what we really wanted was a number from 1 to 10, just like the one we had before.  That’s fine: we’ll just take the ones place (which is between 0 and 9) and treat 0 as 10.  The tens place doesn’t really change the answer at all, but we keep it around to pass back in the next time as state.

Let’s see how this works:

random  1 -> ( 7,  7)
random  7 -> (49,  9)
random 49 -> (40, 10)
random 40 -> (78,  8)
random 78 -> (41,  1)
random 41 -> (85,  5)
random 85 -> (90, 10)
random 90 -> (24,  4)
random 24 -> (67,  7)
random 67 -> (65,  5)
random 65 -> (51,  1)

Excellent!  Now instead of going in a fixed rotation, some numbers are picked several times, and some haven’t been picked yet at all (but they will be, if we keep going), and you can no longer guess what’s coming next just based on the last number you saw.  In this random number generator, the seed was still 1, and the state was a number from 1 to 100.

People who are really interested in good random numbers sometimes talk about the period of a pseudo-random number generator.  The period is how many numbers it picks before it starts over again and gives you back the same sequence.  Our first try had a period of 10, which is rather poor.  Our second try did much better: the period was 100.  That’s still pretty far off, though, from the random number generators in most computers, the period of which can be in the millions or billions.

Real World Pseudo-Random Number Generators

Our two toy pseudo-random number generators were fun, but you wouldn’t use them in real programs.  That’s because operating systems and programming languages already have plenty of ways to generate pseudo-random numbers.  And those were created by people who probably have more time to think about random numbers than you do!  But some of the same ideas come up there.  For example, consider this (specialized) type signature for the random function in the Haskell programming language:

random :: StdGen -> (Int, StdGen)

Look familiar?  StdGen is the state, and choosing a random Int gives you back the Int, and a new StdGen that you can use to get more pseudo-random numbers!  Many programming languages, including Haskell, also have “global” random number generators that remember their state automatically (in Haskell, that is called randomIO), but under the covers, it all comes down to functions like the ones we’ve written here… except a lot more complex.

Where To Get a Seed

We’ve still left one question unanswered: where does the seed come from?  So far, we’ve always been using 1 for the seed, but that means that each time the program runs, it will get the same numbers back.  So we end up with a similar situation to what we saw before, where players will realize that a game starts with the same sequence of random events each time.

To solve this problem, the seed should come from somewhere that won’t be the same each time.  Here are two different ways to seed a random number generator.

  1. Mostly, pseudo-random number generators are seeded from a clock.  Imagine if you looked at the second hand on a clock, used it to get a number from 1 to 60, and used that for your seed.  Then the game would only act the same if it started at the same number of seconds.  Even better, you could take the number of seconds since some fixed time in the past, so you’d get an even bigger difference in seeds.  (Entirely by coincidence, computers often use the number of seconds since January 1, 1970.)
  2. You might try to get a good seed from details of the way the user uses your program.  For example, you can look at the exact place the user first clicks the mouse, or exactly how much time passes between pressing keys.  They will most likely not be exact, and click a few pixels off or type ever so slightly slower, even if they are trying to do exactly the same thing.  So, again, you get a program that acts differently each time.  This is called using entropy.

Most of the time, using the computer’s built-in clock is okay.  But suppose you’re making up a code word.  It would be very bad if someone could guess your code word just by knowing when you picked it!  (They would also need to know how your computer or programming language picks random numbers, but that’s not normally kept secret; they can probably find that out pretty easily.)  Computer security and privacy often depends on picking unpredictable random numbers — ones that people snooping on you won’t be able to guess.  In that case, it’s important that you use some kind of entropy, and not just the clock.  In fact, when security is at stake, you can use entropy to modify the state as well, to make sure things don’t get too predictable.  Most operating systems have special ways of getting “secure” random numbers that handle this for you.

Another example of entropy:

If you play the game Dragon Warrior for the Nintendo, but use an emulator instead of a real Nintendo, then you can save a snapshot of your game before you fight a monster, memorize what the monsters are going to do, and figure out exactly the right way to respond.  When you load the game from the snapshot and try again, as long as you do the same things, the monster will respond in exactly the same way!  That’s because the snapshot saves the state of the random number generator, so when you go back and load from the snapshot, the computer picks the same random numbers.  So if a fight against a monster is going well but you make a disastrous move at the end, you can load your snapshot and repeat the exact same fight up to that point.

The same trick doesn’t work in Dragon Warrior 2 (or later ones), though!  Why not?  Because the company that makes the game started using entropy in their sequel.  So now little things like exactly how long you wait between pressing buttons will change the game.  Since you can’t possibly time everything exactly the same down to hundredths or thousandths of a second, the task is hopeless, and you have to just take your chances and trust to luck.

So as you can see, random numbers can become a very tricky topic.  But ultimately it’s all just a complicated formula, a seed, and a state.


by cdsmith at October 10, 2011 10:08 PM

Edwin Brady

Idris, Mk II

About a month ago, I started working on a complete reimplementation of Idris, because there are a number of things about the current implementation that have been bothering me for a while. It’s now got to a point where it’s beginning to work (in particular, the standard well-typed interpreter is up and running). You can start playing with it now, and follow progress at the github repository.

I’m usually very reluctant to start something again – especially so with Idris since I’ve been working on it for a few years now. But there’s quite a few problems with the current design. In fact, “design” is too strong a word – I didn’t really know when I started what I wanted the language to do and which features I’d need. Moreover, it’s evolved out of a theorem proving system Ivor that was originally meant to do something else, and it’s getting harder and harder to add the new features I want (e.g. mod cons like where clauses, destructuring let, case, and essential gadgets for developing large systems like separate compilation and a decent module system) without having to jump through hoops.

Pleasingly, the new version is not far off catching up with where the old version was. Basic syntax, type checking with implicit syntax and evaluation are all working (mostly better than before…) There is no with rule yet, but it won’t be long. There’ll be a compiler not long after I’ve decided how to handle IO (I want to keep the IO primitives as primitive as I can so that it will be possible to experiment with different high level representations).

The main, significant, distinction between the previous implementation and this is the type checker. In the new version, the type checker is kept extremely simple – no implicit syntax, no unification, no generating proof obligations or coercions. Instead, we have a core type theory with metavariables, and an EDSL (that is, an embedded domain specific language) called Elab which constructs programs using Coq-style tactics, directed by the high level source text, and filling in metavriables by unification where it can. This idea will be unsurprising to anyone who is following the progress of Epigram.

Making Elab an EDSL has several advantages. Adding a new language construct (e.g. pattern matching, now, or maybe coercions, or type classes, some day…) is simply a matter of writing a sequence of tactics to build that construct in the core type theory. To check a pattern match clause, for example, we elaborate the left hand side inferring types for the pattern variables as we go, and use this information to elaborate the right hand side. Another advantage is that it will be easy to expose the EDSL to an Idris programmer for building domain specific decision procedures (e.g. a Ring Solver would be nice). Finally, making the elaborator do all of the work and keeping the type checker simple means that we can be far more confident that only well-typed programs are accepted.

I’ll go back into my hacking shed now, and I’m staying there until all the programs that worked with the previous version compile. Hopefully this won’t take long :) . In the meantime, if you have a go, please let me know what you come up with. In particular, if anything behaves badly, please shout – I want this version to work well!


by edwinb at October 10, 2011 08:43 PM

Alp Mestanogullari

A French community for Haskell

During the past few months, there has been some growing activity in the (hidden) French circles of the Haskell community. A few people, including myself, are trying to increase the interest around Haskell in the French-speaking communities. You may (or may not) have noticed the translation of Learn You A Haskell that is available here, in French. We also have an IRC channel, #haskell-fr (access it through Freenode’s webchat), and a mailing list. Please let us know about you if you are a French-speaking haskeller!

But we would like to get it a step further. We are currently considering the idea of a French hackathon. We already have been offered a room for 3 days in June 2012, in conjunction with a French Perl event. So one of the reasons behind this blog post is to get some feedback about this, in addition to letting people know about the French community. Who would be interested in attending such an event? This would be in Strasbourg, France for the moment. Note that this would not necessarily be restricted to French haskellers! The format would be a pretty much classical Hackathon: potential talks and a lot of Haskell hacking. So please let us know if you’d be interested in attending this (by commenting this post, the IRC channel or the mailing list).

On a side note, these past few days, I updated one of my libraries: statistics-linreg and published one too: kmeans-vector. It performs the k-means clustering algorithm on a list of points, and can lead to “pretty” graphics like the following.

Performing k-means on 10,000 points with 5 clusters


by alpmestan at October 10, 2011 05:10 PM

factis research

stm-stats: Retry statistics for STM transaction

Two days ago, Stefan Wehr talked at the Haskell in Leipzig workshop about the experiences that factis research had with Haskell when creating real-world applications. When he mentioned that we have code that keeps count of retries of STM transaction, someone from the audience asked us to publish the code. So, after some refactoring to make the interface generally usable, here it is. The stm-stats package provides a few functions (in Control.Concurrent.STM.Stats) that can replace atomically: trackSTM and variants that allow you to name the transaction or have more control about the warnings that will be emitted when the number of retries exceeds a certain value. The following code demonstrates how the module works:
import Control.Concurrent
import Control.Concurrent.STM
import Control.Monad

import Control.Concurrent.STM.Stats

main = do
var <- trackSTM $ newTVar 0
forkIO $ forM_ [1..23] $ \i -> do
    threadDelay (100*1000)
    trackNamedSTM "writer" $ writeTVar var i
putStrLn "Starting reader..."
trackNamedSTM "reader" $ do
    i <- readTVar var
    when (i < 23) retry
putStrLn "Reader finished."
dumpSTMStats
When run, you will see this output:
Starting reader...
STM transaction reader finished after 23 retries
Reader finished.
STM transaction statistics (2011-10-09 16:26:27.226675 UTC):
Transaction     Commits    Retries      Ratio
_anonymous_           1          0       0.00
reader                1         23      23.00
writer               23          0       0.00
PS: As I usually post on my own blog, I should explain why I from now on will also post on the factis research company blog. Factis research relies heavily on Free Software and wants to contribute back to the community where possible. But in the course of every-day work, this sometimes falls by the wayside. Therefore, I was hired to help out as the community interface: My task is identifying, packaging and uploading components of internal code that are of wider interest, following up on user requests and bug reports, and talk about it. This module is the first brought to you by this new strategy, but expect more to come.

by Joachim Breitner (noreply@blogger.com) at October 10, 2011 04:11 PM

Chris Smith

Haskell For Kids: Week 6

(This is from last week.  Yes, I’m really late in posting this… sorry!  This week’s summary will be up in a couple days.)

Practice Time

Welcome to the week 6 summary of Haskell for Kids, my middle school level (that is, ages 12 – 13 ish) programming class using Haskell and Gloss at the Little School on Vermijo.  You can go back and review weeks 1, 2, 3, 4, and 5 if you like.

This week was time for practice.  Instead of introducing any new ideas, I’ll just post the practice activity we did together in class on Tuesday.  The way we practiced was very simple: I drew pictures, and we tried to write descriptions of those pictures.  Rather than waste time, let’s just jump right into it.  It might help to download (and maybe print out) the PDF file from the end of week 5 (follow the link earlier) for a reminder of how to write list comprehensions.

I’m not including answers, because it’s too easy to just read ahead!  If you can’t figure one of them out, feel free to ask for help in the comments.

Practice 1

Hints: None!  Hopefully this one is easy.

Practice 2

Hints: The parameter to circle is a number.  Use the variable from the list comprehension for that.

Practice 3

Hints: This one should also be easy.  It’s just a stepping stone to the next practice.

Practice 4

Hint: The list comprehension variable tells you how far to translate.

Practice 5

Hint: Don’t do this with rotate.  You can change just one thing from practice 4 and make this work.  Remember that you can use the list comprehension variable more than once.

Practice 6

Hint: Remember that if you translate something first and then rotate it, it will be rotated around the middle of the picture.

Practice 7

Hint: Here you’ll want to use two variables in your list comprehension.  One of the examples from the reference page shows you how to do this.

Practice 8

Hint: Again you want two variables, but this time one of them will be used in translate and the other one in rotate.

Projects

In addition to this, we just worked on our list comprehension projects.  If you remember from last week, the project was to pick something repetitive and draw it.  I suggested a U.S. flag if you’re in the U.S. and don’t have another idea.  Some of the other projects kids from our class came up with were an iPhone with a grid of icons, a keyboard (the musical type!), and a galaxy with spiral arms and a background of stars.  So be creative!

Oh, and I have to show off Sophia’s keyboard, because she worked really, really hard on it, and I’m very impressed.  It’s a great use of list comprehensions, too.  Here goes…


by cdsmith at October 10, 2011 02:04 AM

October 08, 2011

Stefan Wehr

Talk about developing commercial software in Haskell

Yesterday, I gave a talk at the Haskell in Leizpig workshop about our experience in developing commercial software in Haskell. You can find the slides (in german) here. I was really surprised by the large number of attendees (about 50), given that the workshop's language was german (except the very interesting talk by Kevin Hammond).

by Stefan Wehr (noreply@blogger.com) at October 08, 2011 09:25 PM

October 06, 2011

David Terei

Stanford Haskell Course

This semester there is an exciting new course being taught at Stanford University, CS 240h: Functional Systems in Haskell. It's being being taught by both David Mazières and Bryan O'Sullivan. I'm TA for the course though so it isn't all sunshine and rainbows.


If having monads described as: A programmable semicolon is more intuitive then talking about how they: generalize closure operators on partially ordered sets to arbitrary categories, this is the course for you! We won't be covering semantics of programming languages. We won't be covering abstract math. We will be covering how to effectively use and understand Haskell as a systems language.


The course home page can be found here and Bryan maintains a Github mirror of all the material here. There are no video recordings of the lecture at this point as we aren't in a room with the facilities unfortunately. If the course is a success and continues in the future we will look to get recordings or maybe something like the Stanford Engineering Everywhere project.

by David Terei (noreply@blogger.com) at October 06, 2011 10:29 PM

October 05, 2011

Bryce Verdier

Programming Praxis: The Sum of Two Integers

A couple of months ago the Programming Praxis website put up a challenge to find a sum inside an array of integers (the direct wording of the challenge can be found here) and since I’ve come up with my own solution, this little challenge has provided me with a lot of feedback.

Just to get some of the geeky stuff out of the way, here is the code I wrote for the problem:

  1. import Data.List
  2. import Data.Maybe
  3.  
  4. sumCheck :: Int -> [Int] -> [Int] -> Maybe (Int, Int)
  5. sumCheck _ [] _ = Nothing
  6. sumCheck total (x:xs) ys = if total' == Nothing
  7. then sumCheck total xs ys
  8. else return (x, (ys !! ( fromJust total')))
  9. where total' = (total - x) `elemIndex` ys

In thinking about the problem a little bit I came up with this subtraction approach. My first approach was to use addition and add every item in the array against all the other items. But this method didn’t sit well with me. After a little bike ride I came up with the code you see above.

After I wrote it, I submitted my code to the Haskell-beginners email list asking for critiques and possible enhancements. Arlen Cuss contributed a slight improvement of my code:

  1. sumCheck total (x:xs) ys =
  2. let diff = total - x
  3. in if diff `elem` ys
  4. then Just (x, diff)
  5. else sumCheck total xs ys

And Adityz Siram contributed his version. Which is basically the first algorithm that I came up with and wanted to improve upon. His code is here:
  1. sums i as bs = [(x,y) | x <- as, y <- bs, x + y == i]

Finally, Gary Klindt took all of our code snippets, used some performance analysis tools inside GHC and came up with some run times that are (hopefully) more accurate than running time on an application. Here are those stats:
print $ sumCheck 500 [1..1000] [1..1000]
sumCheck1: 58,648
sumCheck2: 58,484
sumCheck3: 70,016

print $ sumCheck 5000 [1..10000] [1..10000]
sumCheck1: 238,668
sumCheck2: 238,504
sumCheck3: 358,016
(unit: byte)

Out of the three code snippets, my function was in the middle, speed-wise. But I think that it’s also really nice to see how much better it is than the regular addition method. It’s also nice to see how the little change made to my code can improve the overall speed of the function.

At the end of the day I take a little bit of pride in myself for coming up with an improved algorithm for this task on my own. I know that on a hardware level, subtraction takes more time than addition. But I get the improvements I get because I reduce the number of additions and comparisons I have to make in order for the function to be complete. I also estimate the worst case speed for my algorithm to be O(n), which isn’t too shabby.

When I started learning Haskell, one of the things I read on the internet was how the people who programmed it were helpful to one another. I was skeptical when I first read that, but I have to say that all of my doubt has been removed. And it is interactions like this that make me glad to participate in a community as helpful as this one.

by Bryce at October 05, 2011 04:12 PM

October 04, 2011

Shin-Cheng Mu

Constructing list homomorphisms from proofs

Yun-Yan Chi and Shin-Cheng Mu. In the 9th Asian Symposium on Programming Languages and Systems (APLAS 2011). [PDF]

The well-known third list homomorphism theorem states that if a function h is both an instance of foldr and foldl, it is a list homomorphism. Plenty of previous works devoted to constructing list homomorphisms, however, overlook the fact that proving h is both a foldr and a foldl is often the hardest part which, once done, already provides a useful hint about what the resulting list homomorphism could be. In this paper we propose a new approach: to construct a possible candidate of the associative operator and, at the same time, to transform a proof that h is both a foldr and a foldl to a proof that h is a list homomorphism. The effort constructing the proof is thus not wasted, and the resulting program is guaranteed to be correct.

by Shin at October 04, 2011 12:40 PM

October 03, 2011

Douglas M. Auclair (geophf)

Monads in Java

A while back Brent Yorgey http://byorgey.wordpress.com posted a menagerie of types. A very nice survey of types in Haskell, including, among other things, the monadic types.

A reader posted a question, framed with 'you know, it'd be a lot easier for me to understand monads if you provided an implementation in Java.'[1]

Well, that reader really shouldn't have done that.

Really.

Because it got me to thinking, and dangerous things comes out of geophf's thoughts ... like monads in Java.

Let's do this.

Problem Statement

Of course, you just don't write monads for Java. You need some motivation. What's my motivation?

Well, I kept running up against the Null Object pattern, because I kept running up against null in the Java runtime, because there's a whole lot of optionality in the data sets I'm dealing with, and instead of using the Null Object, null is used to represent absence.

Good choice?

Well, it's expedient, and that's about the best I can say for that.

So I had to keep writing the Null Object pattern for each of these optional element types I encountered.

I encounter quite a few of them ... 2,100 of them and counting.

So I generalized the Null object pattern using generic types, and that worked fine ...

But I kept looking at the pattern. The generalization of the Null Object pattern is Nothing, and a present instance is Just that something.

I had implemented non-monadic Maybe.

And the problem with that is that none of the power of Maybe, as a Monad, is available to a generalized Null Object pattern.

Put another way: I wish to do something given that something is present.

Put another-another way: bind.

So, after a long hard look ('do I really want to implement monads? Can't I just settle for 'good enough' ... that really isn't good enough?), I buckled down to do the work of implementing not just Maybe (heh: 'Just Maybe'), but Monad itself.

Did it pay off? Yes, but wait and see. First the implementation. But even before that, let's review what a monad is.

What the Hell is a Monad?

Everybody seems to ask this question when first encountering monads, and everybody seems to come up with their own explanations, so we have tutorials that explain Monads as things as far-ranging from spacesuits to burritos.

My own explanation is to punt ... to category theory.

A monad is simply a triple (that's why it's called a monad, ... because it's a triple), defined thusly:

(T, η, μ)

Simple right?

Don't leave me!

Okay, yes. I understand you are rolling your eyes and saying 'well, that's all Greek to me.'

So allow me to explain in plain language the monad.

Monad is the triple (T, η, μ) where

  • T is the Monad Functor
  • η is the unit function that lifts an ordinary object (a unit) into the Monad functor; and,
  • μ is the join function that takes a composed monad M (M a) and returns the simplified type of the composition, or M a.

And that is (simply) what the hell a Monad is. What a monad can do ... well, there are articles galore about that, and, later, I will provide examples of what Monad gives us ... in Java, no less.

The Implementation

We need to work our way up to Monad. There's a whole framework, a way of thinking, a context, that needs to be in place for Monad to be effective or even to exist. Monad is 'happy' in the functional world, so we need to implement functions.

Or more correctly, 'functionals' ... in Haskell, they are called 'Functors.'

A functor is a container, and the function of functors is to take an ordinary function, which we write: f: a → b (pronounced "the function f from (type) a to (type) b" and means that f takes an argument of type a and returns a result of type b), and give a function g: Functor a → Functor b.

Functor basically is a container for the function fmap: Functor f ⇒ (a → b) → (f a → f b)

Well, is Functor the fundamental type? Well, yes and no.

It is, given that we have functions defined. In Java, we don't have that. We have methods, yes, but we don't have free-standing functions. Let's amend that issue:

<jcode>
<jkey>public interface</jkey> Applicable<T1, T2> {
<jkey>public</jkey> T2 apply(T1 arg) <jkey>throws</jkey> Failure;
};</jcode>

And what's this Failure thingie? It's simply a functional exception, so:

<jcode>
<jkey>public class</jkey> Failure <jkey>extends</jkey> Exception {
<jkey>public</jkey> Failure(String string) {
<jkey>super</jkey>(string);
}


/// and the serialVersionUID for serialization; your IDE can define that
}</jcode>

I call a Function 'Applicable' because in functional languages that's what you do: apply a function on (type) a to get (type) b. And, using Java Generics terminology, type a is T1 and type b is T2. So we're using pure Java to write pure Java code and none will be the wiser that we're actually doing Category Theory via a Haskell implementation ... in Java, right?

Right.

So, now we have functions (Applicable), we can provide the declaration of the Functor type:

<jcode>
<jkey>public interface</jkey> Functor<F, T> {
<jkey>public</jkey> <T1> Applicable<? <jkey>extends</jkey> Functor<F, T>,
? <jkey>extends</jkey> Functor<F, T1>>
fmap(Applicable<T, T1> f) <jkey>throws</jkey> Failure;


<jkey>public</jkey> T arg() <jkey>throws</jkey> Failure;
}</jcode>

The above definition of fmap is the same declaration (in Java) as its functional declaration: it takes a function f: a → b and gives a function g: Functor a → Functor b

Okay, we're half-way there. A Monad is a Functor, and we have Functor declared.

The unit function, η, comes for free in Java: it's called new.

So that leaves the join function μ.

And defining join for a specific monad is simple. Join for the list monad is append. Join for the Maybe monad and the ID monad is arg. What about join for just Monad?

Well, there is no definition of such, but we can declare its type:

<jcode>
<jkey>public interface</jkey> Joinable<F, T> <jkey>extends</jkey> Functor<F, T> {
<jkey>public</jkey> Functor<F, ?> join() <jkey>throws</jkey> Failure;
}</jcode>

Too easy! you complain.

Well, isn't it supposed to be? Join is simply a function that takes a functor and returns functor.

The trick is that we have to ensure that T is of type Functor<F, ?>, and we leave that an implementation detail for each specific monad implementing join.

That was easy!™

Monad

So now that we have all the above, Functor, Join, and Unit, we have Monad:

... BUT WAIT!

And here's the interlude of where the power of Monad comes in: bind.

What is bind? Bind is a function that says: do something in the monadic domain.

Really, that sounds boring, right? But it's a very powerful thing. Because once we are in the monadic domain, we can guarantee things that we cannot in Java category (if there is such).

(There is. I just declared it.)[2]

So, in the monadic domain, a function thus bound can be guaranteed to, e.g., be working with an instance and not a null.

Guaranteed.

AND! ...

Well, what is bind?

Bind is a function that takes an ordinary value and returns a result in the monadic domain:

Monad m ⇒ bind: a → m b

So the 'AND!' of this is that bind can be chained ... meaning we can go from guarantee to guarantee, threading a safe computation from start to finish.

Monad gives us bind.

And the beauty of Monad? bind comes for free, for once join is defined, bind can be defined in terms of join:

Monad m ⇒ bind (f: a → m b) (m a) = join ((fmap f) (m a))

Do you see what is happening here? We are using fmap to lift the function f one step higher into the monadic domain, so fmap f gives the function Monad m ⇒ g: m a → m (m b) so we are not actually passing in a value of type a, we are passing in the Monad m a, and then we get back a type of m (m b) which then join does it magic to simplify to the monadic type m b.

So to the outside world, it looks like we are passing in an a to get an m b but this definition allows monadic chaining, for a becomes m a and then m (m b) is simplified to m b that — and here's the kicker — is passed to the next monadically bound function.

You can chain this as far as you'd like:

m a >>= f >>= g >>= ... >>= h → m z

And since the computation occurs in the monadic domain, you are guaranteed the promise of that domain.

There is the power of monadic programming.

Given that explanation, and remembering that bind can be defined in terms of join, let's define Monad:

<jcode>
<jkey>public abstract class</jkey> Monad<M, A> <jkey>implements</jkey> Joinable<M, A> {
<jkey>public</jkey> <T> Monad<M, T> bind(Applicable<A, Monad<M, T>> f) <jkey>throws</jkey> Failure {
Applicable<Monad<M, A>, Monad<M, Monad<M, T>>> a
= (Applicable<Monad<M, A> Monad<M, Monad<M, T>>>) fmap(f);
Monad<M, Monad<M, T>> mmonad = a.apply(<jkey>this</jkey>);
<jkey>return</jkey> (Monad<M, T>) mmonad.join();
}


<jkey>public</jkey> Monad<M, A> fail(String ex) <jkey>throws</jkey> Failure {
<jkey>throw new</jkey> Failure(ex);
}
}</jcode>

The bind implementation is exactly as previously discussed: bind is the application of the fmap of f with the joined result returned. So, we simply define join for subclasses, usually a trivial definition, and we have the power of bind automagically!

Now fail is an interesting beast in subclasses, because in the monadic domain, failure is not always a bad thing, or even an exceptional one, and can be quite, and very, useful. Some examples, fail represents 'zero' in the monadic domain, so 'failure' for list is the empty list, and 'failure' for Maybe is Nothing. So, when failure occurs, the computation can continue gracefully and not always automatically abort from the computation, which happens with the try/catch paradigm.

There you have it: Monads in Java!

The rest of the story: ... Maybe

So, we could've ended this article right now, and you have everything you need to do monadic programming in Java. But so what? A programming paradigm isn't useful if there are no practical applications. So let's give you one: Maybe.

Maybe is a binary type, it is either Nothing or Just x where x is whatever value that is what we're really looking for (as opposed to the usual case in Java: <jkey>null</jkey>).

So, let's define the Maybe type in Java.

First is the protocol that extends the Monad:

<jcode>
<jkey>public abstract class</jkey> Maybe<A> <jkey>extends</jkey> Monad<Maybe, A> {
</jcode>

Do you see how the Maybe type declares itself as a Monad in its own definition? I just love that.

Anyway.

As mentioned before, fail for Maybe is Nothing:

<jcode>
<jkey>public</jkey> Maybe<A> fail(String ex) <jkey>throws</jkey> Failure {
<jkey>return</jkey> (Maybe<A>) NOTHING;
}</jcode>

(NOTHING is a constant in Maybe to be defined later. Before we define it ('it' being NOTHING), recall that a Monad is a Functor so we have to define fmap. Let's do that now)

<jcode>
<jkey>protected abstract</jkey> <T> Maybe<T>
mbBind(Applicable<A, Monad<Maybe, T>> arg) <jkey>throws</jkey> Failure;


<jkey>public</jkey> <T> Applicable<Maybe<A>, Maybe<T>>
fmap(<jkey>final</jkey> Applicable<A, T> f) <jkey>throws</jkey> Failure {
<jkey>return new</jkey> Applicable<Maybe<A>, Maybe<T>>() {
<jkey>public</jkey> Maybe<T> apply(Maybe<A> arg) <jkey>throws</jkey> Failure {
Applicable<A, Monad<Maybe, T>> liFted =
<jkey>new</jkey> Applicable<A, Monad<Maybe, T>>() {
<jkey>public</jkey> Maybe<T> apply(A arg) <jkey>throws</jkey> Failure {
<jkey>return</jkey> Maybe.pure(f.apply(arg));
}
};
<jkey>return</jkey> (Maybe<T>)arg.mbBind(liFted);
}
};
}</jcode>

No surprises here. fmap lifts the function f: a → b to Maybe m ⇒ g: m a → m b where in this case the Functor is a Monad, and this case, that Monad is Maybe.[3]

There is the small issue(s) of what the <jkey>static</jkey> method pure and the method mbBind are, but these depend on the definitions of the subclasses NOTHING and Just, so let's define them now.

<jcode>
<jkey>public static final</jkey> Maybe<?> NOTHING = <jkey>new</jkey> Maybe() {
<jkey>public</jkey> String toString() {
<jkey>return</jkey> "Nothing";
}
<jkey>public</jkey> Object arg() <jkey>throws</jkey> Failure {
<jkey>throw new</jkey> Failure("Cannot extract a value from Nothing.");
}
<jkey>public</jkey> Functor join() <jkey>throws</jkey> Failure {
<jkey>return this</jkey>;
}
<jkey>protected</jkey> Maybe mbBind(Applicable f) {
<jkey>return this</jkey>;
}
};</jcode>

Trivial definition, as the Null Object pattern is trivial. But do note that the bind operation does not perform the f computation, but skips it, returning Nothing, and forwarding the computation. This is expected behavior, for:

Nothing >>= f → Nothing

The definition of the internal class Just is nearly as simple:

<jcode>
<jkey>public final static class</jkey> Just<J> <jkey>extends</jkey> Maybe<J> {
<jkey>public</jkey> Just(J obj) {
_unit = obj;
}


<jkey>public</jkey> Maybe<?> join() <jkey>throws</jkey> Failure {
<jkey>try</jkey> {
<jkey>return</jkey> (Maybe<?>)_unit;
} <jkey>catch</jkey>(ClassCastException ex) {
<jkey>throw new</jkey> Failure("Joining on a flat structure!");
}
}
<jkey>public</jkey> String toString() {
<jkey>return</jkey> "Just " + _unit;
}
<jkey>public</jkey> Object arg() <jkey>throws</jkey> Failure {
<jkey>return</jkey> _unit;
}
<jkey>protected</jkey> <T> Maybe<T>
mbBind(Applicable<J, Monad<Maybe, T>> f) <jkey>throws</jkey> Failure {
<jkey>return</jkey> (Maybe<T>)f.apply(_unit);
}
<jkey>private final</jkey> J _unit;
}</jcode>

As you can see, the slight variation to NOTHING for Just is that join returns the _unit value if it's the Maybe type (and throws a Failure if it isn't), and mbBind applies the monadic function f to the Just value.

And with the definition of Just we get the <jkey>static</jkey> method pure:

<jcode>
<jkey>public static</jkey> <T> Maybe<T> pure(T x) {
<jkey>return new</jkey> Just<T>(x);
}</jcode>

And then we close out the Maybe implementation with:

<jcode>}</jcode>

Simple.

Practical Example

Maybe is useful for any semideterministic programming, that is: where something may be true, or it may not be. But the question I keep getting, from Java coders, is this: "I have these chains of getter methods in my web interface to my data objects to set a result, but I often have <jkey>null</jkey>s in some values gotten along the chain. How do I set the value from what I've gotten, or do nothing if there's a <jkey>null</jkey> along the way."

"Fine, no problems ..." I begin, but then they interrupt me.

"No, I'm not done yet! I have like tons of these chained statements, and I don't want to abort on the first one that throws a NullPointerException, I just want to pass through the statement with the <jkey>null</jkey> gracefully and continue onto the next assignment. Can your weirdo stuff do that?"

Weirdo stuff? Excusez moi?

I don't feel it's apropos that functionally pure languages have no (mutable) assignment, as that throws provability out the window and is therefore the root of all evil.

No matter how sorely tempted I am.

So, instead I say: "Yeah, that's a bigger problem [soto voce: so you should switch to Haskell!], but the same solution applies."[4]

So, let's go over the problem and a set of solutions.

The problem is something like:

<jcode><jkey>
try</jkey> {
foo.setD(getA().getB().getC().getD());
foo.setE(getA().getB().getC().getE());


bar.setJ(getF().getG().getH().getJ());
bar.setM(getF().getG().getK().getM());


// and ... oh, more than 2000 more assignments ... 'hypothetically'


} <jkey>catch</jkey>(Exception ex) {
ex.printStackTrace();
}</jcode>

And the problem is this: anywhere in any chain of gets, if something goes wrong, the entire computation is stopped from that point, even if there are, oh, let's say, 1000 more valid assignments.

A big, and real, problem. Or 'challenge,' if you prefer.

Now this whole problem would go away if the Null Object pattern was used everywhere. But how to enforce that? In the Java category, you cannot. Even if you make the generic Null Object the base object, the <jkey>null</jkey> is still Java's &bottom — it's 'bottom' — as low as you go in a hierarchy, <jkey>null</jkey> is still an allowable argument everywhere.

And if getA(), or getB(), or getC() return <jkey>null</jkey> you've just failed out of your entire computation with an access attempt to a null pointer.

Solutions

Solution 1: test until you puke

The first solution is to test for null at every turn. And you know what that looks like, but here you go. Because why? Because if you think it or code it, you've got to look at it, or I've got to look at it, so here it is:

<jcode>
A a = getA();
<jkey>if</jkey>(a != <jkey>null</jkey>) {
B b = a.getB();
<jkey>if</jkey>(b != <jkey>null</jkey>) {
C c = b.getC();
<jkey>if</jkey>(c != <jkey>null</jkey>) {
foo.setD(c.getD());
}
}
}</jcode>

What's the problem with this code?

Oh, no problems, just repeat that deep nesting for every single assignment? So you have [oh, let's say 'hypothetically'] 2000 of these deeply nested conditional blocks?

And, besides the fact that it entirely clutters the simple assignment:

<jcode>foo.setD(getA().getB().getC().getD());</jcode>

in decision logic and algorithms that are totally unnecessary and entirely too much clutter.

All I'm doing is assigning a D to a foo! So why do I have to juggle all this conditional code in my head to reach that eventual assignment.

Solution 1 is bunk.

Solution 2: narrow the catch

So, solution 1 is untenable. But we need to assign where we can and skip where we can't. So how about this?

<jcode><jkey>
try</jkey> {
foo.setD(getA().getB().getC().getD());
} <jkey>catch</jkey>(Exception ex) {
// Intentionally empty; silently don't assign if we cannot;
}


<jkey>try</jkey> {
foo.setE(getA().getB().getC().getE());
} <jkey>catch</jkey>(Exception ex) {
// ditto
}


<jkey>try</jkey> {
bar.setJ(getF().getG().getH().getJ());
} <jkey>catch</jkey>(Exception ex) {
// ditto
}


<jkey>try</jkey> {
bar.setM(getF().getG().getK().getM());
} <jkey>catch</jkey>(Exception ex) {
// ditto
}</jcode>

And I say to this solution, meh! Granted, it's much better than solution 1, but, again, you made a computational flow described declaratively in the problem to be these set of staccato statements, having the reader of your code switch into and out of context for every statement.

Wouldn't it be nice to have all the statements together in one block, because, computationally, that's what is happening: a set of data is collected from one place and moved to another, and that is what we wish to describe by keep these assignment grouped in one block.

Solution 3: Maybe Monad

And that's what we can do with monads. It's going to look a bit different that how you're used to the usual Java fair, as we have to lift the statements in the Java category into expressions in the monadic one.

So here goes.

First of all, we need to define generic functions (not methods![5]) for getting values from objects and setting values into object in the monadic domain:

<jcode><jkey>
public abstract class</jkey> SetterM<Receiver, Datum>
<jkey>implements</jkey> Applicable<Datum, Monad<Maybe, Receiver>> {
<jkey>protected</jkey> SetterM(Receiver r) {
<jkey>this</jkey>.receiver = r;
}


// a monadic set function
<jkey>public final</jkey> Monad<Maybe, Receiver> apply(Datum d) <jkey>throws</jkey> Failure {
set(receiver, d);
<jkey>return</jkey> Maybe.pure(receiver);
}


// subclasses implement with the particular set method being called
<jkey>protected abstract void</jkey> set(Receiver r, Datum d);


<jkey>private final</jkey> Receiver receiver;
}</jcode>

This declares a generic setter, so to define setD on the foo instance, we would do the following:

<jcode>
SetterM<Foo, D> setterDinFoo = <jkey>new</jkey> SetterM<Foo, D>(foo) {
<jkey>protected void</jkey> set(Foo f, D d) {
f.setD(d);
}
};</jcode>

The getter functional is similarly defined, with the returned value lifted into (or 'wrapped in,' if you prefer) the Maybe type:

<jcode>
<jkey>public abstract class</jkey> GetterM<Container, Returned>
<jkey>implements</jkey> Applicable<Container, Monad<Maybe, Returned>> {


<jkey>public</jkey> Monad<Maybe, Returned> apply(Container c) <jkey>throws</jkey> Failure {
Maybe<Returned> ans = (Maybe<Returned>)Maybe.NOTHING;
Returned result = get(c);


// c inside the monad is guaranteed to be an instance
// but result has NO guarantees!


<jkey>if</jkey>(result != <jkey>null</jkey>) {
ans = Maybe.pure(result);
// and now ans is inside the monad it must have a value.
// ... even if that value is NOTHING
}
<jkey>return</jkey> ans;
}


<jkey>protected abstract</jkey> Returned get(Container c);
}</jcode>

With the above declaration, a getter monadic function (again, not method) is simply defined:

<jcode>
GetterM<A, B> getterBfromA = <jkey>new</jkey> GetterM<A, B>() {
<jkey>protected B</jkey> get(A a) {
<jkey>return</jkey> a.getB();
}
};</jcode>

In Haskell, the bind operator has a data-flow look to it: (>>=), so writing one of the 'assignments' is as simple as flowing the data to the setter:

getA >>= getB >>= getC >>= getD >>= setD foo

But we have no operator definitions, so we must soldier on using the Java syntax:

<jcode>
<jkey>try</jkey> {
getterA.apply(<jkey>this</jkey>).bind(getterBfromA).bind(getterCfromB).bind(getterDfromC).bind(setterDinFoo);
} <jkey>catch</jkey> {
// a superfluous catch block
}</jcode>

That's one of the assignment expressions.[6]

Let's walk through what happens with a couple of examples.

Let's say that A has a value of a, but B is <jkey>null</jkey>. What happens?

  1. getterA forwards Just a to getterBfromA
  2. getterBfromA gets a <jkey>null</jkey>, converts that into a NOTHING and forwards that to getterCfromB
  3. getterCfromB forwards the NOTHING to getterDfromC
  4. getterDfromC forwards the NOTHING to setterDinFoo
  5. setterDinFoo simply returns NOTHING.
And we're done (that is: we're done doing, literally: NOTHING).

Let's counter with an example where every value has an instance:

  1. getterA forwards Just a to getterBfromA
  2. getterBfromA gets a b, converts that into Just b and forwards that to getterCfromB
  3. getterCfromB gets a c, converts that into Just c and forwards that to getterDfromC
  4. getterDfromC gets a d, converts that into Just d and forwards that to setterDinFoo
  5. setterDinFoo gets the wrapped d and sets that value in the Foo instance.

Voilà!

The beauty of this methodology is that we can put them all into one block, and every expression will be evaluated and in a type-safe manner, too, as NOTHING will protect us from a null pointer access:

<jcode>
<jkey>try</jkey> {
getterA.apply(this).bind(getterBfromA).bind(getterCfromB).bind(getterDfromC).bind(setterDinFoo);
System.out.println("Hey, I've executed the first assignment");
getterA.apply(this).bind(getterBfromA).bind(getterCfromB).bind(getterEfromC).bind(setterEinFoo);
System.out.println("...and the second...");
getterF.apply(this).bind(getterGfromF).bind(getterHfromG).bind(getterJfromH).bind(setterJinBar);
System.out.println("...and third...");
getterF.apply(this).bind(getterGfromF).bind(getterKfromG).bind(getterMfromK).bind(setterMinBar);
System.out.println("...and fourth...");


// another more than 2000 of these expressions
System.out.println("...and we're done with every assignment we could assign...");


} <jkey>catch</jkey> {
// a superfluous catch block
}</jcode>

And, because the monadic Maybe works it does, every assignment that can occur will, and ones that cannot will be 'skipped' (by flowing NOTHING through the rest of the computation, and the proof will be in the pudding ... on the standard output will be all the statements printed.

Summary

We presented an implementation of monads in Java, with a concrete example of the Monad type: the Maybe type. We then showed that by putting a computational set into the monadic domain, the work can be performed in a type-safe manner, even with the possible presence of <jkey>nulls.</jkey> This is just one example of practical application of monadic programming in Java: the framework presented here confers the benefits of research of monadic programming in general to projects done in Java. Use the framework to discover for yourself the leverage monadic programming gives.


End notes

  
[1] Exact verbage:Phil says:January 14, 2009 at 7:01 pmI think that this could all be easily cleared up if one of you FP guys would just show us how to write one of these monad thingies in JavaâTo which a semi-mocking response was: 'you know, objects would be a lot easier for me to understand if you provided an implementation in Haskell.'Exact verbage:Cory says:April 23, 2009 at 6:26 amI may be a bit late to the game here, but Phil, that can be rephrased:I think that this could all be easily cleared up if one of you OO guys would just show us how to write one of these object thingies in HaskellâOf course you can, but it's a different type of abstraction for a different way of thinking about programmingâI write 'semi-mocking,' because: 1. OOP with message passing has been implemented in a Haskell-like language: Common Lisp. The Art of the Metaobject Protocol, a wonderful book, covers the 'traditional' object-oriented programming methodology, and its implementation, quite thoroughly and simply.
[2] I leave it as an exercise to the reader to scope the Java category (Consult the Ω-calculus papers, Cardelli, et al;).
[3] <jcode>Monad<Maybe, A></jcode> and <jcode>Maybe<A></jcode> are type-equivalent (but try telling the poor Java 1.6 compiler that).
[4] "The same solution applies!" Geddit? Haskell is an applicative language, so the same solution applies! GEDDIT?
[5] The distinction I raise here between methods and functions is the following: a method is enclosed in and owned by the defining class. A function, on the other hand, is a free-standing object and not necessarily contained in nor attached to an object.
[6] Again, there is a distinction between an expression and a statement. An expression returns a value; a statement does not. A statement is purely imperative ... 'do something.' On the other hand, an expression can be purely functional ... 'do'ing nothing, as it were, e.g.: 3 + 4. A pure expression has provable properties that can be attached to it, whereas it is extremely difficult to make assertions about statements in the general sense. The upshot is that having pure (non-side-effecting) expressions in code allows us to reason about that code from a firm theoretical foundational basis. Such code is easier to test and to maintain. Side-effecting statement code make reasoning, testing and maintaining such code rather difficult.

by geophf (noreply@blogger.com) at October 03, 2011 10:48 AM

Cartesian Closed Comic

October 02, 2011

Russell O'Connor

The ACM and me

Let me make one thing clear from the beginning: it was the ACM’s choice to remove my publication from their workshop proceedings. I did nothing to stop them. In fact, by waiving my copyright, I made it extraordinarily easy for them to include my work in their proceedings if they wanted.

This is my story. This was the first time I have sumbitted a paper to an ACM workshop. I have sumbitted several papers to other conferences and journals in the past without issue. For this workshop I prepared a paper entitled "Lens is to Functor as Applicative is to Biplate: Introducing Multiplate" to the Workshop for Generic Programming 2011. I went through the usual refereeing process. My paper was accepted by the program committee. I made some revisions based on the referee reports. As typical, I was asked to transfer my copyright to the publisher.

In today’s world, transferring copyright is problematic for researchers like me. We want our papers as widely read as possible in order for them to be as influential as possible. Historically, the best way to do this was to have the paper published, because this would mean that copies of our work would end up getting disseminated to university libraries around the world. Publishing is not free, but in lieu of payment for publishing, we would transfer our copyright to the publisher. However, in today’s world the best way to have my paper widely read is to submit it to an online repository, such as the arXiv, where anyone with internet access can get instant access to my work.

So as per my copyright policy, I uploaded my final version of my paper to the arXiv under a public domain dedication. When I was at Radboud University I would also submit a version to the university digital library as well. One version would be under a non-restrictive creative commons license just to cover all bases. However, the new creative commons public domain dedication seems better today and I do not have access to the Radboud University digital library anymore. Therefore, submitting to the arXiv seems sufficient for now. Because it is in the public domain, I, or anyone else, can re-archive my paper elsewhere at a later time.

Before such submission I check with the conference/journal editors to see if this is acceptable. Most journals and conferences will not accept previously published submissions. However this policy usually refers to previously peer-reviewed publications, and since the arXiv is not peer reviewed, no editor has yet had a problem with this. Most editors will tell me that publication requires a copyright transfer and the publisher might not like my final version published on the arXiv, but they maintain that this is an issue between the publisher and me. I admit that for this workshop publication I let this slip a bit. Due to the time constraints, I ended up contacting the workshop chairs after submitting the the arXiv rather than before, but, as usual, the chairs had no problem with a final copy of my paper appearing on the arXiv.

In previous publications, I print out a copyright transfer form from the publisher. Recall, that transfer of copyright is in lieu of payment for publication. It would be misleading, and possibly illegal, for me to transfer copyright to them after publishing my papers under a public domain dedication or creative commons license without notifying the publisher. Therefore, I always amend the copyright transfer agreement to make a note that I have already published my work under a public domain dedication and creative commons license and I am only transferring copyright to the extend possible (which I believe amounts to nothing). After mailing or faxing the amended copyright transfer agreement to the publisher, no publisher has yet refused to publish my work. They publish it after copy editing it, and stamp their own copyright on it. I find their copyright claim dubious; but I have no incentive to pursue the issue.

With the ACM things are a little different. Firstly, they want me to put their copyright notice onto my work myself. Secondly, they have a dubious on-line electronic copyright transfer assignment instead of a paper form. It is one thing for the publisher to put their own copyright onto my public domain dedicated work, it is another thing for me to do it myself. So instead of putting the standard ACM copyright notice into my paper, I replaced it with the creative commons public domain dedication and submitted that to the printer. Unfortunately the on-line electronic form proved to be more problematic. It is not possible to make amendments to an electronic form. (Believe me, I tried. I looked to see if I could modify the HTML, but the dubious electronic form only submits the state of checkboxes, not the text of what is agreed to.) Not really having any recourse, all I did was wait.

After I got my second reminder to fill out the copyright transfer form, I replied saying that since my paper was in the public domain, it was not necessary, nor possible, for me to transfer copyright. Then the printer asked for a copy of the permission letter from the ACM to change the copyright notice. This is when things started to take a turn for the worse. I had no such letter. So, I asked the chairs of the workshop. They had no problem with such a public domain dedication, however they do not have authorization to let me amend the ACM notice, so my request was forwarded to ACM legal where my request was promptly denied.

At this point in the story, I would like to pause and note that the ACM does have a procedure for accepting public domain papers. If I were a US government employee, I would be required by law to place my work into the public domain. The ACM has arrangements to accept publications from US government employees and uses a modified copyright notice for them. But if you are not such a government employee, well … we shall see.

I replied to ACM legal explaining that I have put a copy of my publication on the arXiv under a public domain dedication and therefore I was unable to transfer copyright to them. ACM legal replied, I understand that you have placed your paper in the public domain and on the arXiv site. It is therefore considered as published and there is no need for ACM to republish it in the WGP proceedings. (Edit: Robert Simmons notes that ACM Transactions on Computational Logic (TOCL) recommends authors to put their final article version into CoRR. I guess parallel publication in the arXiv is okay for some ACM publications but not for others.)

The ACM still allowed me to present my work at the workshop, but they refused to allow my paper to be published in the proceedings. Instead the chairs were allowed to insert a small note stating, We note that one of the papers presented in the workshop is not included in the proceedings. This paper, ‘Functor is to Lens as Applicative is to Biplate: Introducing Multiplate’ by Russell O’Connor, is accessible as arXiv:1103.2841v2 [cs.PL].


As I said above, it is the researcher’s interest to have their publications as widely distributed as possible. When researchers transfer their copyright to the ACM they get a few very limited rights in return. They can post a copy of their work on their own “Home Page” and on their institution’s server. You do not have permission post your paper anywhere else, so no matter how technology changes you are stuck with using your “Home Page” (I imagine if the ACM copyright policy was formed 20 years ago you would be limited to posting on your personal Gopher page). You do not have permission to email copies of your paper to colleagues. You do not have permission to print out your paper and hand it to colleagues. Your colleagues who download copies of your paper from your “Home Page” do not have permission to print out your paper. You are allowed to make derived works of your paper; however you must include an ACM citation. You do not have permission to give anyone permission to include your paper in another publication, say of collected works in type theory, or top 10 most influential computer science papers of this century.

Most researchers appear to ignore the ACM’s copyright policy and freely share their papers in violation of, now ACM’s, copyright. Think you cannot be sued? Think again. Recall that Eric’s Treasure Trove of Mathematics was shut down because he transferred his copyright to his publisher, who then in turn sued him for distributing a derived work of, now their, Treasure Trove.

Many researchers have the attitude that the ACM would never ruin their reputation by going after their own authors for copyright violations. While this may be true for now, there is no guarantee that this will continue in the future. Imagine what happens if in the future the ACM goes bankrupt. Their creditors could swoop in and grab an extremely value stack copyrights to academic publications. They could become copyright trolls, sweeping the internet for illegal exchanges of ACM owned papers by academics. At the same time the creditors could take over the ACM Digital Library, quadruple their fees, and dispose of the less profitable articles. No point in keeping unsaleable articles around. Oh, and since some libraries only have digital subscriptions to ACM publications now, they will lose access to articles published even before this theoretical ACM bankruptcy. Maybe this will never happen with the ACM, but there are lots of organizations like the ACM. One day this will happen to the ACM or the IEEE or some other significant academic publisher. When that happens, there will be no recourse.

Why do researchers put up with transferring their copyright to the ACM? The most common answer to this is that you have to do so in order to get your paper accepted into high quality ACM conferences. However, the ACM does not make conferences high quality (at least I am not aware of any significant resources they provide, granted I have never organized an ACM conference). It is the organisers, committee, and especially the participants that make a conference high quality. I think that if we had not evolved into this situation where top conferences require copyright transfer that we would never accept it.

What shall we do about it? I was originally planning to never participate in an ACM conference again; however one participant that I talked to suggested that I keep doing what I am doing. I can continue to post my final version of my publications on the arXiv and not transfer my to the ACM. The ACM can continue to refuse to put my publications in their proceedings. However, I as long as I can continue to present my work at their workshops and conferences, although not ideal, it is good enough for me. I will continue to list my papers on my CV as peer reviewed. People will still be able to find my publications from my DBLP page. And my publications will be part of the long term CoRR archive, an archive that is likely more resilient that even the ACM’s digital library. If you care to join me, then pretty soon the ACM will have no more papers in their proceedings. Maybe that will make them rethink their policy.

Matt Blaze seems to share similar concerns to me.

October 02, 2011 06:12 PM

October 01, 2011

Clifford Beshers

@ConalElliott re: What resources & practices (teaching Haskell)

Conal Elliott asks: What resources & practices would you recommend for helping a group of Java & C++ programmers learn to work in Haskell? I saw the question first on Twitter, but my compression skills were not up to the task of answering there.

I have two recommendations: teach them the simplest definitions of the fundamentals; read programs with them, out loud, like children's books, skipping nothing.

These steps are often passed hurriedly, by teachers so steeped in a subject they don't realize how different their worlds are, by students eager to build things as quickly in a new programming language as they do in the old. Most likely, the students will assume that they just need to learn new syntax to be off and running. Indeed, I am sure they are capable, self-motivated software professionals and that they would succeed in learning Haskell, just without the deeper understanding that can make it such a joy.

As a test, ask your students to write down the definition of 'type'. I'll bet that their answers will be longer than 'set' and include some mention of bytes, words and big-endian. We were all trained to be mechanics instead of drivers, because all we had were go-carts.

C and its children will have left clutter in their minds. Types will be obstacles, efficiency a drug, correctness assured by machismo. They will gloss over code without really reading it, trying to fit it into an existing model. Saying things out loud helps slow things down, forces them to make connections between symbols, words and definitions so fundamental they are rarely written down. (I backtracked through Paul Hudak's book for hours trying to find out how to say '::'; newcomers ask pretty regularly in #haskell.)

Make sure they know all the places that patterns can appear in code and how they work. Make sure they know that data constructors are functions. Have them verify that with :type in GHCi. And that type constructors are functions, verifying with :kind. They should be familiar with the syntax and semantics of higher-order types before they meet IO, so that they understand it is not magic.

Lists, recursion, functional algorithms, none of these will be challenges, but constructs they have not seen before will give them much more trouble if they cannot deconstruct them to primitives. I cannot imagine Conal omitting such fundamental semantics from his lessons, but I can easily imagine both teacher and students, excited to build new stuff, skimping on the exercises necessary to flush out the old foundations and cement the new.

by Clifford Beshers (noreply@blogger.com) at October 01, 2011 07:45 PM

Gregory Collins

Slides from CUFP 2011: Snap Framework Tutorial

I recently gave a tutorial at CUFP 2011 about web programming in Haskell using the Snap Framework.

The tutorial application is called “Snap Chat”, and is an implementation of a simple multi-user chat room, using Haskell concurrency primitives and HTTP long-polling.

View the slides from the talk in HTML here.

Logging into the chat room

Logging into the chat room

An exciting discussion

An exciting discussion

by Gregory Collins (greg@gregorycollins.net) at October 01, 2011 04:31 PM

September 28, 2011

Twan van Laarhoven

Finding rectangles

This post is based on a part of my masters thesis. The topic of my thesis was OCR of historical documents. A problem that came up there was the following:

Given a binary image, find the largest axis aligned rectangle that consists only of foreground pixels.

These largest rectangles can be used, for instance, to find columns in a page of text. Although in that case one would use large rectangles of background pixels.

Here is an example image,
.
White pixels are background and blue is the foreground. The rectangle with the largest area is indicated in red. The images you encounter in practical application will be much larger than this example, so efficiency is going to be important.

Specification

Let's start with the types of images and rectangles

-- An image is a 2D list of booleans, True is the foreground
type Image = [[Bool]]
-- An axis-aligned rectangle data Rect = Rect { left, top, width, height :: Int } deriving (Eq,Ord,Show)

And some properties of them,

-- The size of an image
imWidth, imHeight :: Image -> Int
imHeight = length
imWidth (x:_) = length x
imWidth []    = 0
-- The area and perimeter of a rectangle area, perimeter :: Rect -> Int area rect = width rect * height rect perimeter rect = 2 * width rect + 2 * height rect

I will say that an image 'contains' a rectangle if all pixels inside the rectangle are foreground pixels.

contains :: Image -> Rect -> Bool
contains im (Rect x y w h) = and pixelsInRect
  where
    pixelsInRect = concatMap cols (rows im)
    rows = take h . drop y . (++repeat [])
    cols = take w . drop x . (++repeat False)

Now the obvious, stupid, way of finding the largest rectangle is to enumerate all rectangles in the image, and pick the largest from that list:

-- List all rectangles contained in an image
allRects :: Image -> [Rect]
allRects im = filter (im `contains`) rects
  where
    rects = [Rect x y w h | x <- [0..iw], y <- [0..ih]
                          , w <- [1..iw-x], h <- [1..ih-y]]
    iw = imWidth im
    ih = imHeight im

For now, I will take 'largest rectangle' to mean one with the maximal area. I will come back to this choice soon.

largestRectspec :: Image -> Rect
largestRectspec = maximalRectBy area . allRects
-- Return the rectangle with maximum f, -- using lexicographical ordering to break ties -- return noRect if there are no rectangles in the input list. maximalRectBy :: Ord a => (Rect -> a) -> [Rect] -> Rect maximalRectBy f = maximumBy (comparing f `mappend` compare) . (noRect:) where noRect = Rect 0 0 0 0

The above code should hopefully be easy to understand. It will find the correct answer for the above example:

> largestRectspec example
Rect {left = 3, top = 2, width = 4, height = 5}

Of course largestRectspec is horribly slow. In an n by n image there are O(n4) rectangles to consider, and checking if one is contained in the image takes O(n2) work, for a total of O(n6).

What is 'largest'?

Before continuing, let's determine what it means for a rectangle to be the largest. We could compare the area of rectangles, as we did before. But it is equally valid to look for the rectangle with the largest perimeter.

Can we pick the maximum according to any arbitrary function f :: (Rect -> a)? Not all of these functions will correspond to the intuitive notion of 'largest'. For example f = negate . area will actually lead to the smallest rectangle. In general there is going to be no efficient way of finding the rectangle that maximizes f. All we could do is optimize contains, to get an O(n4) algorithm.

We should therefore restrict f to be monotonic. What I mean by monotonic is that f x >= f y whenever rectangle x contains rectangle y. In QuickCheck code:

prop_isMonotonic :: Ord a => (Rect -> a) -> Property
prop_isMonotonic f = property $ \x y ->  x `rectContains` y  ==>  f x >= f y
rectContains :: Rect -> Rect -> Bool rectContains (Rect x y w h) (Rect x' y' w' h') = x <= x' && y <= y' && x+w >= x'+w' && y+h >= y'+h'

Area is a monotonic function, and so is perimeter. But you could also add weird constraints. For example, only consider rectangles that are at least 10 pixels tall, or only rectangles that contain the point (123,456).

Maximizing a monotonic function, as opposed to just any function, means that we can skip a lot of rectangles. In particular, whenever rectangle x contains rectangle y, rectangle y doesn't need to be considered. I will call rectangles in the image that are not contained in other (larger) rectangles maximal. The strategy for finding the largest rectangle is then simply to enumerate only the maximal rectangles, and pick the best of those:

largestRectfast :: Image -> Rect
largestRectfast = maximalRectBy area . allMaximalRects

For each maximal rectangle there is (trivially) a monotonic function that is maximal for that rectangle. So we can't do any better without taking the specific function f into account.

Machinery

To find maximal rectangles, we are first of all going to need some machinery for working with images. In particular, zipping images together,

zip2d :: [[a]] -> [[b]] -> [[(a,b)]]
zip2d = zipWith zip
zipWith2d :: (a -> b -> c) -> [[a]] -> [[b]] -> [[c]] zipWith2d = zipWith . zipWith
zipWith2d4 :: (a -> b -> c -> d -> e) -> [[a]] -> [[b]] -> [[c]] -> [[d]] -> [[e]] zipWith2d4 = zipWith4 . zipWith4

And accumulating/scanning over images. This scanning can be done in four directions. Each scanX function takes a function to apply, and the initial value to use just outside the image. The scans that I use here are slightly different from scanl and scanr, because the output will have the same size as the input, instead of being one element larger.

scanLeftward, scanRightward, scanUpward, scanDownward
    :: (a -> b -> b) -> b -> [[a]] -> [[b]]
scanLeftward f z = map (init . scanr f z) scanRightward f z = map (tail . scanl (flip f) z) scanUpward f z = init . scanr (\as bs -> zipWith f as bs) (repeat z) scanDownward f z = tail . scanl (\as bs -> zipWith f bs as) (repeat z)

Here is an example of a scan that calculates the x-coordinate of each pixel,

let x = scanRightward (\a x -> x + 1) (-1) a
x = .

And the y-coordinates are of course

let y = scanDownward (\a x -> x + 1) (-1) a
y = .

Finding lines

If we were looking for one-dimensional images, then a 'rectangle' would just be a single line of pixels. Now each pixel is contained in at most one maximal line of foreground pixels. To find the coordinates of this line, we just need to know the left and right endpoints.

For a foreground pixel, the left endpoint of the line it is in is the same as the left endpoint of its left neighbor. On the other hand, a background pixel is not in any foreground line. So the left endpoint of all lines to the right of it will be at least x+1, where x is the x-coordinate of the background pixel. In both these cases information flows from left to right; and so the left endpoint for all pixels can be determined with a rightward scan.

Unsurprisingly, we can find the right endpoints of all foreground lines with a leftward scan. Now let's do this for all lines in the image. Notice that we need the x coordinates defined previously:

let l = scanRightward (\(a,x) l -> if a then l else x+1) 0 (zip2d a x)
l = 
let r = scanLeftward (\(a,x) r -> if a then r else x) (imWidth a) (zip2d a x)
r = 

In the images I have marked the left and right endpoints of the foreground lines in red. Note also, the values in the background pixels are not important, and you should just ignore them.

Vertically we can of course do the same thing, giving top and bottom endpoints:

let t = scanDownward (\(a,y) t -> if a then t else y+1) 0 (zip2d a y)
t = 
let b = scanUpward (\(a,y) b -> if a then b else y) (imHeight a) (zip2d a y)
b = 

However, combining these left/right/top/bottom line endpoints does not yet give rectangles containing only foreground pixels. Rather, it gives something like a cross. For example using the endpoints for (6,4) leads to the following incorrect rectangle,
.

In fact, there are many rectangles around this point (6,4):
,
and before looking at the area (or whatever function we are maximizing) there is way no telling which is the best one.

If there was some way to find just a single maximal rectangle for each pixel, then we would have an O(n2) algorithm. Assuming of course that we do find all maximal rectangles.

Finding maximal rectangles

Suppose that Rect x y w h is a maximal rectangle. What does that mean? First of all, one of the points above the rectangle, (x,y-1),(x+1,y-1),..,(x+w-1,y-1), must not be the a foreground pixel. Because if all these points are foreground, then the rectangle could be extended upwards, and it would not be maximal. So, suppose that (u,y-1) is a background pixel (or outside the image). Then (u,y) is the top endpoint of the vertical line that contains (u,y+h-1).

If we start from (u,v), we can recover the height of a maximal rectangle using the top endpoint image t. Just take t!!(u,v) as the top coordinate, and u+1 as the bottom. This image illustrates the idea:
.
Here the green point (u,v) has the red top endpoint, and it gives the height and vertical position of the yellow maximal rectangle.

Then to make this vertical line into a maximal rectangle, we just extend it horizontally as far as possible:
.

For this last step, we need to know the first background pixel that will be encountered when extending the rectangle to the left. That is the maximum value of all left endpoints in the rows t,t+1,..,b-1. This maximum can again be determined with a scan over the image:

let lt = scanDownward (\(a,l) lt -> if a then max l lt else minBound) minBound (zip2d a l)
lt = 

For extending to the right the story is exactly the same, only taking the minimum right endpoint instead:

let rt = scanDownward (\(a,r) rt -> if a then min r rt else maxBound) maxBound (zip2d a r)
rt = 

Now we have all the ingredients for finding maximal rectangles:

  • For a foreground pixel (u,v):
  • Take as top t!!(u,v)
  • Take as left lt!!(u,v)
  • Take as right rt!!(u,v)
  • Take as bottom v+1.

Every maximal rectangle can be found in this way. However, not all rectangles we get in this way are maximal. In particular, they could potentially still be extended downward. However, for finding the largest rectangle, it doesn't matter if we also see some non-maximal ones. There might also be duplicates, which again does not matter.

So now finishing up is just a matter of putting all the steps together in a function:

allMaximalRects :: Image -> [Rect]
allMaximalRects a = catMaybes . concat $ zipWith2d4 mkRect lt rt t y
  where
    x  = scanRightward (\_ x -> x + 1) (-1) a
    y  = scanDownward  (\_ y -> y + 1) (-1) a
    l  = scanRightward (\(a,x) l -> if a then l else x+1) 0 (zip2d a x)
    r  = scanLeftward  (\(a,x) r -> if a then r else x) (imWidth a) (zip2d a x)
    t  = scanDownward  (\(a,y) t -> if a then t else y+1) 0 (zip2d a y)
    lt = scanDownward  (\(a,l) lt -> if a then max l lt else minBound) minBound (zip2d a l)
    rt = scanDownward  (\(a,r) rt -> if a then min r rt else maxBound) maxBound (zip2d a r)
    mkRect l r t y
        | l /= minBound = Just $ Rect l t (r-l) (y-t+1)
        | otherwise     = Nothing

A quick QuickCheck shows that largestRectfast finds the same answer as the slow specification:

propfast_spec = forAll genImage $ \a -> largestRectspec a == largestRectfast a
> quickCheck propfast_spec
+++ OK, passed 100 tests.

Conclusion

It is possible to find all maximal rectangles that consist entirely of foreground pixels in an n*n image in O(n2) time. That is linear in the number of pixels. Obviously it is not possible to do any better in general.

You may wonder whether this method also works in higher dimensions. And the answer to that question is no. The reason is that there can be more than O(n3) maximal cubes in a three dimensional image. In fact, there can be at least O(n(d-1)2) maximal hypercubes in d dimensions. Just generalize this image to 3D:
. Or click here for a 3D version.

September 28, 2011 07:49 PM

September 26, 2011

Robert Harper

Heartbeat

Just a note in response to several requests to say that I am still around, and plan to resume blogging soon.   I have been distracted by more pressing concerns, including teaching at the Oregon Programming Languages Summer School (OPLSS), preparing a submission to  POPL, visiting the Max Planck Institute for Software Systems, and trying hard to finish my book (on which I’ve made substantial progress over the summer).


Filed under: Programming, Research, Teaching

by Abstract Type at September 26, 2011 08:53 PM

September 24, 2011

Dominic Orchard

Constraint kinds in Haskell, finally bringing us constraint families

Back in 2009 Tom Schrijvers and I wrote a paper entitled Haskell Type Constraints Unleashed [1] which appeared at FLOPS 2010 in April. In the paper we fleshed out the idea of adding constraint synyonyms and constraint families to GHC/Haskell, building upon various existing proposals for class families/indexed constraints. The general idea in our paper, and in the earlier proposals, is to add a mechanism to GHC/Haskell to allow constraints, such as type class or equality constraints, to be indexed by a type in the same way that type families and data families allow types to be indexed by types.

As an example of why constraint families are useful, consider the following type class which describes a simple, polymorphic, embedded language in Haskell (in the “finally tagless“-style [2]) (this example appears in [1]):

class Expr sem where
    constant :: a -> sem a
    add :: sem a -> sem a -> sem a

Instances of Expr provide different evaluation semantics for the language. For example, we might like to evaluate the language for numerical values, so we might try and write the following instance:

data E a = E {eval :: a}

instance Expr E where
     constant c = E c
     add e1 e2 = E $ (eval e1) + (eval e2)

However, this instance does not type check. GHC returns the type error:

    No instance for (Num a)
      arising from a use of `+'
    In the second argument of `($)', namely `(eval e1) + (eval e2)'
    ...

The + operation requires the Num a constraint, yet the signature for add states no constraints on type variable a, thus the constraint is never satisfied in this instance. We could add the Num a constraint to the signature of add, but this would restrict the polymorphism of the language: further instances would have this constraint forced upon them. Other useful semantics for the language may require other constraints e.g. Show a for pretty printing. Should we just add more and more constraints to the class? By no means!

Constraint families, as we describe in [1], provide a solution: by associating a constraint family to the class we can vary, or index, the constraints in the types of add and constant by the type of an instance of Expr. The solution we suggest looks something likes:

class Expr sem where
    constraint Pre sem a
    constant :: Pre sem a => a -> sem a
    add :: Pre sem a => sem a -> sem a -> sem a

instance Expr E where
    constraint Pre E a = Num a
    ... -- methods as before

Pre is the name of a constraint family that takes two type parameters and returns a constraint, where the first type parameter is the type parameter of the Expr class.

We could add some further instances:

data P a = P {prettyP :: a}

instance Expr P where
    constraint Pre P a = Show a
    constant c = P c
    add e1 e2 = P $ prettyP e1 ++ prettyP e2

At the time of writing the paper I had only a prototype implementation in the form of a preprocessor that desugared constraint families into some equivalent constructions (which were significantly more awkward and ugly of course). There has not been a proper implementation in GHC, or of anything similar. Until now.

At CamHac, the Cambridge Haskell Hackathon, last month, Max Bolingbroke started work on an extension for GHC called “constraint kinds”. The new extensions unifies types and constraints such that the only distinction is that constraints have a special kind, denoted Constraint. Thus, for example, the |Show| class constructor is actually a type constructor, of kind:

Show :: * -> Constraint

For type signatures of the form C => t, the left-hand side is now a type term of kind Constraint. As another example, the equality constraints constructor ~ now has kind:

~ :: * -> * -> Constraint

i.e. it takes two types and returns a constraint.

Since constraints are now just types, existing type system features on type terms, such as synonyms and families, can be reused on constraints. Thus we can now define constraint synonyms via standard type synonms e.g.

type ShowNum a = (Num a, Show a)

And most excitingly, constraint families can be defined via type families of return kind Constraint. Our previous example can be written:

class Expr sem where
    type Pre sem a :: Constraint
    constant :: Pre sem a => a -> sem a
    add :: Pre sem a => sem a -> sem a -> sem a

instance Expr E where
    type Pre E a = Num a
    ...

Thus, Pre is a type family of kind * -> * -> Constraint. And it all works!

The constraint kinds extension can be turned on via the pragma:

{-# LANGUAGE ConstraintKinds #-}

Max has written about the extension over at his blog, which has some more examples, so do go check that out. As far as I am aware the extension should be hitting the streets with version 7.4 of GHC. But if you can’t wait it is already in the GHC HEAD revision so you can checkout a development snapshot and give it a whirl.

In my next post I will be showing how we can use the constraint kinds extension to describe abstract structures from category theory in Haskell that are defined upon subcategories. I already have a draft note about this if you can’t wait!


References


[1] Orchard, D. Schrijvers, T.: Haskell Type Constraints Unleashed, FLOPS 2010
, [author's copy with corrections] [On SpringerLink]

[2] Carrete, J., Kiselyov, O., Shan, C. C.: Finally Tagless, Partially Evaluated, APLAS 2007


by dorchard at September 24, 2011 05:01 PM

September 23, 2011

Jason Dagit

Galois offers Haskell course: The Tao of Functional Programming

This year Galois will be offering a course in professional Haskell development. Details on our main website: http://corp.galois.com/haskell-course

by dagitj (noreply@blogger.com) at September 23, 2011 04:38 PM

September 21, 2011

Joyride Laboratories

Nikki 0.4: cannons, sounds, sharing and downloading levels

Download and Upload Nikki Levels!

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="270" src="http://www.youtube.com/embed/ubYm7JG3CHc" width="531"></iframe>

Video: 0.4 changes [Vimeo | YouTube]

We released Nikki and the Robots 0.4.0 (0.4.1.1 actually), which you can download here or via in-game updater.

The new features are:

  • New robot: Cannon! It shoots cannon balls that disappear after 10 seconds
  • Players can download and upload levels in the main/editor menus
  • Author name and license agreement added for saving/uploading levels
  • Sounds added for menu, jumping, buttons and batteries
  • Last level switch to be pressed now blinks green
  • Added "transient" switches (won't stay pressed)

Some news not directly related to the release:

by Iwan at September 21, 2011 10:00 PM

September 18, 2011

David Amos

Commutative Algebra and Algebraic Geometry


Last time we saw how to create variables for use in polynomial arithmetic. This time I want to look at some of the things we can do next.

First, let's define the variables we are going to use:

> :l Math.CommutativeAlgebra.GroebnerBasis
> let [t,u,v,x,y,z,x',y',z'] = map glexvar ["t","u","v","x","y","z","x'","y'","z'"]


So now we can do arithmetic in the polynomial ring Q[t,u,v,x,y,z,x',y',z']. For example:

> (x+y)^2
x^2+2xy+y^2


The branch of mathematics dealing with the theory of polynomial rings is called commutative algebra, and it was "invented" mainly in support of algebraic geometry. Algebraic geometry is roughly the study of the curves, surfaces, etc that arise as the solution sets of polynomial equations. For example, the solution-set of the equation x^2+y^2=1 is the unit circle.

If we are given any polynomial equation f = g, then we can rewrite it more conveniently as f-g = 0. In other words, we only need to track individual polynomials, rather than pairs of polynomials. Call the solution set of f = 0 the zero-set of f.

Sometimes we're interested in the intersection of two or more curves, surfaces, etc. For example, it is well known that the hyperbola, parabola and ellipse all arise as "conic sections" - that is, as the intersection of a cone with a plane. So define the zero-set of a collection (or system) of polynomials to be the set of points which are zeros of all the polynomials simultaneously. For example, the zero-set of the system [x^2+y^2-z^2, z-1] is the unit circle x^2+y^2=1 situated on the plane z=1.


Okay, so how can commutative algebra help us to investigate curves and surfaces? Is there a way for us to "do geometry by doing algebra"? Well, first, what does "doing geometry" consist of? Well, at least some of the following:
- Looking at the shapes of curves and surfaces
- Looking at intersections, unions, differences and products of curves and surfaces
- Looking at when curves or surfaces can be mapped into or onto other curves or surfaces
- Looking at when two different curves or surfaces are equivalent, in some sense (for example, topologically equivalent)

(That phrase "curves and surfaces" is not only clumsy but also inaccurate, so from now on I'll use the proper term, "variety", for the zero-set of a system of polynomials, whether it's a set of isolated points, a curve, a surface, some higher dimensional thing, or a combination of some of the preceding.)

So can we do all those things using algebra? Well, let's have a go.

Let's start by looking at intersections and unions of varieties (remember, that's just the fancy name for curves, surfaces, etc.).

Well, we've already seen how to do intersections. If a variety V1 is defined by a system of polynomials [f1...fm], and a variety V2 is defined by [g1...gn], then their intersection is defined by the system [f1...fm,g1...gn] - the zero-set of both sets of polynomials simultaneously. We'll call this the "sum" of the systems of polynomials. (Note to the cognoscenti: yes, I'm really talking about ideals here.)

sumI fs gs = gb (fs ++ gs)

Don't worry too much about what that "gb" (Groebner basis) call is doing. Let's just say that it's choosing the best way to represent the system of polynomials. For example:

> sumI [x^2+y^2-z^2] [z-1]
[x^2+y^2-1,z-1]


Notice how the gb call has caused the first polynomial to be simplified slightly. The same variety might arise as the zero-set of many different systems of polynomials. That's something that we're going to need to look into - but later.

Okay, so what about unions of varieties. So suppose V1 is defined by [f1...fm], V2 is defined by [g1...gn]. A point in their union is in either V1 or V2, so it is in the zero-set of either [f1...fm] or [g1...gn]. So how about multiplying the polynomials together in pairs. That is, let's look at the system [fi*gj | fi <- fs, gj <- gs]. Call the zero-set of this system V. Then clearly, any point in either V1 or V2 is in V, since we then know that either all the fs or all the gs vanish at that point, and hence so do all the products. Conversely, suppose that some point is not in the union of V1 and V2. Then there must exist some fi, and some gj, which are non-zero at that point. Hence there is an fi*gj which is non-zero, so the point is not in V.

This construction is called, naturally enough, the product of the systems of polynomials.

productI fs gs = gb [f * g | f <- fs, g <- gs]


> productI [x^2+y^2-z^2] [z-1]
[x^2z+y^2z-z^3-x^2-y^2+z^2]


Just in case you're still a little unsure, let's confirm that a few arbitrary points in the union are in the zero-set of this polynomial:

> eval (x^2*z+y^2*z-z^3-x^2-y^2+z^2) [(x,100),(y,-100),(z,1)]
0
> eval (x^2*z+y^2*z-z^3-x^2-y^2+z^2) [(x,3),(y,4),(z,5)]
0


The first expression evaluates the polynomial at the point (100,-100,1), an arbitrary point on the plane z=1. The second evaluates at (3,4,5), an arbitrary point on the cone x^2+y^2=z^2. Both points are in the zero-set of our product polynomial.

Since we're in the neighbourhood, let's have a look at the other conic sections. First, let's rotate our coordinate system by 45 degrees, using the substitution x'=x+z, z'=z-x. (Okay, so this also scales - to save us having to handle a sqrt 2 factor.)



> let cone' = subst (x^2+y^2-z^2) [(x,(x'-z')/2),(y,y'),(z,(x'+z')/2)]
> cone'
-x'z'+y'^2


In these coordinates, the intersection of the cone with the plane z'=1 is the parabola x'=y'^2:

> sumI [cone'] [z'-1]
[y'^2-x',z'-1]


Alternatively, the intersection with the plane y'=1 is the hyperbola x'z'=1:

> sumI [cone'] [y'-1]
[x'z'-1,y'-1]


Okay, so we've made a start on seeing how to do geometry by doing algebra, by looking at unions and intersections of varieties. There's still plenty more to do. We mustn't forget that we have some unfinished business: we need to understand when different polynomial systems can define the same variety, and in what sense the gb (Groebner basis) function finds the "best" representation. That will have to wait for another time.

Incidentally, for the eval and subst functions that I used above, you will need to take the new release HaskellForMaths v0.4.0. In this release I also removed the older commutative algebra modules, so I revved the minor version number.

by DavidA (noreply@blogger.com) at September 18, 2011 06:47 PM

Conrad Parker

Iteratees at Tsuru Capital

Tsuru Capital is a small company. We build our internal systems for live trading and offline analysis in Haskell, and we're proud to be sponsoring ICFP 2011. We use iteratees throughout our systems, and have actively encouraged all our staff to contribute changes upstream and participate in community design discussions. By being part of the open source community and taking part in peer-review, we all end up with better software.

Over time various Tsuru staff members have worked on tools using iteratees, including (grepping the CONTRIBUTORS files): Bryan Buecking, Michael Baikov, Elliott Pace, Conrad Parker, Akio Takano, and Maciej Wos. There's been some lively discussions and many small patches providing functions that we use in production every day.

Last year Conal Elliott provided some mentoring to Tsuru staff, during which we worked through a denotational semantics for iteratees. This resulted in discussions on both the iteratee project list and haskell-cafe about Semantics of iteratees, enumerators, enumeratees.

By using iteratees in production we've contributed various simple but practical functions, including:

  • enumFdFollow, an enumerator (data source) which allows you to process the growing tail of a log file as it is being written.
  • ioIter, an iteratee that uses an IO action to determine what to do. Typically this is action involves some user interaction, such as a user issuing commands like play/pause/next/prev.
  • ListLike functions last (an iteratee that efficiently returns the last element of a stream), mapM_ and foldM.
  • mapChunksM_, a more efficient version of mapM_ that operates on the underlying chunks, eg. logger = mapChunksM_ (liftIO . print).
  • takeWhile, and its enumeratee variant takeWhileE


  • endianRead8, an iteratee for reading 64bit values with a given endianness. I've used this in ght as well as an internal project.

Stream conversion We've done quite a bit of work on stream conversion, as we use a few different layers of data processing. The iteratee architecture allows you to isolate the data source, conversion and processing functions; much of what we've worked on involves ensuring the converters (enumeratees) can control or translate control messages, so that commands like "seek" do not get lost. We've also built combinators to simplify the task of creating new stream converters.
  • convStateStream, which converts one stream into another while continually updating an internal state. Importantly for variable bitrate binary data, it can produce elements of the output stream from data that spans stream chunks.
  • (>) and (<). These allow stream converters to be composed without rewriting boilerplate. Jon Lato gives a good example using these in the StackOverflow answer to Attoparsec Iteratee.
  • zip, zip[345], sequence_ for using multiple iteratees to process a single stream instance, and (for zip*) collecting the results.
  • eneeCheckIfDone*: This family of functions (eneeCheckIfDoneHandle, eneeCheckIfDonePass, eneeCheckIfDoneIgnore) can be used with
    unfoldConvStreamCheck to make a version of unfoldConvStream which respects seek messages.


Parallel stream processing We often want to do multiple unrelated analysis tasks on a data stream. Whereas sequence_ takes a list of iteratees to run simultaneously and handles each input chunk by mapM across that list, psequence_ runs each input iteratee in a separate forkIO thread. For a real-world example, see Michael Baikov's post about psequence, psequence_, parE, parI.

Thanks

Thanks to John Lato for consistently and reliably maintaining the iteratee package, providing thoughtful feedback and graciously suggesting improvements.

<script src="http://reddit.com/static/button/button1.js" type="text/javascript"></script>

by Conrad Parker (noreply@blogger.com) at September 18, 2011 09:10 AM

September 15, 2011

Bjorn Buckwalter

Yesod Cookbook: Internationalized Form

I posted the following example of an internationalized form to the Yesod Cookbook a few days ago. Yesod has a pretty awesome library for programming with forms and also a flexible but currently under-documented system for internationalization. However, how the two interact, while straightforward, might not be immediately obvious to the newcomer. Hence the motivation for the below example which shows how to internationalize both custom messages and Yesod's built-in FormMessages (mostly form error messages).

Familiarity with Yesod at the level of the Yesod Book is assumed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
{-

INTERNATIONALIZED FORM

Below is an example of how to internationalize a Yesod application,
including internationalization (i18n) of a form and Yesod's built-in
form error messages.

In the example we create a simple site that will show messages in
Swedish or English depending on the browser's preferred language.
If neither Swedish or English are preferred languages the site will
default to Swedish.

Familiarity with Yesod at the level of the Yesod book is assumed,
comments are concentrated to the parts relevant to i18n.

-}

{-# LANGUAGE QuasiQuotes
           , TemplateHaskell
           , MultiParamTypeClasses
           , OverloadedStrings
           , TypeFamilies
  #-}

import Yesod
import Yesod.Form.I18n.Swedish
import Control.Applicative ((<$>), (<*>))
import Data.Text (Text)


-- Our foundation data type.
data MyApp = MyApp

mkYesod "MyApp" [parseRoutes|
/    RootR GET
|]

instance Yesod MyApp where
    approot _ = ""


-- Here we provide internationalization of Yesod's built in form
-- messages (mostly error messages). For brevity it is assumed that
-- Swedish translations are exported as 'swedishFormMessage' by the
-- module Yesod.Form.I18n.Swedish (an implementation can be copied
-- from https://gist.github.com/1209328).
instance RenderMessage MyApp FormMessage where
    renderMessage _ []        = swedishFormMessage -- Default to Swedish
    renderMessage _ ("sv":ls) = swedishFormMessage
    renderMessage _ ("en":ls) = defaultFormMessage -- English
    renderMessage m (_   :ls) = renderMessage m ls

-- Next we define the custom messages present on the site and their
-- rendering functions for different languages.
data Msg = Model
         | Year
         | Please

-- Rendering function for English.
renderEnglish Model  = "Model"
renderEnglish Year   = "Year"
renderEnglish Please = "Please fill in your car's details"

-- Rendering function for Swedish.
renderSwedish Model  = "Modell"
renderSwedish Year   = "Årgång"
renderSwedish Please = "Vänligen fyll i uppgifterna för din bil"

-- The instance used to select the appropriate rendering function.
-- This is almost identical to the instance for FormMessage above.
instance RenderMessage MyApp Msg where
    renderMessage _ []        = renderSwedish -- Default to Swedish
    renderMessage _ ("sv":ls) = renderSwedish
    renderMessage _ ("en":ls) = renderEnglish
    renderMessage m (_   :ls) = renderMessage m ls


-- The data model.
data Car = Car
    { carModel :: Text
    , carYear :: Int
    }
  deriving Show

-- In our form we use our messages Model and Year as field labels.
carAForm :: AForm MyApp MyApp Car
carAForm = Car
    <$> areq textField (fs Model) Nothing
    <*> areq intField  (fs Year)  Nothing
    where
        fs msg = FieldSettings msg Nothing Nothing Nothing

carForm :: Html -> Form MyApp MyApp (FormResult Car, Widget)
carForm = renderTable carAForm


-- Our handler just shows the form, with submitted values pre-filled.
-- Here we also use the Please message.
getRootR :: Handler RepHtml
getRootR = do
    ((_, widget), enctype) <- runFormGet carForm
    defaultLayout [whamlet|
        <p>_{Please}
        <form method=get action=@{RootR} enctype=#{enctype}>
            <table
                ^{widget}
            <p><input type=submit>
        |]


-- | Launch the app on port 3000.
main :: IO ()
main = warpDebug 3000 MyApp

Flattr this

by Björn Buckwalter (noreply@blogger.com) at September 15, 2011 03:26 PM

September 11, 2011

Clifford Beshers

Two Degrees from 9/11

On the morning of September 11, 2001, I didn’t board a bus to New York City, and in not doing so, buffered myself from a world of hurt.

I wasn’t clever or prescient, just lucky. I was at the bus stop, I’d seen the bus and flagged it down. But just then, two women jogging by stopped and asked if I had heard what happened. I wanted to keep eye contact with the driver, but the anxiety in their voices made me turn and listen. I glanced back to the driver as he slowed, saw him looking for confirmation, but I didn’t signal him again and he drove on.

I went back home, turned on the TV and soon saw the first tower fall. Had those ladies not happened by, I might well have seen that tower fall from the bus instead.

One friend was not so lucky. He was at jury duty that morning, just ten blocks away from the towers. He saw both towers fall as he walked back to his office in SoHo, breathing the first fumes of the fires, crying uncontrollably at the insanity. Days later, he discovered that a good friend and colleague had been high up in a tower, unable to evacuate. That same victim was actually the boss of another friend of mine, who then had to step up and fill that position.

Another friend was a SCUBA instructor who had trained dozens of NY firefighters to dive. Over a dozen of his students died in the collapse.

Earlier that year, I had attended the wedding of a diver I knew, held at Windows on the World, the restaurant at the top of the north tower. He and his new bride were devastated that so many of the restaurant staff they had worked so closely with to plan the event were lost.

It seemed that everyone lost someone, except for me. My connections were always indirect, like some tragic form of the Kevin Bacon game where I was always degree two. I do not know how much that separation lessened the blow, but it was significant.

On Friday, I went to the NJ side of the Hudson River near the Lincoln Tunnel to see the damage for myself. Though still several miles away, the plume of smoke was trivial to spot, and the lack of that distinctive skyline was like a splinter in my mind.

Still, I did not find myself overwhelmed by anger and grief, like those more closely linked. I looked at New York City, not the way you look when you’ve grown up in a place, but with the eyes of a tourist. The New York metropolitan area is huge. Standing on the palisades, I could see a dense urban landscape, seemingly extending to infinity in all directions.

Suddenly, all I could think of was how little physical damage the attacks had done. Perspective has that effect. The WTC towers were so huge, their presence swamped the mind, especially as one got closer. Just standing at the base and looking up, it felt like they were arching over you instead of going straight up. Their destruction was equally overwhelming. But from where I stood on the palisades, with two degrees of separation in my pocket, I could focus on the thousands of huge buildings, housing millions of people for miles around and see that they were all fine.

I’m not saying the impact wasn’t real, even for someone of degree two. Business dried up and I took advantage of a job offer in San Diego, where I’ve been ever since. All my friends mentioned above stayed and went through some hard times. I sometimes wonder, if I had caught that bus, would I have been degree one, and what might have been different as a result.

by Clifford Beshers (noreply@blogger.com) at September 11, 2011 08:45 AM

Cartesian Closed Comic

September 02, 2011

Alejandro Serrano Mena

EclipseFP 2.1.0 released!

As JP Moresmau points in this blog entry, a new release of EclipseFP is out! It includes bug fixes and new functionality, including my work during the Summer of Code.

These past days JP has been working very hard to fix some bugs my code had, and added support for Unicode characters in Scion Browser (which means that you can browse base-unicode-symbols, for example), so I'm really grateful to him.

Apart from that, the new EclipseFP website is out and contains information about how to install this plug-in your system. If you find any bug, don't forget to tell us so we can improve EclipseFP as much as possible!

As a final note, next Monday 5 of September I'll be speaking about EclipseFP in the beautiful city of A Coruña, in Spain, during the Taller de Programación Funcional.

by Alejandro Serrano (noreply@blogger.com) at September 02, 2011 03:20 PM

September 01, 2011

Chung-chieh Shan

Goodbye Rutgers

“FREE BOOKS! by taking them away” I wrote on my office door. I had an hour before my next appointment to reduce my life at Rutgers to a carry-on suitcase. So I had to leave behind everything that I can buy again, even those volumes soaked with memories: my first linguistics class, my first book chapter, conferences where colleagues blew me away with their wit and intellect. You know, you’re all mentors to me.

Sorting the shelves gave me the thrill of picking an old ripe scab. Like a sudden disaster that engulfs your house, like a subway door that closes after you and shuts out a latecomer, the scythe made me declare which arm belongs to my memories and which leg belongs to me. Highly recommended.

After all, books (like children and love) are better free.

This fall semester, I’m excited to be visiting Cornell linguistics, home of Mats Rooth’s computational linguistics lab. For excitement in the spring, I plan to visit Tsukuba computer science, home of Yukiyoshi Kameyama’s programming logic group. For both of these opportunities, I am grateful. After them, all bets are off. I want to keep learning and teaching, whatever that means. Maybe it’s time for me to work on a new citizenship. I’m open to suggestions.

The sun is brilliant. I didn’t just run into an old friend yesterday. I was also delighted to experience Bollywood for the first time, to get a tooth crowned for the first time. I won’t just meet up with an older and younger friend tomorrow. I will also be delighted to hear a new talk, to find a new thought.

September 01, 2011 06:09 AM

August 28, 2011

Holden Karau

Automatic spelling corrections on Github

English has never been one of my strong points (as is fairly obvious by reading my blog), so my latest side project might surprise you a bit. Inspired by the results of tarsnap’s bug bounty and the first pull request received for a new project(slashem - a type safe rogue like DSL for querying solr in scala) I decided to write a bot for github to fix spelling mistakes.



The code its self is very simple (albeit not very good, it was written after I got back from clubbing @ JWZ’s club [DNA lounge]). There is something about a lack of sleep which makes perl code and regexs seem like a good idea. If despite the previous warnings you still want to look at the code https://github.com/holdenk/holdensmagicalunicorn is the place to go. It works by doing a github search for all the README files in markdown format and then running a limited spell checker on them. Documents with a known misspelled word are flagged and output to a file. Thanks to the wonderful github api the next steps is are easy. It forks the repo and clones it locally, performs the spelling correction, commits, pushes and submits a pull request.



The spelling correction is based on Pod::Spell::CommonMistakes, it works using a very restricted set of misspelled words to corrections.



Writing a “future directions” sections always seems like such a cliche, but here it is anyways. The code as it stands is really simple. For example it only handles one repo of a given name, and the dictionary is small, etc. The next version should probably also try and only submit corrections against the conical repo. Some future plans extending the dictionary. In the longer term I think it would be awesome to attempt detect really simple bugs in actual code (things like memcpy(dest,0,0)).



You can follow the bot on twitter holdensunicorn .



Comments, suggestions, and patches always appreciated. - holdenkarau (although I’m going to be AFK at burning man for awhile, you can find me @ 6:30 & D)

by Holden Karau (noreply@blogger.com) at August 28, 2011 10:22 PM

August 26, 2011

Petr Rockai

soc reloaded: outcomes

This blog is kind of a final report for the summer. I had a progress report drafted for about two weeks, so let me paste that here just for the record. To read about the actual results, please skip to the “State of the Code” section below.

The Last Progress Report

This is the second (and last) progress update for this summer of code project. It was written something like a week before the pencil’s down, but I got disheartened and instead of finishing and posting, I went on to code some more. Here it goes…

Since my last report, I have decided to turn somewhat more radical again. The original plan was to stick with the darcs codebase and do most (all) of the work within that, based primarily on writing tests for the testsuite and not exposing anything of the new functionality in a user-visible fashion. I changed my mind about this. The main reason was that the test environment, as it is, makes certain properties hard to express: a typical test-suite works with assertions (HUnit) and invariants (QC). In this environment, expressing ideas like “the displayed patches are aesthetically pleasing” or “the files in the repository have reasonable shape” is impractical at best.

An alternative would have been to make myself a playground using the darcs library to expose the new code. But the fact is, our current codebase is entrenched in all kinds of legacy issues, like handling filenames and duplicated code. It makes the experimenter’s life harder than necessary, and it also involves rebuilding a whole lot of code that I never use, over and over.

All in all, I made a somewhat bold decision to cut everything that lived under Darcs.Patch (plus a few dependencies, as few as possible) into a new library, which I named patchlib, in the best tradition of cmdlib, pathlib and fslib. At that point, I also removed custom file path handling from that portion of code, removed the use of a custom Printer (a pretty-printer implementation) module and a made few other incompatible changes.

Of course, the testing code went along. The net result, at least for me, was an ability to build and test a much smaller piece of self-contained code. It also allowed me to experiment with APIs a bit, where those were used all over darcs, which made it, within the big codebase, impossible to advance without expending disproportionate amount of work on every change. Of course, part of that will be paid back when we decide to port darcs over to use patchlib.

I originally planned this report for the start of this week, but then I got caught in a big refactor of the ApplyMonad/Apply classes (again). The refactor was triggered by the need to pretty-print patches, which is not a completely easy task (made more complicated by the fact that UUIDs are meaningless for the user, so formatting patches without context is essentially useless now). Anyway, I am now much happier with how the ApplyMonad class looks (the ApplyMonadBase thing was genuinely hideous… good riddance). As a net result the ApplyMonad class and, even more importantly, the ApplyMonad transformer (used in applyTo{Tree,State} among other things) is substantially easier to use on the client side (while maybe very slightly harder on the provider side, it is also much clearer, IMHO). Overall win.

As for formatting and summarising patches, I have created a new Display class (I plan to nuke the existing viewing classes more or less, when I manage to make meaningful Display instances for V1 Prim). The API lets you format patches based on their ending or starting context. Since bits of the patches need to be fetched from the hashed store, the display needs to run in a LoadMonad. Since any reasonable patch formatter also needs to pass state around (and since the actual type of state passed around will be different for different Prim implementations, we hide the state in a type family of monad transformers; this also opens the option to use something else than StateT when appropriate).

(This is the end of the “progress” post. The remainder more or less describes the end state of the project.)

State of the Code

You can look at the code in my darcs repositories. Specifically, to play around with a “demo”, you need pathlib, fslib, patchlib, cmdlib and gorsvet. A bit more about the individual libraries:

  • pathlib is small library for handling file paths; it uses strict bytestring representation, and adds a certain amount of type safety and a few form invariants
  • fslib is the successor of hashed-storage; it deals with accessing the filesystem in an efficient manner, with good support for hashing files and using the hashes for efficient comparisons
  • patchlib, as outlined above, came into existence as a “fork” of the Darcs.Patch hierarchy, add and take some; the code is self-contained, and has a set of QC/HUnit tests, although these definitely need extending to cover more of the functionality (and some of the functionality needs to be removed)
  • cmdlib is my commandline parsing library, and is needed by the test client gorsvet (see next point)
  • gorsvet is an experimental client using version 3 primitive patches, and a very simple hashed repository format; its main purpose is to demo the implementation of V3 prims, in addition to existing QC coverage;

About patchlib

One of the main problems of the darcs codebase today is insufficient modularity, and patchlib is an experiment in an attempt to address that concern. Efforts to bring about significant improvement of the situation from “inside” (by restructuring existing code) have to date failed. Even though there have been local improvements, the overall problem stubbornly persists.

Hand in hand with modularity problems come issues with unclear and underspecified (both internal end external) APIs. Since the separation between different components of darcs is blurry at best, the pressure to introduce clean, testable interfaces is minimal. The external library, on the other hand, is forced to put up a presentable façade. Luckily, the Patch subhierarchy in darcs is, compared to remainder of darcs, in a fairly good shape in this respect.

Eventually, patchlib should provide leverage to work with darcs(-style) patches, including at least:

  • Implementation of primitive patches, both version 1 (as used by darcs 1 and 2), compatible with existing repositories, and version 3 (with better semantics and more efficient representation; subject of this SoC project).
  • Implementation of the “real” patches, in version 1 (as used by darcs 1) and version 2 (as used by darcs 2), and at some point, version 3 (of as yet unspecified properties).
  • Implementation of PatchInfo and “named” patches, which implement changesets in the darcs sense, and allow tracking their metadata. With PatchInfo (and to a limited extent, patch implementations themselves) goes a set of matchers and support code for interactive dependency-aware patch selection.

With a homogenous set of APIs, mostly mediated by type classes and appropriate instances, to:

  • Create primitive patches (mainly through diffing, but also by version-specific direct construction). The most important type class is Diff, defined in Data.Patch.Diff. The interface allows multiple equivalent representations of a single change, intended to provide an interface for heuristic detection of high-level patch types like hunk move or token replace.
  • Create real and named patches by building them from constituent primitives.
  • Store and load all types of patches. This role used to be filled by the ShowPatchBasic and ShowPatch classes, but is being superseded by Store and Load instances (the classes are provided by fslib for generic hash-based stores).
  • Format (and summarise) patches for user-friendly display, using relevant context to improve the legibility and usefulness of the output. The new class to achieve this is Display. Previously, this role was filled by ShowPatch, with its showContextPatch method (which is, however, significantly under-used by darcs). Since V3 primitive patches are substantially harder to read without context, the interface for user-level rendering of patches mediated by Display mandates context use.
  • Commute, invert (all patch types) and merge (“real” and “named” patches). The primary classes are Commute and Merge, currently both faithful copies of their darcs versions. They will however need to be changed to allow these operations access to a LoadMonad, for the benefit of delayed, on-demand loading of extensive text data (applies to hunks at the moment). Currently, the Commute class allows this, by substituting the Maybe monad for a CommuteMonad constraint, when compared to the original darcs definition.
  • Load, store and convert (to/from legacy format) PatchInfo objects that are used to track metadata in Named patches.

About V3 Prims in patchlib

The version 3 primitives in their current incarnation in patchlib have the following traits (based on the list from the proposal):

  • File content and file location/existence are tracked separately, tied together through universally-unique identifiers (of 256 bits). This avoids a number of conflict scenarios and may allow further novel features, like non-history-disruptive project splits and merges or subtree “checkouts”.
  • Hunk content is detached from the hunk patch itself through a hash (unless shorter than twice the length of the hash representation). This makes the patch files themselves very compact and allows most commute rules to avoid loading the hunk content altogether (improving speed and reducing memory footprint). It also makes handling of binary files vastly more efficient.
  • The hunks are byte-based, making it substantially more efficient to apply a patch to a file, since no newline scanning needs to happen.
  • The same hunk format that is used for text files is reused for binary hunks, basically encoding a range substitution operation on the binary file. A binary delta algorithm can be plugged in to compute more efficient binary hunks, although even full content replace is much cheaper than the binary patch type available in V1 prims.
  • A set of primitive patches implementing a hunk “move” operation is implemented, and is passing the generic commutation / application tests. Unfortunately, at this point there is no diffing algorithm to detect such moves, although one is planned for the future.
  • Hunk and hunk move patches are the only content-editing patches available to date. Further patch types can be added to the library without restriction as long as the format is not frozen (at least token replace and indentation patch types are planned before any such freeze). New object types and accompanying patch types may be added in later backward-compatible revisions of the format.

About gorsvet

Gorsvet is a toy implementation of a repository layer that uses V3 primitive patches. (Un)fortunately, the V3 prims violate fundamental assumptions made by the repository and command layers of darcs, which means that an integration is substantially more expensive than fits a single SoC project. However, as outlined above, it is useful to be able to play around with the V3 prim patches in a realistic environment. Therefore, gorsvet. I made it into a rather thin user shell based on cmdlib and a prototype repository layer. The UI more or less resembles darcs (without interactivity, since that’d be superfluous for a tool of this scope). You are of course welcome to try the experimental tool out: the online help should give you an idea how to use it.

One thing I would like to discuss a bit here is the repository format. Since the patch types are incompatible anyway, we are fully liberated from backward compatibility considerations. The next darcs repository format can be designed from scratch, keeping in mind the shortcomings of the previous two formats. The implementation in gorsvet is a peek at what the result might look like. Anyway, we still need a better “composite” patch layer, which represents conflicts (and sits one level above primitive patches), since the current (version 2) composite patches in darcs are quite unsatisfactory. That also means we have plenty of time to play around with the repository format, which is more or less independent of the composite patch format.

As for the format itself, I went for as simple as possible (but no simpler). So far, I have 2 files and a hashed store: .gorsvet/hashed is a sha256-based, uncompressed “dumping grounds” for stuff of all kinds. The files are (not implemented as of this writing) .gorsvet/index (with the same purpose as _darcs/index — fast, efficient working copy access) and .gorsvet/meta — a small set of root pointers. This has two very beneficial side effects: very atomic oupdates and transactional semantics (free transaction rollback). Compression and garbage collection of the hashed store can then be sorted out separately (and does not affect semnatics of the repository either way). There are currently 4 root pointers in meta: shadow, pristine, inventory and order. The pristine is the same thing as in darcs, shadow is a similar thing, but at any time reflects the current state of the working copy (it is automatically updated from working every time, before anything else happens with the repository). The reason is mainly that the working copy has entirely wrong semantics for diffing V3 primitive patches: most importantly, UUID tracking is implemented through shadow.

The remaining two root pointers represent two new data structures (and inventory is different from what darcs calls an inventory). The order simply lists patchinfo hashes (a handle for each patch that does not change through commutation) in the application order for the repository: in this sense, order replaces hashed_inventory known from darcs. It is pretty compact, but on its own also useless (since it give us no way to get the patches themselves). The inventory then, on the other hand, is an efficient map (currently a sorted pair list, written and read as a Data.Map) from patchinfo hashes to the current patch storage hashes. Moreover, the patchinfo objects are, unlike in darcs, stored in the hashed store as separate entities, and the patchinfo hash can be used to efficiently fetch the patchinfo itself. Therefore, the named patch can be assembled from inventory and order puts the patch into correct context.

Apart from storing and showing things, I have also implemented a “pull” command for gorsvet, but it’s currently fairly unusable, since any conflict automatically means failure (there is no layer to handle conflicts; we could use version 2 darcs patches, but I think it would constitute a dangerous slippery slope: we definitely want a more solid implementation, and also want to avoid a double transition).

Future Work

The obvious future work lies in the conflict handling. There are two main options in this regard: either re-engineer a patch-level, commute-based representation of conflicts (in the spirit of mergers and conflictors), as V3 “composite” patches, or alternatively, use a non-patch based mechanism for tracking conflicts and resolutions. It’s still somewhat early to decide which is a better choice, and they come with different trade-offs. Nevertheless, the decision, and the implementation, constitute a major step towards darcs 3.

The other major piece of work that remains is the repository format: in this area, I have done some research in both the previous and this year’s project, but there are no definitive answers, even less an implementation. I think we now have a number of good ideas on how to approach this. We do need to sort out a few issues though, and the decision on the conflict layer also influences the shape of the repository.

Each of these two open problems is probably about the size of an ambitious SoC project. On top of that, a lot of integration work needs to happen to actually make real use of the advancements. We shall see how much time and resources can be found for advancing this cause, but I am relatively optimistic: the primitive level has turned out fairly well, and to me it seems that shedding the shackles of legacy code sprawl can boost the project as a whole significantly forward.

(PS: While I agree, on the theoretical level, that nuking significant amounts of legacy code carries non-trivial risks, for a small volunteer project like darcs it is imperative to be fun. And a trench war against legacy code is not fun. Writing new things and exploring possibilities, on the other hand, is fun. Which means we need a bit more of the latter and a bit less of the former, even though the project sits in the more conservative camp — afterall, we handle rather precious data… Even when taking natural conservativism into account, a well-motivated, honest subsystem rewrite is better than a half-hearted, someone-has-to-do-it maintenance of a piece of code that everyone hates…)

August 26, 2011 11:40 AM

August 22, 2011

Mikhail Glushenkov

Parallelising cabal-install: Results

The parallel cabal-install GSoC project has reached the stage where it can be useful to a wider audience. This post describes the current state of the code, provides a guide to installing & using it, and outlines the roadmap for the near future.

Building

The parallel version of cabal-install is available from this Darcsden repository. Since this branch contains changes to both cabal-install and the Cabal library itself, you are advised to use cabal-dev for building:

$ darcs get http://darcsden.com/23Skidoo/Cabal
$ cd Cabal
$ cabal-dev install cabal/ cabal-install/

This will install the new cabal executable to ./cabal-dev/bin/cabal. You can then move it to the location of your choosing - for example, replacing the cabal executable you have in $PATH (don’t forget to do a backup!).

Running

To install packages from Hackage in parallel, just run cabal install -jN, where N is the number of “jobs”, or worker threads. You will see something like the following:

$ cabal install -j2 transformers quickcheck
Resolving dependencies...
[1] Downloading QuickCheck-2.4.1.1...
[2] Downloading transformers-0.2.2.0...
[2] Configuring transformers-0.2.2.0...
[1] Configuring QuickCheck-2.4.1.1...
[2] Building transformers-0.2.2.0...
[2] Preprocessing library transformers-0.2.2.0...
[1] Building QuickCheck-2.4.1.1...
[1] Preprocessing library QuickCheck-2.4.1.1...
[2] [ 1 of 21] Compiling Data.Functor.Product ( Data/Functor/Product.hs,
dist/build/Data/Functor/Product.o )

...

[2] [21 of 21] Compiling Control.Monad.Trans.Writer.Strict (
Control/Monad/Trans/Writer/Strict.hs,
dist/build/Control/Monad/Trans/Writer/Strict.o )
[install] Installing library in
/home/cabal-test/.cabal/lib/transformers-0.2.2.0/ghc-7.0.4
[install] Registering transformers-0.2.2.0...

...

[1] [13 of 13] Compiling Test.QuickCheck.All ( Test/QuickCheck/All.hs,
dist/build/Test/QuickCheck/All.o )
[install] Installing library in
/home/cabal-test/.cabal/lib/QuickCheck-2.4.1.1/ghc-7.0.4
[install] Registering QuickCheck-2.4.1.1...

Here, all output prefixed with [1] and [2] comes from the worker threads, and everything prefixed with [install] comes from the install thread (installation step is serialised to avoid races).

Additionally, you can play with the +RTS -N option, though it shouldn’t produce a noticeable speed difference.

A good way to stress the parallel install code is to back up your ~/.ghc and ~/.cabal directories and then run cabal install world:

# Backing up old data:
$ mv ~/.ghc ~/.ghc.bak
$ mv ~/.cabal ~/.cabal.bak

# Recreating the ~.cabal directory:
$ cabal update
$ cp ~/.cabal.bak/world ~/.cabal/

# Rebuilding the world (N = number of concurrent jobs):
$ cabal install -jN world

If you encounter any bugs, please drop me a line in the comments or submit a bug report at the Cabal bug tracker.

Future work

First of all, the parallel patches need to be merged into the mainline repository, which may take some time.

Next, there are possibilities for improvement. Current version of the code works at the package granularity (a package is always built by a single thread), which doesn’t work so well for giant packages like Agda and Darcs. We can get better speedup by ditching ghc --make and instead compiling each module separately with ghc -c; the dependency graph can be extracted with ghc -M. This should be also integrated with the build command.

Third, there is the question of other compilers (e.g. Hugs). At the moment, only GHC is supported by the parallel install code, but it should be possible to adapt it to work with other compilers. This is not considered a top priority, though.

Acknowledgements

Thanks to my mentor, Johan Tibell, for code reviews and helpful comments; and to Google for sponsoring my work during the summer.

August 22, 2011 12:00 AM

August 21, 2011

Brandon Simmons

A brief tutorial-introduction to ‘fclabels’ 1.0

Sebastiaan Visser et al. last week announced the release of v1.0 of fclabels. This is a short introduction to the changes for people who are already somewhat familiar with the package.

Que? The fclabels package is a haskell library providing “first class labels” for haskell data types and some Template Haskell code for automatically generating said labels. This makes for a more composable and much more powerful alternative to haskell’s built-in record syntactic sugar. Here is a good introductory tutorial.

The new version features (besides some better names) a type for lenses that can fail. S.V. et al. added this functionality in a brilliant and mostly (except for the name changes) backwards compatible way. We’ll see how it all works now.

We’ll prepare a module for fclabels in the usual way:


{-# LANGUAGE TemplateHaskell, TypeOperators #-}
import Control.Category
import Data.Label
import qualified Data.Label.Maybe as M
import Prelude hiding ((.)id)

The only odd thing above is our qualified import of Data.Label.Maybe which provides getters, setters etc. that can fail.

We want to generate lenses for the following type, so we do the underdash thing:


36 data Example = X { _int :: Int , _char :: Char}
37              | Y { _int :: Int , _example :: Example }

Notice how _int is a valid record for both constructors X and Y, where the other two are partial functions.

Lastly, the usual splice for generating labels:


39 $(mkLabels[''Example])

Let’s load our file into GHCi and play a little:

Prelude> :l test.hs
[1 of 1] Compiling Main             ( test.hs, interpreted )
Ok, modules loaded: Main.
*Main> get int $ X 1 'a'
1
*Main> M.get int $ X 1 'a'
Just 1
*Main> M.get char $ X 1 'a'
Just 'a'
*Main> M.get example $ X 1 'a'
Nothing
*Main> get example $ X 1 'a'

<interactive>:1:5:
    No instance for (Control.Arrow.ArrowZero (->))
      arising from a use of `example'
    Possible fix:
      add an instance declaration for (Control.Arrow.ArrowZero (->))
    In the first argument of `get', namely `example'
    In the expression: get example
    In the expression: get example $ X 1 'a'
</interactive>

Above we saw that:

  • the total int label could be used in both get and Data.Label.Maybe.get
  • M.get on our “partial lenses” (example and char) safely returned a Maybe value.
  • pure get used with partial lenses doesn’t type check

We need to look at some types (note I’ve cleaned up some of these type signatures, yours may look uglier):

*Main> :t get
get :: (f :-> a) -> f -> a
*Main> :t M.get
M.get :: (f M.:~> a) -> f -> Maybe a
*Main> :info (:->)
type (:->) f a = Data.Label.Pure.PureLens f a
  	-- Defined in Data.Label.Pure
*Main> :info (M.:~>)
type (M.:~>) f a = Data.Label.Maybe.MaybeLens f a
  	-- Defined in Data.Label.Maybe

You can see above that setters and getters are monomorphic. But how then were we able to use int in both get and M.get? Time to look at the types of our TH-generated lenses:

*Main> :t int
int
  :: Control.Arrow.Arrow (~>) => Lens (~>) Example Int
*Main> :t char
char
  :: (Control.Arrow.ArrowZero (~>),
      Control.Arrow.ArrowChoice (~>)) =>
     Lens (~>) Example Char
*Main> :t example
example
  :: (Control.Arrow.ArrowZero (~>),
      Control.Arrow.ArrowChoice (~>)) =>
     Lens (~>) Example Example

Whoah! Lenses themselves are polymorphic in the Arrow class; partial lenses have additional restrictions of ArrowZero and ArrowChoice allowing for failure and choice.

This is wizardry of the highest order.

Notice how we can still compose lenses freely, whether partial or total:

*Main> :t (.)
(.) :: Category cat => cat b c -> cat a b -> cat a c
*Main> M.get (int . example) (Y 1 (X 2 ‘a’))
Just 2

A note on mkLabelsNoTypes

I had been using the noTypes variation to generate lenses that I was exporting in a module, because the TH-assigned type sigs looked pretty nasty. If you do this with >1.0 you will run into the monomorphism restriction and ambiguous type variables.

Instead I would suggest defining exported lables by hand and giving them a nicer looking (or monomorphic if you prefer) type sig.

You can even let fclabels generate the code and just paste it in, e.g.:

Prelude> :set -ddump-splices
Prelude> :l test.hs
[1 of 1] Compiling Main             ( test.hs, interpreted )
test.hs:1:1: Splicing declarations
    mkLabels ['Example]
  ======>
    test.hs:40:3-21
    char ::
      forall ~>[a1kk]. (Control.Arrow.ArrowChoice ~>[a1kk],
                        Control.Arrow.ArrowZero ~>[a1kk]) =>
      Lens ~>[a1kk] Example Char
    {-# INLINE[0] CONLIKE char #-}
    char
      = let
          c[a1kl]
            = (Control.Arrow.zeroArrow Control.Arrow.||| Control.Arrow.returnA)
        in
          Data.Label.Abstract.lens
            (c[a1kl]
           . Control.Arrow.arr
               (\ p[a1km]
                  -> case p[a1km] of {
                       X {} -> Right (_char p[a1km])
                       _ -> Left GHC.Unit.() }))
            (c[a1kl]
           . Control.Arrow.arr
               (\ (v[a1kn], p[a1ko])
                  -> case p[a1ko] of {
                       X {} -> Right (p[a1ko] {_char = v[a1kn]})
                       _ -> Left GHC.Unit.() }))
...

by jberryman at August 21, 2011 06:39 PM

Marco Túlio Gontijo e Silva

ghc -fvia-C and new binutils

I was trying to build the curl hackage package on Debian unstable with ghc 7.0.3, and one of its modules were failing to build. I searched for the error message and found this GHC ticket. As mentioned in the ticket, I had to downgrade binutils to 2.20. The version of binutils in sid is 2.21.

Another possibility would be to change curl to avoid using -fvia-C, but I didn't want to modify the package.

August 21, 2011 02:47 PM

Using XMonad with Netbeans (or other Java apps)

After testing Emacs, Eclipse, KDevelop and CodeBlocks for writing C++ code, I decided to stick with Netbeans. It seems to be the most simple to configure and yet full of features and plugins. I was having one problem with it that when I focused out the window, and then focused on it again, it would not really grab the focus, in the sense that I would not be able to type without first clicking with the mouse. At first I thought the problem was with Netbeans, then with the JRE. I tried using sun-java6-jre, I tried upgrading my openjdk-6-jre, and nothing worked. I searched a little bit more and got to a discussion about this problem in ion3, and that made me think that the problem could be related to the window manager I use, XMonad. After searching a bit about it, I found on the XMonad FAQ some work arounds for problems with Java apps, but they didn't solve my problem. Then, I found the solution on this bug report. I got the darcs version of xmonad and XMonadContrib, included takeTopFocus on logHook, and now it's working!

August 21, 2011 02:47 PM

Edward Amsden

Beer Part Two

So since my first post I've actually had occasion to try a couple of other beers. My buddy Ryan brought over a can of Labatt Blue Light Lime. I'm not sure I enjoy the addition of lime flavor to beer.

That same evening, I split a bottle of Blue Moon Summer Honey Wheat with him. That I did enjoy. Wheat beers comprise the majority of what I've had so far, and I must say they are growing on me. I don't think honey can help but improve most things. Beer turned out to be no exception.

Finally, since Ryan is leaving for Germany, another friend and I took him to the RIT-local bar: MacGregors. The selection for the three of us that evening was a pint each of Great Lakes Holy Moses (another wheat beer, with orange and coriander). Though it seemed a bit stronger than the Blue Moon, (the time separation is enough that direct comparison isn't possible) it was quite enjoyable alongside a burger on wheat bread.

That puts me up to date blogging about beer.

by Edward Amsden (noreply@blogger.com) at August 21, 2011 04:05 AM

August 18, 2011

Sam Anklesaria

Ending GSoC

As the firm pencils down date for Google Summer of Code is coming up Monday, I figure I'd better make sure everyone understands what I've done and how to use it.
  • GhcOptions Record
  • This record has fields for every GHC option used by cabal. It replaces the ad hoc lists of strings that were used previously. GhcOptions is also a Monoid instance, so one can mapped more specific options onto basic ones. To allow this instance, lots of fields are wrapped in All, Any, First, and Last constructors, which can be kind of a pain when filling the fields manually, but it's a small price to pay for composability.
  • Repl Function
  • This addition to the Cabal API will start an interpreter session. By default, it will load all files from the library component of a package. If the user passes specific files, the function will try to build the component that contains these files, failing if no such component can be found or if multiple components contain the files. If the user wants to load some specific component, this can be specified by the --component flag. Like other functions in the cabal API, there are number of hooks that the user can specify to change default interpretation behavior: preRepl, replHook, and postRepl. The names kind of speak for themselves.
  • GHCi Support
  • The repl function eventually calls an interpret function defined individually for each compiler. Currently, the only compiler with interpretation defined is GHC. GHC's interpret implimentation also defines a series of macros that overload the standard :load and :reload interpreter commands to call cabal repl --enable-reload on the specified component before loading so as to do whatever preprocessing is necessary.
  • Other Changes to the Cabal API
  • Because GHC's interpret function calls the cabal-install tool, I had to add cabal to the list of known programs (including ar, ghc, ghc-pkg, etc). Also, because interpretation does not necessarily require building every single package, the topologically sorted build order list used in building is no longer sufficient. I've added a compBuildMap field to LocalBuildInfo which maps components to their topologically sorted list of dependencies.
  • Cabal Repl
  • The cabal-install tool now has an repl command:
      Usage: cabal repl [FILENAME] [FLAGS]
            --enable-reload
        Enable file reloading from within an interpreter.
            --disable-reload
        Disable file reloading from within an interpreter.
            --component[=COMPONENT]
        The component of the cabal package to interpret
          Examples:
            cabal repl
          Only library component in the package
            cabal repl Examples/Foo.hs
          Specific file
            cabal repl Bar.hs --component=myexe
          File from specific component
            cabal repl --component=library
          Library contained in the package
        The usual flags apply as well, of course, and the --enable-reload option isn't really meant for anyone except cabal itself to use when reloading packages.

      Oh yeah. Here's the code, and here is the haddock stuff.

      by Anklesaria (noreply@blogger.com) at August 18, 2011 06:27 PM

      August 14, 2011

      Edward Amsden

      Granola

      I've been tweaking this through a few batches and this is what seems to work.

      • 3 cups rolled oats
      • 1/2 cup brown sugar
      • 1/2 cup butter
      • 1/3 cup honey

      Combine oats and brown sugar. Melt butter, stir in honey. Combine with dry ingredients and stir until oats are coated. Spread evenly on a cookie sheet with foil. Bake at 325F for 13-20 minutes (until just lighter than golden brown), stirring every 5-7 minutes.

      by Edward Amsden (noreply@blogger.com) at August 14, 2011 12:44 AM

      August 09, 2011

      Andrew Calleja

      4Blocks Source Code

      Today I decided to try my hand at releasing a cabal package with the 4Blocks code. I finally managed: http://hackage.haskell.org/package/4Blocks-0.2. (I made a newbie mistake in 0.1 so please ignore that release).

      Unfortunately the library can be compiled only with, as far as i know:

      • GHC 6.8.3
      • Gtk2hs 0.9.13

      The reasons why are in the README file in the cabal package, copy/pasted here:

      Some notes:
      - The game currently works only with GHC 6.8.3 due to its use of Gtk2hs 0.9.13.
      - I haven’t tested with anything later but it is likely to fail due the fact that later versions of Gtk2hs have a different system of handling events.
      - To make this project compatible with later versions of GHC and Gtk2hs two changes are required:
      - Remove the function “permutations” which was copied from a later GHC base library
      - Alter key-event handling to the version used by later Gtk2hs: some functions (in CommandKeys.hs) were simply introduced in order to disallow some of the keys used in the game (namely rotation) to trigger continuously when a key is held pressed. I believe this kind of behaviour can be managed automatically with Gtk2hs’ new event handling mechanism, however I haven’t had time to recode accordingly myself.
      - I hope to write a patch for this in the near future.

      So yeah, fairly old now, but I thought, I’d share the code all the same as some people have requested it earlier. Hopefully I will release the code with my AI later on after my project is over and done with in summer and also a patch to make it work with newer versions of GHC and Gtk2hs.

      I have to say I really enjoyed coding the game in Haskell and if you have any comments for me, regarding better ways to code stuff, silly things I did, or anything else please let me know by leaving a comment!


      by drewcalleja at August 09, 2011 09:58 PM

      Space Generals

      Hi all!

      It’s been more than 4 months since my last post! I’ve been really busy lately creating a website for a game called Space Generals which we are going to use for our research in game AI. The game itself is actually a turn-based one much in the style of Risk with a little twist which will allow us to study hierarchical AI and domain-specific embedded scripting languages. The website’s front-end is developed in Java using the GWT framework while its back-end, which carries out turn-processing and game-logic, is developed in Haskell. Here are some screenshots of the game:

      Login View Star Map View: galaxies and planets Planet View: continents and countries

      More information about the game and how to play can be found in the User Manual.

      Feel free to register and play the game. You need at least one more friend to play a game and up to 5 players can play together.  Also, if possible report any errors and provide suggestions using the game’s:

      Email Address

      Twitter

      Facebook Group

      Also, it would be of great interest to us for you to share your game-playing strategies using the above methods as we shall be using them to create different AIs using the DSEL we are working on at the moment.

      Thanks! We hope you enjoy the game!

      P.S. Currently only Mozilla Firefox and Google Chrome are supported.


      by drewcalleja at August 09, 2011 09:58 PM

      August 08, 2011

      Disciple/DDC

      How I learned to stop worrying and love de Bruijn indices

      When I started learning how to formalise languages in Coq, one of my initial stumbling blocks was choosing an approach to name binding. There are at least three fancy sounding contenders: Locally Nameless, Nominal Reasoning and Parametric Higher Order Abstract Syntax. Of course, you could also use good old de Bruijn indices, but proponents of fancy approaches warn about the large volume of lemmas regarding lifting and substitution you'll need to prove before you can start the "real work".

      Back in April I scratched around for about two weeks reading up on all the different approaches, trying to identify the "best" one (whatever that means). Then I read the following comment by Xavier Leroy on the POPLmark website. He had used the Locally Nameless approach, but had stated: "If the purpose is just to solve the challenge and be done with it, I would rather go for pure de Bruijn". That was enough to convince me to stop worrying and just get into it. I've spent about 250 hours now doing my own proofs, and I can report that using de Bruijn isn't really that bad. Yes, you need to prove some lemmas about lifting and substitution, but my entire response to naysayers is summed up henceforth: "The only computational operation that lambda calculus has is substitution, so don't pretend you can get away without proving something about it." Corollary: HOAS doesn't count.

      The Lemmas

      You need to know what the lemmas are though, and I advise cribbing them from someone else's proof. I learned them from Arthur Charguéraud's proof, but I don't know where they are from originally. You're more than welcome to copy them from mine.

      Once you know what the lemmas are, and how they work in STLC, extending them to new languages is straightforward. For reference, the following are the definitions of the lifting and substitution operators for STLC. Note that if you just want to prove Progress and Preservation for STLC, you don't actually need the lemmas. They're needed in bigger languages such as SystemF2 to show that type substitution is commutative. Nevertheless, for illustrative purposes we'll just stick with STLC expressions.

      Inductive exp : Type :=
      
      | XVar : nat -> exp
      | XLam : ty -> exp -> exp
      | XApp : exp -> exp -> exp.

      Fixpoint
      liftX (d: nat) (* current binding depth in expression *)
      (xx: exp) (* expression containing references to lift *)
      : exp
      := match xx with
      | XVar ix
      => if le_gt_dec d ix
      (* Index is referencing the env, so lift it across the new elem *)
      then XVar (S ix)

      (* Index is locally bound in the expression itself, and
      not the environment, so we don't need to change it. *)
      else xx

      (* Increase the current depth as we move across a lambda. *)
      | XLam t1 x1
      => XLam t1 (liftX (S d) x1)

      | XApp x1 x2
      => XApp (liftX d x1) (liftX d x2)
      end.

      Fixpoint
      substX (d: nat) (* current binding depth in expression *)
      (u: exp) (* new expression to substitute *)
      (xx: exp) (* expression to substitute into *)
      : exp
      := match xx with
      | XVar ix
      => match nat_compare ix d with
      (* Index matches the one we are substituting for. *)
      | Eq => u

      (* Index was free in the original expression.
      As we've removed the outermost binder, also decrease this
      index by one. *)
      | Gt => XVar (ix - 1)

      (* Index was bound in the original expression. *)
      | Lt => XVar ix
      end

      (* Increase the depth as we move across a lambda. *)
      | XLam t1 x2
      => XLam t1 (substX (S d) (liftX 0 u) x2)

      | XApp x1 x2
      => XApp (substX d u x1) (substX d u x2)
      end.

      Drawing the deBruijn environment

      Here is a nice lemma to get started.
      Lemma lift_lift
      
      : forall n m t
      , lift n (lift (n + m) t)
      = lift (1 + (n + m)) (lift n t).
      This is one of the lemmas you need for proving commutativity of substitution. Although it looks complicated at first glance, it will make significantly more sense when you know how to draw it.

      First, consider the standard typing judgement form:
      Env |- Exp :: Type
      
      Assume there are de Bruijn indices in the expression. Some of these indices point to lambdas in the expression itself, corresponding to locally bound variables, while some point to lambdas in the environment, corresponding to free variables. Here is an example:



      Suppose I apply (lift 2) to the above expression. Although it would be helpful for educational purposes to do this directly using the definition of 'lift' above (try it!), I'll tell you the shortcut instead. Firstly, if we assume that the definition of 'lift' is correct, none of the bound indices will be changed during the application. Ignoring bound indices, the only ones remaining are free indices. These are the indices that point into the environment.

      Now, although applying 'lift' to an expression may increment these free indices, instead of thinking about indices being incremented, it's easier to think about environment elements being shifted. I'll explain what I mean by that in a moment, but for now let's just do it...

      Here is the same expression as before, with the environment positions numbered in blue:


      Shifting the elements after position two yields the following:



      In the above diagram I've used '_' to represent a hole in the environment. This is a place where I could insert a new element, and be guaranteed that none of the indices in the expression point to this new element. This in turn means that if I apply (lift 2) to the expression, then insert a new element into the hole, then the resulting judgement is guaranteed to give me the same type as before (t5 in this case).

      Of course, applying 'lift' to the expression doesn't actually change the environment. In this example I've moved 't1' and 't2' to the left for illustrative purposes, but this is an operation separate from applying 'lift' to the expression itself. However, applying 'lift' does changes the indices, and these indices are pointers into the environment. If we like, we could instead think about moving the pointers (the arrow heads) instead of the actual environment elements. This is an important change in perspective. Instead of thinking about 'lift' as incrementing indices in the expression, we want to think about 'lift' as shifting environment pointers to the left.


      Let's look at our lemma again:
      Lemma lift_lift
      
      : forall n m t
      , lift m (lift (n + m) t)
      = lift (1 + (n + m)) (lift n t).

      As 't' is quantified, we need to prove this for all possible terms 't'. Using our change in perspective, this also means that we want to prove the lemma for all possible sets of environment pointers. Given the environment pointers for 't', the expression '(lift (n + m) t)' transforms them into a new set (by adding a hole at position n + m). Likewise the expression 'lift (1 + (n + m)) (lift n t)' transforms the environment pointers of 't' by adding a hole a position 'n', and then adding another hole at position '1 + (n + m)'. Here is a picture of the full situation. I've just drawn the contiguous set of environment pointers as rectangles, with the holes are marked in grey.


      In the first line we have set of environment pointers for 't'. Applying 'lift n' adds a hole a position 'n'. Alternatively, applying 'lift (n + m)' adds a hole at position '(n + m)'. Given 'lift n t', if we add another hole at (1 + n + m) we get the last set shown. Likewise, if we take the set 'lift (n + m) t' and add a hole at position 'n' then we also get the last line.

      That's it. That complicated looking lemma above can be drawn with a few rectangles, and provided the definition of 'lift' does what it's supposed to, the lemma is obviously true. For a language like SystemF2 you need about 5 lemmas concerning lifting and substitution. They're all easy to draw, and the proofs are straightforward once you understand the general approach. In the next post I'll show how to draw substitution, which works in a similar way.

      by Ben Lippmeier (noreply@blogger.com) at August 08, 2011 08:10 AM

      August 01, 2011

      Nicolas Wu

      The Conditional Choice Operator

      A recent reddit post asking for a library of conditional one-liners and combinators reminded me of one of my favourite operators: the conditional choice. Most programmers are used to the so-called McCarthy conditional:

      if p then x else y
      

      An alternative way of viewing this is as a ternary operator:

      x <| p |> y
      

      This operator is known as the conditional choice, but I’m told that it was introduced by Tony Hoare, so maybe naming it the Hoare conditional would make more sense (it makes an appearance in Hoare’s work on Communicating Sequential Processes and Unifying Theories of Programming, but I haven’t found any references to its original introduction).

      Laws

      Rendering conditonals as ternary operators makes it clear that there are a number of nice properties that hold true:

      • Idempotency

        x <| p |> x == x
        
      • Left-Identity

        x <| True |> y == x
        
      • Right-Identity

        x <| False |> y == y
        
      • Left-Distributivity

        x <| p |> (y <| q |> z) == (x <| p |> y) <| q |> (x <| p |> z)
        
      • Right-Distributivity

        (x <| p |> y) <| q |> z == (x <| q |> z) <| p |> (y <| q |> z)
        
      • Symmetry

        x <| p |> y == y <| not p |> x
        
      • Conjunction-Associativity

        (x <| p |> y) <| q |> z == x <| p && q |> (y <| q |> z)
        
      • Disjunction-Associativity

        x <| p |> (y <| q |> z) == (x <| p |> y) <| p || q |> z
        
      • Conjunction-Collapse

        x <| p |> (y <| p && q |> z) == x <| p |> z
        
      • Disjunction-Collapse

        (x <| p || q |> y) <| q |> z == x <| q |> z
        
      • Abiding (Interchange)

        x # y <| p |> v # w == (x <| p |> v) # (y <| p |> w)
        

      These laws are easily proved by considering the cases where p and q are True and False.

      Implementation

      In Haskell, implementing this operator is quite simple. First we’ll define the right bracket, which takes a predicate and a value x, and returns Nothing if the predicate is True, and returns Just x when it is False:

      > (|>) :: Bool -> a -> Maybe a
      > True  |> _ = Nothing
      > False |> y = Just y
      

      The left bracket is equivalent to fromMaybe, where the resulting value from the application of the right bracket (which evaluates the predicate) is consumed. If the result was Nothing, then we use the value x, otherwise we have Just y, and return y as the result.

      > (<|) :: a -> Maybe a -> a
      > x <| Nothing = x
      > _ <| Just y  = y
      

      Finally we give the operators low infixity precedence, and make them right associative:

      > infixr 0 <|
      > infixr 0 |>
      

      Defining the operator this way makes the ternary operator right associative, so that:

      x <| p |> y <| q |> z == x <| p |> (y <| q |> z)
      

      Right associativity here is useful so that reading from left to right, the result is the expression to the left of the first predicate that is true.

      August 01, 2011 12:00 AM

      July 29, 2011

      Mihai Maruseac

      Benchmarks

      Recently I discovered the joy of answering questions on Stack Overflow. I had a SO account for two years but I didn’t use it until a few weeks ago. This article started after receiving a little feedback on one of my answers.

      As you see, I was trying to give a benchmark for those two cases. Yet, it was reported as being an ugly hack and not a try at comparing two solutions. I post this short article here not because I wanted a place to cry that I was criticized but because I desired some advice on what should a good benchmark contain and how to write a proper one.

      Thanks :)


      by Mithrandir at July 29, 2011 09:26 PM

      July 18, 2011

      Mikael Vejdemo Johansson (Syzygy-)

      Some thoughts on fire photography

      First off: I would like to apologize to the readers I still have that I never get around to updating here. I am aware that I’m writing less than once I was planning to.

      Aaaaanyway.

      I have grown very interested in photography, as those of you who have seen my photo-a-day blog will have noticed. With the growing interest, I also notice a growing awareness of photography and of pictures, and I start thinking about how I would do things.

      Such as this last weekend. There was a wedding. There was a wedding celebration. At one point we ended up down at a beach, watching a fire show. A friend of mine was running out of space on his memory card, so left his camera with me and ran to get the next card.

      So I took some photos of the fire dancing (I haven’t gotten access to them myself just yet, they’ll be posted on my photo blog when I get hold of them). And I realized some things about how to photograph fire dancers. I’ll enumerate a few of me recent (though I am convinced, not original) insights here. Worth noticing is that they are pretty much all inherently contradictory, emphasizing the *craft* aspect of photography.

      Light juggling

      1. Darkness matters. Large parts of the beauty of fire photography comes from actually seeing the fire, even from the fire being the main actor in the image. Hence, it helps if the fire light doesn’t have to compete with ambient light. Pick a nice location, wait until late enough in the evening that ambient light is dim if not dark.

      2. Long exposures. Conveying the motion of the flames in a fire dance gets easier and all the more impressive if each individual flame becomes a bright, vivid streak of light, tracing out the curve of the dance. You get this with long exposures. The longer the better: time really does translate pretty much directly to vivid visuals here.

      3. Sharp facial features. A dancer dancing blurs. This is kinda the point of the item #2 above. Blurring the fire, and limbs, imbues a sense of motion, a sense of action to the picture. However, blurring the face removes the feeling of humanity more than anything else. Keeping the dancer immobile giving them sharp, recognizable features while still moving their extremities and the fire will make for a truly iconic fire dancing picture.

      4. Slow dances. Related to all of the above, a slow fire dance will accomplish several things for us as photographers:
      a. Dancing slowly means the fire moves slowly. If you’ve ever watched a fire dancer, you’ll notice that when the fire moves slowly, it burns with a large, bright yellow flame, illuminating the area around it. Instead, when moving fast, the fire flickers, burns in a dim hot blue, and illuminates much less.
      b. Dancing slowly means the dancer moves less, helping to keep the dancer sharp at the core.

      After the performance this weekend, I talked to the performers, and I might be given the chance of making a dedicated photo shoot with them juggling fire. I am already making plans for the photo shoot: how to stage it, what to do and how. Expose for about -1, maybe -2ev, against the background, illuminating the juggler with their fire, but keeping the background visible and interesting. Posing them — with the fire — on a jetty, so that the fire dance is reflected in the water. And asking them to try and keep their face stationary, while twirling fire, and exposing at somewhere in the range of 1/2-3″.

      At least that’s my current plan.

      by Michi at July 18, 2011 10:52 AM

      July 13, 2011

      Jeremy Gibbons

      Upwards and downwards accumulations

      Continuing my work in regress, this post revisits—with the benefit of much hindsight—what I was working on for my DPhil thesis (which was summarized in a paper at MPC 1992) and in subsequent papers at MPC 1998 and in SCP in 2000. This is the topic of accumulations on data structures, which distribute information across the data structure. List instances are familiar from the Haskell standard libraries (and, to those with a long memory, from APL); my thesis presented instances for a variety of tree datatypes; and the later work was about making it datatype-generic. I now have a much better way of doing it, using Conor McBride’s derivatives.

      Accumulations

      Accumulations or scans distribute information contained in a data structure across that data structure in a given direction. The paradigmatic example is computing the running totals of a list of numbers, which can be thought of as distributing the numbers rightwards across the list, summing them as you go. In Haskell, this is an instance of the {\mathit{scanl}} operator:

      \displaystyle  \begin{array}{lcl} \mathit{scanl} &::& (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow [\beta] \\ \mathit{scanl}\,f\,e\,[\,] &=& [e] \\ \mathit{scanl}\,f\,e\,(a:x) &=& e : \mathit{scanl}\,f\,(f\,e\,a)\,x \medskip \\ \mathit{totals} &::& [{\mathbb Z}] \rightarrow [{\mathbb Z}] \\ \mathit{totals} &=& \mathit{scanl}\,(+)\,0 \end{array}

      A special case of this pattern is to distribute the elements of a list rightwards across the list, simply collecting them as you go, rather than summing them. That’s the {\mathit{inits}} function, and it too is an instance of {\mathit{scanl}}:

      \displaystyle  \mathit{inits} = \mathit{scanl}\,\mathit{snoc}\,[\,] \quad\mathbf{where}\; \mathit{snoc}\,x\,a = x \mathbin{{+}\!\!\!{+}} [a]

      It’s particularly special, in the sense that it is the most basic {\mathit{scanl}}; any other instance can be expressed in terms of it:

      \displaystyle  \mathit{scanl}\,f\,e = \mathit{map}\,(\mathit{foldl}\,f\,e) \cdot \mathit{inits}

      This is called the Scan Lemma for {\mathit{scanl}}. Roughly speaking, it states that a {\mathit{scanl}} replaces every node of a list with a {\mathit{foldl}} applied to that node’s predecessors. Read from right to left, the scan lemma is an efficiency-improving transformation, eliminating duplicate computations; but note that this only works on expressions {\mathit{map}\,f \cdot \mathit{inits}} where {f} is a {\mathit{foldl}}, because only then are there duplicate computations to eliminate. It’s an important result, because it relates a clear and simple specification on the right to a more efficient implementation on the left.

      However, the left-to-right operators {\mathit{inits}}, {\mathit{foldl}}, and {\mathit{scanl}} are a little awkward in Haskell, because they go against the grain of the cons-based (ie, right-to-left) structure of lists. I leave as a simple exercise for the reader the task of writing the more natural {\mathit{tails}}, {\mathit{foldr}}, and {\mathit{scanr}}, and identifying the relationships between them. Conversely, one can view {\mathit{inits}} etc as the natural operators for snoc-based lists, which are constructed from nil and snoc rather than from nil and cons.

      Upwards and downwards accumulations on binary trees

      What would {\mathit{inits}}, {\mathit{tails}}, {\mathit{scanl}}, etc look like on different—and in particular, non-linear—datatypes? Let’s consider a simple instance, for homogeneous binary trees; that is, trees with a label at both internal and external nodes.

      \displaystyle  \mathbf{data}\;\mathsf{Tree}\,\alpha = \mathit{Leaf}\,\alpha \mid \mathit{Fork}\,\alpha\,(\mathsf{Tree}\,\alpha)\,(\mathsf{Tree}\,\alpha)

      for which the obvious fold operator is

      \displaystyle  \begin{array}{lcl} \mathit{fold} &::& (\alpha\rightarrow\beta) \rightarrow (\alpha\rightarrow\beta\rightarrow\beta\rightarrow\beta) \rightarrow \mathsf{Tree}\,\alpha \rightarrow \beta \\ \mathit{fold}\,f\,g\,(\mathit{Leaf}\,a) &=& f\,a \\ \mathit{fold}\,f\,g\,(\mathit{Fork}\,a\,t\,u) &=& g\,a\,(\mathit{fold}\,f\,g\,t)\,(\mathit{fold}\,f\,g\,u) \end{array}

      I’m taking the view that the appropriate generalization is to distribute data “upwards” and “downwards” through such a tree—from the leaves towards the root, and vice versa. This does indeed specialize to the definitions we had on lists when you view them vertically in terms of their “cons” structure: they’re long thin trees, in which every parent has exactly one child. (An alternative view would be to look at distributing data horizontally through a tree, from left to right and vice versa. Perhaps I’ll come back to that another time.)

      The upwards direction is the easier one to deal with. An upwards accumulation labels every node of the tree with some function of its descendants; moreover, the descendants of a node themselves form a tree, so can be easily represented, and folded. So we can quite straightforwardly define:

      \displaystyle  \begin{array}{lcl} \mathit{scanu} &::& (\alpha\rightarrow\beta) \rightarrow (\alpha\rightarrow\beta\rightarrow\beta\rightarrow\beta) \rightarrow \mathsf{Tree}\,\alpha \rightarrow \mathsf{Tree}\,\beta \\ \mathit{scanu}\,f\,g\,(\mathit{Leaf}\,a) &=& \mathit{Leaf}\,(f\,a) \\ \mathit{scanu}\,f\,g\,(\mathit{Fork}\,a\,t\,u) &=& \mathit{Fork}\,(g\,a\,(\mathit{root}\,t')\,(\mathit{root}\,u'))\,t'\,u' \\ & & \quad\mathbf{where}\; t' = \mathit{scanu}\,f\,g\,t ; u' = \mathit{scanu}\,f\,g\,u \end{array}

      where {\mathit{root}} yields the root of a tree:

      \displaystyle  \begin{array}{lcl} \mathit{root} &::& \mathsf{Tree}\,\alpha \rightarrow \alpha \\ \mathit{root}\,(\mathit{Leaf}\,a) &=& a \\ \mathit{root}\,(\mathit{Fork}\,a\,t\,u) &=& a \end{array}

      As with lists, the most basic upwards scan uses the constructors themselves as arguments:

      \displaystyle  \begin{array}{lcl} \mathit{subtrees} &::& \mathsf{Tree}\,\alpha \rightarrow \mathsf{Tree}\,(\mathsf{Tree}\,\alpha) \\ \mathit{subtrees} &=& \mathit{scanu}\,\mathit{Leaf}\,\mathit{Fork} \end{array}

      and any other scan can be expressed, albeit less efficiently, in terms of this:

      \displaystyle  \mathit{scanu}\,f\,g = \mathit{fmap}\,(\mathit{fold}\,f\,g) \cdot \mathit{subtrees}

      The downwards direction is more difficult, though. A downwards accumulation should label every node with some function of its ancestors; but these do not form another tree. For example, in the homogeneous binary tree

      the ancestors of the node labelled {3} are the nodes labelled {2,4,3}. One could represent those ancestors simply as a list, {[2,4,3]}; but that rules out the possibility of a downwards accumulation treating left children differently from right children, which is essential in a number of algorithms (such as the parallel prefix and tree drawing algorithms in my thesis). A more faithful rendering is to define a new datatype of paths that captures the left and right turns—a kind of non-empty cons list, but with both a “left cons” and a “right cons” constructor:

      \displaystyle  \mathbf{data}\;\mathsf{Path}\,\alpha = \mathit{Single}\,\alpha \mid \mathit{LCons}\,\alpha\,(\mathsf{Path}\,\alpha) \mid \mathit{RCons}\,\alpha\,(\mathsf{Path}\,\alpha)

      (I called them “threads” in my thesis.) Then we can capture the data structure representing the ancestors of the node labelled {3}

      by the expression {\mathit{RCons}\,2\,(\mathit{LCons}\,4\,(\mathit{Single}\,3))}. I leave it as an exercise for the more energetic reader to work out a definition for

      \displaystyle  \mathit{paths} :: \mathsf{Tree}\,\alpha \rightarrow \mathsf{Tree}\,(\mathsf{Path}\,\alpha)

      to compute the tree giving the ancestors of every node, and for a corresponding {\mathit{scand}}.

      Generic upwards accumulations

      Having seen ad-hoc constructions for a particular kind of binary tree, we should consider what the datatype-generic construction looks like. I discussed datatype-generic upwards accumulations already, in the post on Horner’s Rule; the construction was given in the paper Generic functional programming with types and relations by Richard Bird, Oege de Moor and Paul Hoogendijk. As with homogeneous binary trees, it’s still the case that the generic version of {\mathit{subtrees}} labels every node of a data structure of type {\mathsf{T}\alpha = \mu\mathsf{F}\alpha} with the descendants of that node, and still the case that the descendants form a data structure also of type {\mathsf{T}\alpha}. However, in general, the datatype {\mathsf{T}} does not allow for a label at every node, so we need the labelled variant {\mathsf{L}\alpha = \mu\mathsf{G}\alpha} where {\mathsf{G}(\alpha,\beta) = \alpha \times \mathsf{F}(1,\beta)}. Then we can define

      \displaystyle  \mathit{subtrees}_{\mathsf{F}} = \mathit{fold}_{\mathsf{F}}(\mathit{in}_{\mathsf{G}} \cdot \mathit{fork}(\mathit{in}_{\mathsf{F}} \cdot \mathsf{F}(\mathit{id},\mathit{root}), \mathsf{F}(!,\mathit{id}))) :: \mathsf{T}\alpha \rightarrow \mathsf{L}(\mathsf{T}\alpha)

      where {\mathit{root} = \mathit{fst} \cdot \mathit{in}_{\mathsf{G}}^{-1} = \mathit{fold}_{\mathsf{G}}\,\mathit{fst} :: \mathsf{L}\alpha \rightarrow \alpha} returns the root label of a labelled data structure—by construction, every labelled data structure has a root label—and {!_{\alpha} :: \alpha \rightarrow 1} is the unique arrow to the unit type. Moreover, we get a datatype-generic {\mathit{scanu}} operator, and a Scan Lemma:

      \displaystyle  \begin{array}{lcl} \mathit{scanu}_{\mathsf{F}} &::& (\mathsf{F}(\alpha,\beta) \rightarrow \beta) \rightarrow \mathsf{T}\alpha \rightarrow \mathsf{L}\beta \\ \mathit{scanu}_{\mathsf{F}}\,\phi &=& \mathsf{L}\,(\mathit{fold}_{\mathsf{F}}\,\phi) \cdot \mathit{subtrees}_{\mathsf{F}} \\ &=& \mathit{fold}_{\mathsf{F}}(\mathit{in}_{\mathsf{G}} \cdot \mathit{fork}(\phi \cdot \mathsf{F}(\mathit{id},\mathit{root}), \mathsf{F}(!,\mathit{id}))) \end{array}

      Generic downwards accumulations, via linearization

      The best part of a decade after my thesis work, inspired by the paper by Richard Bird & co, I set out to try to define datatype-generic versions of downward accumulations too. I wrote a paper about it for MPC 1998, and then came up with a new construction for the journal version of that paper in SCP in 2000. I now think these constructions are rather clunky, and I have a better one; if you don’t care to explore the culs-de-sac, skip this section and the next and go straight to the section on derivatives.

      The MPC construction was based around a datatype-generic version of the {\mathsf{Path}} datatype above, to represent the “ancestors” of a node in an inductive datatype. The tricky bit is that data structures in general are non-linear—a node may have many children—whereas paths are linear—every node has exactly one child, except the last which has none; how can we define a “linear version” {\mathsf{F}'} of {\mathsf{F}}? Technically, we might say that a functor is linear (actually, “affine” would be a better word) if it distributes over sum.

      The construction in the paper assumed that {\mathsf{F}} was a sum of products of literals

      \displaystyle  \begin{array}{lcl} \mathsf{F}(\alpha,\beta) &=& \sum_{i=1}^{n} \mathsf{F}_i(\alpha,\beta) \\ \mathsf{F}_i(\alpha,\beta) &=& \prod_{j=1}^{m_i} \mathsf{F}_{i,j}(\alpha,\beta) \end{array}

      where each {\mathsf{F}_{i,j}(\alpha,\beta)} is either {\alpha}, {\beta}, or some constant type such as {\mathit{Int}} or {\mathit{Bool}}. For example, for leaf-labelled binary trees

      \displaystyle  \mathbf{data}\;\mathsf{Tree}\,\alpha = \mathit{Tip}\,\alpha \mid \mathit{Bin}\,(\mathsf{Tree}\,\alpha)\,(\mathsf{Tree}\,\alpha)

      the shape functor is {\mathsf{F}(\alpha,\beta) = \alpha + \beta \times \beta}, so {n=2} (there are two variants), {m_1=1} (the first variant has a single literal, {\alpha}) and {m_2=2} (the second variant has two literals, {\beta} and {\beta}), and:

      \displaystyle  \begin{array}{lcl} \mathsf{F}(\alpha,\beta) &=& \mathsf{F}_1(\alpha,\beta) + \mathsf{F}_2(\alpha,\beta) \\ \mathsf{F}_1(\alpha,\beta) &=& \mathsf{F}_{1,1}(\alpha,\beta) \\ \mathsf{F}_{1,1}(\alpha,\beta) &=& \alpha \\ \mathsf{F}_2(\alpha,\beta) &=& \mathsf{F}_{2,1}(\alpha,\beta) \times \mathsf{F}_{2,2}(\alpha,\beta) \\ \mathsf{F}_{2,1}(\alpha,\beta) &=& \beta \\ \mathsf{F}_{2,1}(\alpha,\beta) &=& \beta \\ \end{array}

      Then for each {i} we define a {(k_i+1)}-ary functor {\mathsf{F}'_i}, where {k_i} is the “degree of branching” of variant {i} (ie, the number of {\beta}s occurring in {\mathsf{F}_i(\alpha,\beta)}, which is the number of {j} for which {\mathsf{F}_{i,j}(\alpha,\beta)=\beta}), in such a way that

      \displaystyle  \mathsf{F}'_i(\alpha,\beta,\ldots,\beta) = \mathsf{F}_i(\alpha,\beta)

      and {\mathsf{F}'_i} is linear in each argument except perhaps the first. It’s a bit messy explicitly to give a construction for {\mathsf{F}'_i}, but roughly speaking,

      \displaystyle  \mathsf{F}'_i(\alpha,\beta_1,\ldots,\beta_{k_i}) = \prod_{j=1}^{m_i} \mathsf{F}'_{i,j}(\alpha,\beta_1,\ldots,\beta_{k_i})

      where {\mathsf{F}'_{i,j}(\alpha,\beta_1,\ldots,\beta_{k_i})} is “the next unused {\beta_i}” when {\mathsf{F}_{i,j}(\alpha,\beta)=\beta}, and just {\mathsf{F}_{i,j}(\alpha,\beta)} otherwise. For example, for leaf-labelled binary trees, we have:

      \displaystyle  \begin{array}{lcl} \mathsf{F}'_1(\alpha) &=& \alpha \\ \mathsf{F}'_2(\alpha,\beta_1,\beta_2) &=& \beta_1 \times \beta_2 \end{array}

      Having defined the linear variant {\mathsf{F}'} of {\mathsf{F}}, we can construct the datatype {\mathsf{P}\alpha = \mu\mathsf{H}\alpha} of paths, as the inductive datatype of shape {\mathsf{H}} where

      \displaystyle  \mathsf{H}(\alpha,\beta) = \mathsf{F}(\alpha,1) + \sum_{i=1}^{n} \sum_{j=1}^{k_i} (\mathsf{F}_i(\alpha,1) \times \beta)

      That is, paths are a kind of non-empty cons list. The path ends at some node of the original data structure; so the last element of the path is of type {\mathsf{F}(\alpha,1)}, which records the “local content” of a node (its shape and labels, but without any of its children). Every other element of the path consists of the local content of a node together with an indication of which direction to go next; this amounts to the choice of a variant {i}, followed by the choice of one of {k_i} identical copies of the local contents {\mathsf{F}_i(\alpha,1)} of variant {i}, where {k_i} is the degree of branching of variant {i}. We model this as a base constructor {\mathit{End}} and a family of “cons” constructors {\mathit{Cons}_{i,j}} for {1 \le i \le n} and {1 \le j \le k_i}.

      For example, for leaf-labelled binary trees, the “local content” for the last element of the path is either a single label (for tips) or void (for bins), and for the other path elements, there are zero copies of the local content for a tip (because a tip has zero children), and two copies of the void local information for bins (because a bin has two children). Therefore, the path datatype for such trees is

      \displaystyle  \mathbf{data}\;\mathsf{Path}\,\alpha = \mathit{End}\,(\mathsf{Maybe}\,\alpha) \mid \mathit{Cons}_{2,1}\,(\mathsf{Path}\,\alpha) \mid \mathit{Cons}_{2,2}\,(\mathsf{Path}\,\alpha)

      which is isomorphic to the definition that you might have written yourself:

      \displaystyle  \mathbf{data}\;\mathsf{Path}\,\alpha = \mathit{External}\,\alpha \mid \mathit{Internal} \mid \mathit{Left}\,(\mathsf{Path}\,\alpha) \mid \mathit{Right}\,(\mathsf{Path}\,\alpha)

      For homogeneous binary trees, the construction gives

      \displaystyle  \mathbf{data}\;\mathsf{Path}\,\alpha = \mathit{External}\,\alpha \mid \mathit{Internal}\,\alpha \mid \mathit{Left}\,\alpha\,(\mathsf{Path}\,\alpha) \mid \mathit{Right}\,\alpha\,(\mathsf{Path}\,\alpha)

      which is almost the ad-hoc definition we had two sections ago, except that it distinguishes singleton paths that terminate at an external node from those that terminate at an internal one.

      Now, analogous to the function {\mathit{subtrees}_\mathsf{F}} which labels every node with its descendants, we can define a function {\mathit{paths}_\mathsf{F} : \mathsf{T}\alpha \rightarrow \mathsf{L}(\mathsf{P}\alpha)} to label every node with its ancestors, in the form of the path to that node. One definition is as a fold; informally, at each stage we construct a singleton path to the root, and map the appropriate “cons” over the paths to each node in each of the children (see the paper for a concrete definition). This is inefficient, because of the repeated maps; it’s analogous to defining {\mathit{inits}} by

      \displaystyle  \begin{array}{lcl} \mathit{inits}\,[\,] &=& [[\,]] \\ \mathit{inits}\,(a:x) &=& [\,] : \mathit{map}\,(a:)\,(\mathit{inits}\,x) \end{array}

      A second definition is as an unfold, maintaining as an accumulating parameter of type {\mathsf{P}(\alpha)\rightarrow\mathsf{P}(\alpha)} the “path so far”; this avoids the maps, but it is still quadratic because there are no common subexpressions among the various paths. (This is analogous to an accumulating-parameter definition of {\mathit{inits}}:

      \displaystyle  \begin{array}{lcl} \mathit{inits} &=& \mathit{inits}'\,\mathit{id} \medskip \\ \mathit{inits}'\,f\,[\,] &=& f\,[\,] \\ \mathit{inits}'\,f\,(a:x) &=& f\,[\,] : \mathit{inits}'\,(f \cdot (a:))\,x \end{array}

      Even with an accumulating “Hughes list” parameter, it still takes quadratic time.)

      The downwards accumulation itself is defined as a path fold mapped over the paths, giving a Scan Lemma for downwards accumulations. With either the fold or the unfold definition of paths, this is still quadratic, again because of the lack of common subexpressions in a result of quadratic size. However, in some circumstances the path fold can be reassociated (analogous to turning a {\mathit{foldr}} into a {\mathit{foldl}}), leading finally to a linear-time computation; see the paper for the details of how.

      Generic downwards accumulations, via zip

      I was dissatisfied with the “…”s in the MPC construction of datatype-generic paths, but couldn’t see a good way of avoiding them. So in the subsequent SCP version of the paper, I presented an alternative construction of downwards accumulations, which does not go via a definition of paths; instead, it goes directly to the accumulation itself.

      As with the efficient version of the MPC construction, it is coinductive, and uses an accumulating parameter to carry in to each node the seed from higher up in the tree; so the downwards accumulation is of type {\gamma \times \mathsf{T}\alpha \rightarrow \mathsf{L}\beta}. It is defined as an unfold, with a body {g} of type

      \displaystyle  \gamma \times \mathsf{T}\alpha \rightarrow \mathsf{G}(\beta, \gamma \times \mathsf{T}\alpha)

      The result {\mathsf{G}(\beta, \gamma \times \mathsf{T}\alpha)} of applying the body will be constructed from two components, of types {\mathsf{G}(\beta, \gamma)} and {\mathsf{G}(1, \mathsf{T}\alpha)}: the first gives the root label of the accumulation and the seeds for processing the children, and the second gives the children themselves.

      These two components get combined to make the whole result via a function

      \displaystyle  \mathit{zip} :: \mathsf{G}(\alpha,\beta) \times \mathsf{G}(\gamma,\delta) \rightarrow \mathsf{G}(\alpha \times \gamma, \beta \times \delta)

      This will be partial in general, defined only for pairs of {\mathsf{G}}-structures of the same shape.

      The second component of {g} is the easier to define; given input {\gamma \times \mathsf{T}\alpha}, it unpacks the {\mathsf{T}\alpha} to {\mathsf{F}(\alpha,\mathsf{T}\alpha)}, and discards the {\gamma} and the {\alpha} (recall that {\mathsf{L}\alpha=\mu\mathsf{G}\alpha} is the labelled variant of {\mathsf{T}\alpha=\mu\mathsf{F}\alpha}, where {\mathsf{G}(\alpha,\beta) = \alpha \times \mathsf{F}(1,\beta)}).

      For the first component, we enforce the constraint that all output labels are dependent only on their ancestors by unpacking the {\mathsf{T}\alpha} and pruning off the children, giving input {\gamma \times \mathsf{F}(\alpha,1)}. We then suppose as a parameter to the accumulation a function {f} of type {\gamma \times \mathsf{F}(\alpha,1) \rightarrow \beta \times \mathsf{F}(1,\gamma) = \mathsf{G}(\beta,\gamma)} to complete the construction of the first component. In order that the two components can be zipped together, we require that {f} is shape-preserving in its second argument:

      \displaystyle  \mathsf{F}(!,!) \cdot \mathit{snd} = \mathsf{F}(!,!) \cdot f \cdot \mathit{snd}

      where {! : \alpha \rightarrow 1} is the unique function to the unit type. Then, although the {g} built from these two components depends on the partial function {\mathit{zip}}, it will still itself be total.

      The SCP construction gets rid of the “…”s in the MPC construction. It is also inherently efficient, in the sense that if the core operation {f} takes constant time then the whole accumulation takes linear time. However, use of the partial {\mathit{zip}} function to define a total accumulation is a bit unsatisfactory, taking us outside the domain of sets and total functions. Moreover, there’s now only half an explanation in terms of paths: accumulations in which the label attached to each node depends only on the list of its ancestors, and not on the left-to-right ordering of siblings, can be factored into a list function (in fact, a {\mathit{foldl}}) mapped over the “paths”, which is now a tree of lists; but accumulations in which left children are treated differently from right children, such as the parallel prefix and tree drawing algorithms mentioned earlier, can not.

      Generic downwards accumulations, via derivatives

      After another interlude of about a decade, and with the benefit of new results to exploit, I had a “eureka” moment: the linearization of a shape functor is closely related to the beautiful notion of the derivative of a datatype, as promoted by Conor McBride. The crucial observation Conor made is that the “one-hole contexts” of a datatype—that is, for a container datatype, the datatype of data structures with precisely one element missing—can be neatly formalized using an analogue of the rules of differential calculus. The one-hole contexts are precisely what you need to identify which particular child you’re talking about out of a collection of children. (If you’re going to follow along with some coding, I recommend that you also read Conor’s paper Clowns to the left of me, jokers to the right. This gives the more general construction of dissecting a datatype, identifying a unique hole, but also allowing the “clowns” to the left of the hole to have a different type from the “jokers” to the right. I think the explanation of the relationship with the differential calculus is much better explained here; the original notion of derivative can be retrieved by specializing the clowns and jokers to the same type.)

      The essence of the construction is the notion of a derivative {\Delta\mathsf{F}} of a functor {\mathsf{F}}. For our purposes, we want the derivative in the second argument only of a bifunctor; informally, {\Delta\mathsf{F}(\alpha,\beta)} is like {\mathsf{F}(\alpha,\beta)}, but with precisely one {\beta} missing. Given such a one-hole context, and an element with which to plug the hole, one can reconstruct the whole structure:

      \displaystyle  \mathit{plug}_\mathsf{F} :: \beta \times \Delta\mathsf{F}(\alpha,\beta) \rightarrow \mathsf{F}(\alpha,\beta)

      That’s how to consume one-hole contexts; how can we produce them? We could envisage some kind of inverse {\mathit{unplug}} of {\mathit{plug}}, which breaks an {\mathsf{F}}-structure into an element and a context; but this requires us to invent a language for specifying which particular element we mean—{\mathit{plug}} is not injective, so {\mathit{unplug}} needs an extra argument. A simpler approach is to provide an operator that annotates every position at once with the one-hole context for that position:

      \displaystyle  \mathit{positions}_\mathsf{F} :: \mathsf{F}(\alpha,\beta) \rightarrow \mathsf{F}(\alpha, \beta \times \Delta\mathsf{F}(\alpha,\beta))

      One property of {\mathit{positions}} is that it really is an annotation—if you throw away the annotations, you get back what you started with:

      \displaystyle  \mathsf{F}(\mathit{id},\mathit{fst})\,(\mathit{positions}\,x) = x

      A second property relates it to {\mathit{plug}}—each of elements in a hole position plugs into its associated one-hole context to yield the same whole structure back again:

      \displaystyle  \mathsf{F}(\mathit{id},\mathit{plug})\,(\mathit{positions}\,x) = \mathsf{F}(\mathit{id},\mathit{const}\,x)\,x

      (I believe that those two properties completely determine {\mathit{plug}} and {\mathit{positions}}.)

      Incidentally, the derivative {\Delta\mathsf{F}} of a bifunctor can be elegantly represented as an associated type synonym in Haskell, in a type class {\mathit{Diff}} of bifunctors differentiable in their second argument, along with {\mathit{plug}} and {\mathit{positions}}:

      \displaystyle  \begin{array}{lcl} \rlap{\(\mathbf{class}\; \mathit{Bifunctor}\,f \Rightarrow \mathit{Diff}\,f \;\mathbf{where}\)} \\ \qquad \rlap{\(\mathbf{type}\; \mathit{Delta}\,f :: \ast \rightarrow \ast \rightarrow \ast\)} \\ \qquad \mathit{plug} &::& (\beta, \mathit{Delta}\,f\,\alpha\,\beta) \rightarrow f\,\alpha\,\beta \\ \qquad \mathit{positions} &::& f\,\alpha\,\beta \rightarrow f\,\alpha\,(\beta, \mathit{Delta}\,f\,\alpha\,\beta) \end{array}

      Conor’s papers show how to define instances of {\mathit{Diff}} for all polynomial functors {\mathsf{F}}—anything made out of constants, projections, sums, and products.

      The path to a node in a data structure is simply a list of one-hole contexts—let’s say, innermost context first, although it doesn’t make much difference—but with all the data off the path (that is, the other children) stripped away:

      \displaystyle  \mathsf{P}\alpha = \mathsf{List}(\Delta\mathsf{F}(\alpha,1))

      This is a projection of Huet’s zipper, which preserves the off-path children, and records also the subtree in focus at the end of the path:

      \displaystyle  \mathsf{Zipper}_\mathsf{F}\,\alpha = \mathsf{List}(\Delta\mathsf{F}(\alpha,\mu\mathsf{F}\alpha)) \times \mu\mathsf{F}\alpha

      Since the contexts are listed innermost-first in the path, closing up a zipper to reconstruct a tree is a {\mathit{foldl}} over the path:

      \displaystyle  \begin{array}{lcl} close_\mathsf{F} &::& \mathsf{Zipper}_\mathsf{F}\,\alpha \rightarrow \mu\mathsf{F}\alpha \\ close_\mathsf{F}\,(p,t) &=& \mathit{foldl}\,(\mathit{in}\cdot\mathit{plug})\,t\,p \end{array}

      Now, let’s develop the function {\mathit{paths}}, which turns a tree into a labelled tree of paths. We will write it with an accumulating parameter, representing the “path so far”:

      \displaystyle  \begin{array}{lcl} \mathit{paths}_\mathsf{F} &::& \mathsf{T}\alpha \rightarrow \mathsf{L}(\mathsf{P}\alpha) \\ \mathit{paths}_\mathsf{F}\,t &=& \mathit{paths}'_\mathsf{F}\,(t,[\,]) \end{array}

      Given the components {\mathit{in}_\mathsf{F}\,x} of a tree and a path {p} to its root, {\mathit{paths}'_\mathsf{F}} must construct the corresponding labelled tree of paths. Since {\mathsf{L} = \mu\mathsf{G}} and {\mathsf{G}(\alpha,\beta) = \alpha \times \mathsf{F}(1,\beta)}, this amounts to constructing a value of type {\mathsf{P}\alpha \times \mathsf{F}(1, \mathsf{L}(\mathsf{P}\alpha))}. For the first component of this pair we will use {p}, the path so far. The second component can be constructed from {x} by identifying all children via {\mathit{positions}}, discarding some information with judicious {!}s, consing each one-hole context onto {p} to make a longer path, then making recursive calls on each child:

      That is,

      \displaystyle  \begin{array}{lcl} \mathit{paths}'_\mathsf{F} &::& \mathsf{T}\alpha\times\mathsf{P}\alpha \rightarrow \mathsf{L}(\mathsf{P}\alpha) \\ \mathit{paths}'_\mathsf{F}\,(\mathit{in}_\mathsf{F}\,x,p) &=& \mathit{in}_\mathsf{G}(p, \mathsf{F}(!, \mathit{paths}'_\mathsf{F} \cdot \mathit{id}\times((:p)\cdot\Delta\mathsf{F}(\mathit{id},!)) )\,(\mathit{positions}\,x)) \end{array}

      Downwards accumulations are then path functions mapped over the result of {\mathit{paths}}. However, we restrict ourselves to path functions that are instances of {\mathit{foldr}}, because only then are there common subexpressions to be shared between a parent and its children (remember that paths are innermost-first, so related nodes share a tail of their ancestors).

      \displaystyle  \begin{array}{lcl} \mathit{scand}_\mathsf{F} &::& (\Delta\mathsf{F}(\alpha,1)\times\beta\rightarrow\beta) \rightarrow \beta \rightarrow \mathsf{T}\alpha \rightarrow \mathsf{L}\beta \\ \mathit{scand}_\mathsf{F}\,f\,e &=& \mathit{map}\,(\mathit{foldr}\,f\,e) \cdot \mathit{paths} \end{array}

      Moreover, it is straightforward to fuse the {\mathit{map}} with {\mathit{paths}}, to obtain

      \displaystyle  \begin{array}{lcl} \mathit{scand}_\mathsf{F}\,f\,e\,t &=& \mathit{scand}'_\mathsf{F}\,f\,(t,e) \medskip \\ \mathit{scand}'_\mathsf{F}\,f\,(\mathit{in}_\mathsf{F}\,x,e) &=& \mathit{in}_\mathsf{G}(e, \mathsf{F}(!, \mathit{scand}'_\mathsf{F}\,f \cdot \mathit{id}\times g )\,(\mathit{positions}\,x)) \\ & & \quad\mathbf{where}\; g\,d = f\,(\Delta\mathsf{F}(\mathit{id},!)\,d, e) \end{array}

      which takes time linear in the size of the tree, assuming that {f} and {e} take constant time.

      Finally, in the case that the function being mapped over the paths is a {\mathit{foldl}} as well as a {\mathit{foldr}}, then we can apply the Third Homomorphism Theorem to conclude that it is also an associative fold over lists. From this (I believe) we get a very efficient parallel algorithm for computing the accumulation, taking time logarithmic in the size of the tree—even if the tree has greater than logarithmic depth.


      by jeremygibbons at July 13, 2011 05:21 PM

      July 12, 2011

      Sam Anklesaria

      Cabal and GHCi

      For my Summer of Code project, I'm giving cabal an repl command. At first, I thought a simple filename argument would be enough. More complex Cabal packages require a slightly more detailed input, however. Often times, there's more to a Cabal package than a library. Sometimes there's tests. Sometimes an executable. And all these components can have different ways of building themselves. This makes interpretation a bit of a problem. If a file is part of multiple components, which method of building should be used? Both should work, certainly, but choosing a component arbitrarily leaves little control to the user. In cases like this, it seems that cabal repl must take a component as well. Let's give it a flag --component=COMPONENT, where COMPONENT is the name of some library or executable or test suite that's a component in the cabal package. Fair enough.

      But there's more. But what if a user wants to test a component in its entirety, not just one particular file? Here they only need to specify the component. So we say cabal repl --component=COMPONENT. What happens? What's loaded into GHCi? Should nothing be loaded, requiring the user to :load every module he wants to work with? Should everything be loaded, leading to potentially nasty name collisions? Should everything be loaded, but qualified, meaning that users will have to type hilariously long prefixes before all their functions? Should there be yet another flag for what course of action to take? Please, comment and tell me, because I have no idea.

      by Anklesaria (noreply@blogger.com) at July 12, 2011 10:47 PM

      Petr Rockai

      soc reloaded: progress 1

      Oops! There has been no update for a long while from me, although I have been busy with code/patches. So far, I have tackled two areas: generalising and cleaning up the existing Patch testsuite, so I could apply it to the in-progress V3 Prim code later. This has been quite successful, although it took a little longer than I would have liked. With the new structure, (QC) properties for single patches, commutes and merges can be applied to any concrete patch type that supports the respective operations. Therefore, I now have coverage of both V3 Prims as standalone patches (for single patch and commute properties) and also when used with RealPatch (the implementation of non-primitive darcs 2 patches).

      The latter part was then to make all these tests pass. Since I finished the respective Read/Show instances yesterday, all tests pass. Commute, apply and friends have been done couple days before that. So the next step is to write more tests that can demonstrate where the code needs to be augmented.

      Hunk storage

      I have slightly changed the actual (on-disk) hunk format slightly. For now, “detached” hunk-text storage is not quite supported, I am keeping that post-midterms. But the format still counts on that being possible. We do need a new monad (class) for writing patches though, since the Show instance is somewhat inadequate: the detached storage needs to be handled somehow.

      Anyway, the format now looks like this:

      hunk NNN .whitespace_encoded_old .new_text
      

      (we use the same method for encoding whitespace as we do in filenames here). We might want to change to a format that’s faster to produce/parse, since hunks typically do contain whitespace. On the other hand, only very short hunks will be encoded in this form. Also, an empty string is encoded as “!”. So the hunk text (old and new) can take following forms: .whitespace_encoded, ! (empty) or “@”. The last form, @ should take 2 parameters, a hash and length, like e.g. this: @123456789ABCD<...>:65000. That means that most of the time, we can ignore the hash and only fool around with the numbers (offset and lengths) when commuting patches. Of course, applying them is a different matter: nevertheless, we still do have a substantial advantage over V1 prims there, since each hunk is a simple splice/catenate operation on bytestrings. With V1 prims, we had to chop up the hunk at newlines and remove the +/- signs.

      As for implementation, this means we need to abstract commute over a monad class, which besides commutation failure can express an “fetch text for this hash” operation. This might be simpler than it was in the Apply case, a lot of code had to be modified to accomodate for V3 Prims, since the existing commute runs in the Maybe monad and we can make Maybe an instance of CommuteMonad. Nevertheless, to make actual use of this, toplevel code that runs commutes will definitely need to be modified, and in effect, all the intermediate code that relies on Maybe for commutation will need to be modified as well. This part could become actually more hairy than was the case with Apply.

      The question of coalesce

      Apart from apply and commute, there is currently one more “core” operation in Darcs patches: coalesce. This operation takes two primitive patches and decides whether they can be merged into a single primitive patch. This can only happen if the patches do not commute. Unfortunately, coalesce does not preserve commutation behaviour: move a b :> move b c gets coalesced into move a c, which modifies its commutation behaviour with (add, remove, move) patches mentioning b. On the other hand, coalesce is normally only used during operations like amend-record, rebase and when handling the “pending” patch. All these operations modify the identity of a patch, and therefore it shouldn’t matter much that coalesce fails to commute with commute.

      On the downside, coalesce is currently a first-class operation that cannot be derived from the remaining. Most importantly, it is “redundant” with the diff operation, that constructs patches from two states. The problem is that with current (V1) prims, there is no diff operation for some patches. If we had reliable diff for all patch types, we could implement coalesce in terms of diff, commute and apply (pseudocode):

      coalesce context (a :> b) | isJust (commute (a :> b)) =
          diff context (apply context $ a :> b)
                                | otherwise = a :> b
      

      This would work as long as there was a deterministic diff operation, i.e. one with the property (for a being a primitive patch) that diff ctx (apply ctx a) == a for all a. This diff operation does not need (and indeed cannot be made) universal over different patch types, but fortunately that doesn’t matter. We can always call it with a specific patch type as one of its inputs:

      diff TextHunk ctx1 ctx2
      

      I believe this operation should be possible to have, and it would also allow significant improvements in the “record” user experience: darcs could try various differs on a pair of states and offer “better” patches than just hunks (like eg. replace, move, etc.).

      The ability to implement coalesce in terms of other operations is important because even more than commute, the implementation of coalesce is O(n^2), with n being the number of different primitive patch types: it needs to take into account any pair of types. With the above approach, since coalesce always yields a single resulting patch, we can implement it as follows:

      • try to commute a with b; if this works, there’s nothing to coalesce
      • if it fails, take a context in which (a :> b) can be applied, ideally as small as possible (since diff is somewhat expensive)
      • try to diff the original and post-a/b context, using both the “a” differ and “b” differ: if either succeeds (i.e. produces a single primitive patch), then we have a winner; otherwise, we cannot coalesce either

      An UI digression

      This is not GSoC related: you can skip this section if all you care about is GSoC…

      Anyway, when I am talking about user interface. I think a substantial improvement in the way “semantic” patches work would come with the ability to infer those patches “automatically”. In the spirit of the above diff operation that is available with every patch type. However, it is hard to do a semantic “diff” on significantly divergent repository states. This is simply because further changes obscure the relationships of entities in question. When the only thing that changes is a “mv”, it is easy to detect. But when you also edit the file, it is no longer possible to tell for sure if this is a move or a new object.

      What would help substantially, then, would be to be able to run diff much more often. This prompts a workflow change. Of course, we cannot ask people to commit every time they do a small change. However, we could ask them to “amend” an “in-progress” patch when they do. This would be especially useful if they can be coached into stashing their changes before and after things like applying “sed” to whole codebase, moving around files etc. This would be basically a supercharged version of “darcs mv”: it would say “I did something, you figure what is the right way to represent it”. It adds the burden of having to call this both before and after the “contentious” operation. But it also adds significant benefits (IMHO).

      The other thing that would this kind of “open” patch (that you keep adding things to, until you are satisfied, and then you commit) allow is progressive (time-sensitive) revert. This is something that I would be completely sold on: if I kept telling the VCS to note down my changes reasonably often, I could get, in exchange, a whole-repository (but still granular) undo operation.

      (It is not hard to imagine that you could also have more than one “open” patch at a time, sorting changes into different buckets for semi-related changes. The UI for that one would be more tricky though.)

      The story of Apply

      To get back to GSoC though. For what it’s worth, the test-suite part of the work and a sketch of the V3 prim implementation is already in the screened branch. The changes to the Apply class are almost ready for getting into screened as well; they are currently available as patch635.

      The basic challenge with Apply was that V1 prims and V3 prims apply to different kinds of things. While V1 applies to a filesystem tree (basically your working copy), this is not the case with V3 anymore. The V3 state is modelled as a map from UUIDs to Objects. However, it would be extremely uncool to have two different “Apply” type classes to apply different kinds of patches: this would also mean that all higher layers of patch code would need to duplicate their apply implementations. Not cool.

      Therefore, associated types to the rescue: I have added an ApplyState associated type to the Apply class. The Prim level patches then decide what they can be applied to (currently a Tree, from hashed-storage, for V1 and an ObjectMap, which still needs to be fleshed out in more detail, for V3). Any higher levels inherit their apply state from the prims. Cool.

      Of course, it’s not as simple, since we actually have to implement that cool apply method. This was traditionally (well, since I merged my new annotate code; not that traditional I guess) been done through the ApplyMonad class. Now ApplyMonad used to have operations like “create a file”, “create a directory” or “write this bytestring to this file”. That’s cool for V1 prims, but not very useful for V3 prims. So ApplyMonad needed to be generalised over multiple apply states. This forced a multi-parameter type class, since there are no functional dependencies to save us (and therefore associated types do not apply either). This is because we expect some monads (e.g. IO) to be instances of both ObjectMap- and Tree-based ApplyMonad. In general, I didn’t want to limit this to special monads, although we might go for that option later if it turns out to be superior.

      Anyway, the ApplyMonad class is a bit of a meta-class, since the actually useful set operations is different for different apply state representations. But since the methods can carry their own constraints, I have added a couple of fully generic methods (get current state, set current state and the like) and a set that only applies to ObjectMap and one that only applies to Tree. This doesn’t seem to pose significant trouble. Haskell doesn’t seem to have higher-order type classes that would solve this maybe slightly more elegantly. (I don’t even know if they are possible. Don’t crucify me if they aren’t.)

      Anyway, long story short, we now have a single apply method in a single Apply class, that works on both V1 and V3 prims, as witnessed by running the same set of tests, which sometimes do invoke apply, on both implementations.

      The story of Commute

      There isn’t much of a story behind this one so far. As I outlined above, there are things coming here as well, but they are not required to allow V3 prims per se: only needed for the detached storage optimisation. The commute as it is implemented works, at least as far as tests are concerned. This goes hand in hand with some kind of StoreMonad and LoadMonad abstraction, that will actually allow us to implement the detached storage. The CommuteMonad can then be a

      class (LoadMonad m) => CommuteMonad m where
          commuteFail :: m a
      

      kind of deal. A LoadMonad superclass constraint can (should) appear on ApplyMonad as well. For now, the instances don’t need to be too elaborate (they can simply fail to fetch anything at all, which will work just fine for V1 prims).

      The story of unsafeInterleaveIO

      I don’t like unsafeInterleaveIO. At all. My last summer of code was, after a fashion, about removing a significant source of unsafe, ugly and outright dangerous lazy IO from darcs. I believe it was a significant success. Now the LoadMonad / StoreMonad abstraction has a potential to rid us of another source of lazy IO in darcs: currently, the reading of patch content is unsafeInterleaveIO`d and the rest of the code treats the repository as a kind-of-pure data structure built of patches. Of course, the unsafeInterleaveIO is unsafe because it breaks the type system. Since darcs uses it a lot, there is no telling which value is actually pure. Arguing about runtime behaviour of lazy code is hard enough when it is actually pure. Random IO thunks lurking inside pure values (kind of like an alien in Sigourney’s stomach) turn it into a nightmare.

      This will involve a lot of code being lifted into a monad, fortunately a significantly restricted monad. In practice, it’ll be IO more often than not (although in the testsuite, it’ll probably be the like of Maybe, or StateT Maybe). What matters is not that the code will execute in IO, but it is statically well protected: it cannot access the IO monad and it has to be explicit about side effects (like loading things from disk). Therefore, we give things types that actually reflect impurity, without allowing them to spin out of control with side effects (like they could if they were simply in the IO monad). A static type system win.

      Conclusion

      Ok, I guess this is more than enough for today. I’ll try to keep you folks more informed about the progress in the second half of the endeavour. On the other hand, I am a bit worried that these posts are more useful as a note-to-self resource than for general reading by others. Well, let’s hope, dear reader, that you found at least something of this post useful and/or interesting.

      July 12, 2011 10:27 AM

      July 09, 2011

      Lennart Augustsson

      Impredicative polymorphism, a use case

      In a recent question on stackoverflow I made a comment about how Haskell can be considered a good imperative language because you can define abstractions to make it convenient. When I was going to make my point by implementing a simple example of it I found that what I wanted to do no longer works in ghc-7.0.4. Here's a simple example of what I wanted to do (which works in ghc-6.12.3). It's a simple library that makes Haskell code look a bit like Python.

      {-# LANGUAGE ExtendedDefaultRules, TupleSections #-}
      module Main where
      import qualified Prelude
      import Boa
      
      last_elt(xs) = def $: do
          assert xs "Empty list"
          lst <- var xs              -- Create a new variable, lst
          ret <- var (xs.head)
          while lst $: do
              ret #= lst.head        -- Assign variable ret
              lst #= lst.tail
          return ret
      
      first_elt(xs) = def $: do
          l <- var xs
          l.reverse                  -- Destructive reverse
          return (last_elt(l))
      
      factorial(n) = def $: do
          assert (n<=0) "Negative factorial"
          ret <- var 1
          i <- var n
          while i $: do
              ret *= i
              i -= 1
          return ret
      
      test = def $: do
          print "Hello"
          print ("factorial 10 =", factorial(10))
      
      main = do
          test
          l <- varc [1, 2, 3]
          print ("first and last:",)
          print (first_elt(l),)
          print (last_elt(l))
      

      On the whole it's pretty boring, except for one thing. In imperative languages there is (usually) a disctinction between l-values and r-values. An l-value represent a variable, i.e., something that can be assigned, whereas an r-value is simply a value. So in Python, in the statement x = x + 1 the x on the left is an l-value (it's being assigned), whereas on the right it's an r-value. You can use the same notation for both in most imperative languages. If you want to do the same in Haskell you have (at least) two choices. First you can unify the concepts of l-value and r-value and have a runtime test that you only try to assign variables. So, e.g., 5 = x would type check but have a runtime failure. I will not dwell further on this since it's not very Haskelly.
      Instead, we want to have l-values and r-values being statically type checked. Here's the interesting bit of the types for a simple example.
       
      data LValue a
      data RValue a
      instance (Num a) => Num (RValue a)
      
      class LR lr
      instance LR RValue
      instance LR LValue
      
      var :: RValue a -> IO (forall lr . (LR lr) => lr a)
      (#=) :: LValue a -> RValue a -> IO ()
      
      foo = do
          x <- var 42
          x #= x + 1
      

      We have two type constructors LValue and RValue representing l-values and r-values of some type a. The r-values is an instance of Num. Furthermore, the class LR where the type is either LValue or RValue.
      The var function creates a new variable given a value. The return type of var is the interesting part. It says that the return value is polymorphic; it can be used either as an l-value or as r-value. Just as we want.
      The assignment operator, (#=), takes an l-value, an r-value, and returns nothing.

      So in the example we expect x to have type forall lr . (LR lr) => lr a, in which case the assignment will type check.
      If we try to compile this we get
          Illegal polymorphic or qualified type:
            forall (lr :: * -> *). LR lr => lr a
          Perhaps you intended to use -XImpredicativeTypes
          In the type signature for `var':
            var :: RValue a -> IO (forall lr. LR lr => lr a)
      

      The problem is that universal quantification is not normally allowed as the argument to a type constructor. This requires the impredicative polymorphism extension to ghc. If we turn it on it compiles fine in ghc-6.12.3.
      But, with ghc-7.0.4 we get
          Couldn't match expected type `LValue a0'
                      with actual type `forall (lr :: * -> *). LR lr => lr a1'
          In the first argument of `(#=)', namely `x'
          In the expression: x #= x + 1
      

      I can't really explain the rational behind the change in the ghc type system (Simon say it's simpler now), but I feel this has really made ghc worse for defining DSELs. When you define a DSEL you usually use the do-notation for the binding construct in the embedded language. This is the only programmable binding construct (by overloading (>>=)), so there is little choice. With the change in ghc-7 it means you can no longer make DSELs with a polymorhic binding construct (like I wanted here), because the binder seems to be monomorphic now

      Please Simon, can we have polymorphic do bindings back?

      by augustss (noreply@blogger.com) at July 09, 2011 05:54 PM

      July 07, 2011

      Thomas M. DuBuisson

      HacPDX II – More Hacking!

      HacPDX II is set for July 22-24, 2011 (Friday – Sunday)! Be sure to register or else, exactly like HacPDX 1, you might not get network access.

      I hope to spend most of my time working on Hackage (see here and here) with other interested Haskellers both locally and via IRC (#hacpdx on freenode). Still, I might sqeeze in some time for my seemingly never-complete side projects… we’ll see!

      Hope to see you there!


      by tommd at July 07, 2011 04:52 AM

      July 03, 2011

      RandProc

      Announce: release of v0.4 of `RandProc` module

      v0.4 of `RandProc` has just been posted to Hackage.

      In this release:

      • Examples of using the `RandProc` library to work problems from `Random Processes` text have been added.
      • `README` file has been fleshed out a bit.

      by dbanas at July 03, 2011 03:02 PM

      June 27, 2011

      RandProc

      Announce: release of v0.2 of `RandProc` library

      v0.2 of the `RandProc` library has been released to Hackage.

      Changes in this release:

      • Improved several severe performance bottlenecks. (The `goodDie` space can now actually be checked, without exhausting memory/time.)
      • Cleaned up the code, as per `hlint` suggestions.

      by dbanas at June 27, 2011 07:47 PM

      Sean Seefried

      Generic dot products

      Companion slides

      This is the first in a series of posts about program derivation. In particular, I am attempting to derive a matrix multiplication algorithm that runs efficiently on parallel architectures such as GPUs.

      As I mentioned in an earlier post, I’ve been contributing to the Accelerate project. The Accelerate EDSL defines various parallel primitives such as map, fold, and scan (and many more).

      The scan primitive (also known as all-prefix-sums) is quite famous because it is useful in a wide range of parallel algorithms and, at first glance, one could be forgiven for thinking it is not amenable to parallelisation. However, a well-known work efficient algorithm for scan was popularised by Guy Blelloch which performs \(O(n)\) work.

      The algorithm is undeniably clever. Looking at it, it is not at all obvious how one might have gone about developing it oneself. A recent series of posts by Conal Elliott set out to redress this situation by attempting to derive the algorithm from a high level specification. His success has inspired me to follow a similar process to derive a work efficient matrix multiplication algorithm.

      The process I am following is roughly as follows:

      • generalise the concept of matrix multiplication to data structures other than lists or arrays.

      • develop a generic implementation that relies, as far as possible, on reusable algebraic machinery in type classes such as Functor, Applicative, Foldable and Traversable.

      • use this generic implementation as a specification to derive an efficient algorithm. To call it a hunch that the underlying data structure is going to be tree-like is an understatement.

      This post is a preamble. It develops a generic dot product implementation that will serve as a specification for the derivation of an efficient algorithm in a later post.

      Background

      In order to understand this post I highly recommend that you read Conor McBride and Ross Paterson’s paper: Applicative programming with effects. A basic grasp of linear algebra would also be helpful.

      What is a dot product?

      In mathematics the dot product is usually defined on vectors. Given two vectors of length \(n\), \((a_1, \dots, a_n)\) and \((b_1, \dots, b_n)\) the dot product is defined as:

      \(a_1 b_1 + \dots + a_n b_n\)

      or without the use of the pernicious “\(\dots\)”:

      \(\sum_{i=1}^{n}{a_i b_i}\)

      The implementation for lists is fairly straightforward.

      dot :: Num a => [a] -> [a] -> a
      dot xs ys = foldl (+) 0 (zipWith (*) xs ys)

      This definition will work just fine on two lists of different length, owing to the definition of zipWith.

      zipWith :: (a -> b -> c) -> [a] -> [b] -> [c]
      zipWith f [] _ = []
      zipWith f _ [] = []
      zipWith f (x:xs) (y:ys) = f x y : zipWith f xs ys

      This is fine for lists but will become problematic later when we look at other data structures.

      There is no reason that this definition shouldn’t be extended to other data structures such as \(n\)-dimensional arrays or even trees. Let’s look at how we might define dot products on trees.

      Dot product on Trees

      We define trees as follows (however it does not really matter whether only the leaves contain numbers or whether branch nodes can too):

      data Tree a = Leaf a | Branch (Tree a) (Tree a)

      For the sake of succinctness, I will represent trees using nested pairs denoted with curly braces. e.g. Branch (Leaf 1) (Leaf 2) becomes {1,2}, Branch (Leaf 1) (Branch (Leaf 2) (Leaf 3)) becomes {1,{2,3}}.

      What should be the dot product of {1,{2,3}} and {4,{5,6}}? A reasonable answer would be 1 * 4 + 2 * 5 + 3 * 6 == 32. For each leaf in the first tree find the corresponding leaf in the second tree, multiply them together and then sum all the results together.

      This definition relies on the two trees having the same shape. To see why let’s see if we can we define a function in the style of zipWith for trees. Unfortunately, this is problematic.

      zipWithT f (Leaf a) (Leaf b)           = Leaf (f a b)
      zipWithT f (Branch s t) (Branch s' t') = Branch (zipWithT f s s')
      (zipWithT f t t')
      zipWithT f (Leaf a) (Branch s' t') = {- ? -} undefined
      zipWithT f (Branch s t) (Leaf b) = {- ? -} undefined

      There’s a problem with the last two cases. While I won’t go so far as to say that there is no definition we could provide, it’s clear that there are a number of choices that could be taken. In each case one needs to take an arbitrary element from the Branch argument and apply function f to it and the Leaf argument.

      Even if there is a definition that makes reasonable sense can we say whether it’s possible to provide a zipWith-like definition for an arbitrary data structure?

      An alternative is to modify our data structures to contain types that represent the shape of the data structure. We can then define dot such that it must take two arguments of exactly the same shape.

      Data structures with shapes

      I’ll illustrate this approach with vectors first, before moving onto trees. Vectors are just lists with their length encode into their type.

      Vectors

      First, we add some essentials to the top of our module.

      {-# LANGUAGE GADTs, EmptyDataDecls, FlexibleInstances, DeriveFunctor, DeriveFoldable #-}
      {-# LANGUAGE ScopedTypeVariables, FlexibleContexts, UndecidableInstances #-}

      Now we define two new data types, Z and S, representing Peano numbers. Both data types are empty since we will never be using their values.

      data Z
      data S n

      Now vectors.

      infixr 5 `Cons`
      data Vec n a where
      Nil :: Vec Z a
      Cons :: a -> Vec n a -> Vec (S n) a

      If you haven’t seen these data types before it’s worth noting that you can define total (vs partial) versions of head and tail. Trying to take the head of an empty vector simply doesn’t type check.

      headVec :: Vec (S n) a -> a
      headVec (Cons x _) = x

      tailVec :: Vec (S n) a -> Vec n a
      tailVec (Cons _ xs) = xs

      With can now define zipWithV.

      zipWithV :: (a -> b -> c) -> Vec n a -> Vec n b -> Vec n c
      zipWithV f Nil Nil = Nil
      zipWithV f (Cons x xs) (Cons y ys) = f x y `Cons` zipWithV f xs ys

      Unfortunately, GHC’s type checker does not detect that a case such as the one below is impossible. (In fact, if your warnings are turned up high enough GHC will warn that two patterns are missing in the definition above.)

      -- Although this pattern match is impossible GHC's type checker
      -- won't complain
      zipWithV f (Cons x xs) Nil = {- something -} undefined

      Trees

      The length of a tree is not quite a meaningful enough representation. Instead we represent its shape as a nested tuples of the unit (()) type.

      data Tree sh a where
      Leaf :: a -> Tree () a
      Branch :: Tree m a -> Tree n a -> Tree (m,n) a

      For example:

      {1,{2,3}} :: Tree ((),((),())) Integer

      The new definition of zipWithT only differs in its type.

      zipWithT :: (a -> b -> c) -> Tree sh a -> Tree sh b -> Tree sh c
      zipWithT f (Leaf a) (Leaf b) = Leaf (f a b)
      zipWithT f (Branch s t) (Branch s' t') = Branch (zipWithT f s s')
      (zipWithT f t t')

      Now finish off the definitions:

      foldlT :: (a -> b -> a) -> a -> Tree sh b -> a
      foldlT f z (Leaf a) = f z a
      foldlT f z (Branch s t) = foldlT f (foldlT f z s) t

      dotT :: Num a => Tree sh a -> Tree sh a -> a
      dotT t1 t2 = foldlT (+) 0 (zipWithT (*) t1 t2

      Generalising to arbitrary data structures

      Any seasoned Haskell veteran knows the utility of type classes such as Functor, Applicative, and Foldable. We have now seen that a dot product is essentially a zipWith followed by a fold. (It makes little difference whether its a left or right fold).

      Since zipWith is really just liftA2 (found in module Control.Applicative) on the ZipList data structure. This leads us to the following definition:

      dot :: (Num a, Foldable f, Applicative f) => f a -> f a -> a
      dot x y = foldl (+) 0 (liftA2 (*) x y)

      This function requires instances for Functor, Foldable and Applicative. Given that instances for the first two type classes are both easy to write (and in some cases derivable using Haskell’s deriving syntax), I will only discuss Applicative instances in this post. (The instances for vectors and shape-encoded trees are left as an exercise for the reader.)

      One might reasonably wonder, must the two arguments to dot have the same shape as before? It turns out that, yes, they do and for similar reasons. I’ll demonstrate the point by looking at how to define Applicative instances for lists, vectors and trees.

      Lists

      The default Applicative instance for lists is unsuitable for a generic dot product. However, the Applicative instance on its wrapper type ZipList is adequate but has an unsatisfying definition for pure (to say the least).

      instance Applicative ZipList where
      pure x = ZipList (repeat x)
      ZipList fs <*> ZipList xs = ZipList (zipWith id fs xs)

      Of course, this is necessary for lists since we can’t guarantee that two lists of the same length will be applied together. How else would you define pure to make it work on an arbitrary length lists xs?

      (+) <$> (pure 1) <*> (ZipList xs)

      The definition of pure is much more satisfying for vectors.

      Vectors

      Obviously we want a similar definition for pure as for lists (ZipList). But we don’t want to produce an infinite list, just one of the appropriate length.

      Defining the Applicative instance for vectors leads us to an interesting observation which holds true in general. For any data structure which encodes its own shape:

      1. You need one instance of Applicative for each constructor of the data type.
      2. The instance heads must mirror the types of the constructors.

      In the code below there are two instances and each instance head closely mirrors the data constructor’s type. e.g. Cons :: a -> Vec n a -> Vec (S n) a mirrors instance Applicative (Vec n) => Applicative (Vec (S n)).

      instance Applicative (Vec Z) where
      pure _ = Nil
      Nil <*> Nil = Nil

      instance Applicative (Vec n) => Applicative (Vec (S n)) where
      pure a = a `Cons` pure a
      (fa `Cons` fas) <*> (a `Cons` as) = fa a `Cons` (fas <*> as)

      That’s it. Function pure will produce a vector of just the right length.

      Trees

      Unlike the case for lists, it’s hard to define an Applicative instance for non-shape-encoded trees. Let’s have a look.

      instance Applicative Tree where
      pure a = Leaf a
      (Leaf fa) <*> (Leaf b) = Leaf (fa b)
      (Branch fa fb) <*> (Branch a b) = Branch (fa <*> a) (fb <*> b)
      (Leaf fa) <*> (Branch a b) = {- ? -} undefined
      (Branch fa fb) <*> (Leaf a) = {- ? -} undefined

      This problem has been noticed before on the Haskell-beginners mailing list. The response is interesting because it mentions the “unpleasant property of returning infinite tree[s]”; the same problem we had with lists!

      With shape-encoded trees this is not a problem. Function pure produces a tree of the appropriate shape. Also, note how the head of the second instance mirrors the definition of the Branch constructor (:: Tree m a -> Tree n a -> Tree (m,n) a)

      instance Applicative (Tree ()) where
      pure a = Leaf a
      Leaf fa <*> Leaf a = Leaf (fa a)

      instance (Applicative (Tree m), Applicative (Tree n))
      => Applicative (Tree (m,n)) where
      pure a = Branch (pure a) (pure a)
      (Branch fs ft) <*> (Branch s t) = Branch (fs <*> s) (ft <*> t)

      Arbitrary binary associative operators.

      Phew, that’s it. We now have an implementation for dot that will work on an arbitrary data structure as long as one can define Functor, Foldable and Applicative instances. We have also learned that it is a good idea to encode the data structure’s shape in its type so that Applicative instances can be defined. (This will be important later on when we want to take the transpose of generic matrices, but I’m getting ahead of myself.)

      But what if you want to use binary associative operators other than addition and multiplication for the dot product? This is easy using Haskell’s Monoid type class, and it plays nicely with the Foldable type class. In fact, it allows us to omit any mention of identity elements using the method fold:: (Foldable t, Monoid m) => t m -> m. We define an even more generic dot product as follows:

      dotGen :: (Foldable f, Applicative f, Monoid p, Monoid s)
      => (a -> p, p -> a) -> (a -> s, s-> a) -> f a -> f a -> a
      dotGen (pinject, pproject) (sinject, sproject) x y =
      sproject . fold . fmap (sinject . pproject) $ liftA2 mappend px py
      where
      px = fmap pinject x
      py = fmap pinject y

      This function takes two pairs of functions for injecting into and projecting from monoids. We can then define our original dot function using the existing Sum and Product wrapper types.

      dot :: (Num a, Foldable f, Applicative f) => f a -> f a -> a
      dot = dotGen (Product, getProduct) (Sum, getSum)

      In the next episode…

      In my next post we will consider generic matrix multiplication. This operation is defined over arbitrary collections of collections of numbers and, naturally, makes use of our generic dot product. Until then, adios.

      Slides

      On 17 Nov 2011 I gave a talk at fp-syd about this work. You can find the slides here.

      June 27, 2011 12:00 AM

      June 23, 2011

      Gergely Patai

      Renewed LambdaCube, Bullet bindings and Stunts example

      After many long hours of labour, I’m happy to announce the new release of the LambdaCube 3D engine on Hackage! Almost two years passed since the previous one, and the project spent the majority of that time in dormancy. The most important changes are the following (mostly straight from the latest HCAR):
      • removed dependency on the high-level OpenGL bindings: from now on, the library only builds on OpenGLRaw, and the OpenGL specific code is limited to the GL render system module, which we plan to move into a separate package;
      • switched to the vect library (from Vec), which was created for 3D applications from the get-go;
      • dropped fgl dependency for scene graph handling, and switched to a simpler and much more efficient solution based on bytestring-trie;
      • simplified support for procedurally created content through vector vertex buffers, which subsumes user-supplied loaders for any format as a special case;
      • introduced the LCM (LambdaCube Monad) abstraction, which hides the management of the world state and generally simplifies the engine code;
      • more efficient scene management with frustum culling;
      • added support for light sources;
      • identified the public interface and documented it (the rest is only exposed for the sake of library implementers).

      Besides the engine, we updated the bindings for the Bullet physics library as well, which covers a big portion of the existing functionality in its present state. This is a raw binding that exposes a C style interface.

      We also set out to create a complex example that people can immediately start playing with. This is a modern remake of the classic racing game Stunts, available on Hackage under the same name. The example serves two primary purposes: it is a test case for the 3D engine and the physics binding as well as a starting point for future users of the library. We made sure that everything compiles and works fine on the three major platforms supported by GHC. The quickest way to see the example in action is to try the Windows executable, which is also handled fine by Wine. Fortunately, the compilation process is less painful than one would expect (even on Windows!), so those who want to start hacking on it right away should feel encouraged to do so; we provided a starter guide to make the first steps easier. For the even lazier ones, here’s a taste of the example:

      <iframe allowfullscreen="" frameborder="0" height="349" src="http://www.youtube.com/embed/kDu5aCGc8l4" width="560"></iframe>

      Plans for the future

      One of the original design goals was to create an engine that can handle Ogre3D content out of the box. Unfortunately, this proved to be a bad decision, since we ended up with a lot of fixed-function legacy right away, and the implementation in its current form is rather wasteful of resources due to its naivety, especially with respect to the choice of data structures.

      The next version will definitely get rid of all the Ogre3D specific functionality. Our plan is to create a small and efficient core package that allows the programmer to easily set up the rendering pipeline and manage mesh hierarchies. We are targeting the OpenGL 3.3 core profile, so the engine will build exclusively on the programmable pipeline. The current idea is to define a simple DSL for describing pipelines that’s similar to GPipe in its basic philosophy, but not embedded in Haskell, which allows us to have full control over its structure. The engine would provide an API to build pipelines (allocating buffer objects and managing shader programs) out of these descriptions and to feed them with content (textures and primitive streams resulting from the flattening of the scene), while taking care of caching and optimising context switches. All this functionality would be wrapped in a declarative interface, naturally.

      While it would be possible to create separate packages to support existing formats (e.g. Ogre3D or COLLADA), we have something much more exciting in mind: integration with Blender. Blender can load content in several formats and present it in a basic form, reducing everything to raw vertex and index buffers, and the data can be accessed from Haskell code through Thrift. It is also possible to wrap LambdaCube in a rendering plugin that would display content inside Blender, thereby giving us an integrated content authoring solution. Some of this actually works already, but it’s not ready yet for public consumption.

      By the way, Csaba will be present on this year’s CamHac, so if you’re one of the attendants, feel free to bombard him with questions and ideas!

      by Patai Gergely (noreply@blogger.com) at June 23, 2011 04:30 PM

      Noam Lewis

      I, Robot

      So what? I’ve graduated (actually almost a year ago).

      You can see the presentation I gave for my project:
      <iframe frameborder="0" height="342" marginheight="0" marginwidth="0" src="https://docs.google.com/present/embed?id=dgc77gfh_27g5tb89z5" width="410"></iframe>

      Here is the robot from my project:

      Haskell-controlled Segway RMP

      And here is a draft of the poster I made (click for larger version):

      A picture from the projects conference:

      Robot + Poster in conference


      by sinelaw at June 23, 2011 07:01 AM

      June 21, 2011

      Holumbus

      Last Bug Standing

      We just launched a new version of the Hayoo! indexer, which finally fixes the last remaining major issue: Functions from type classes are now included in the index as well and hence available in the search. So, if you ever wondered, why ‘compare’ of the ‘Ord’ class never showed up in the search results … now you know.

      Additionally, a rebrush of the Hayoo! web frontend is on the way, now using the latest and greatest in available Haskell web technology: The Snap framework. Although overall Hayoo! development slowed down a bit, I think it is a good guess to expect the new frontend going live somewhen this year.

      Until then, have fun searching for the best stuff our beloved programming language has to offer!

      Oh, and as always, comments and suggestions are highly appreciated. Even better, contribute! The code is on github, go grab a copy!

      by tbh at June 21, 2011 09:40 PM

      June 20, 2011

      David Sankel

      Haskell’s evaluation isn’t magic

      One common complaint with lazy functional programming is that reasoning about the performance of your code is harder than in a strict language. Will an expression be repeatedly evaluated or a closure repeatedly accessed? Will code use up more or less heap space? For that matter, many people have no clue as to how lazy evaluation works in the first place.
      Frustrated Lady

      This blog post show that execution in a lazy language is both understandable and predictable.

      Haskell is in essence a heavily sugared and extended version of a typed λ-calculus (i.e. λ-calculus with a type system attached). If you don’t understand simple λ-calculus, you should read a tutorial, such as the first two chapters of Introduction to Lambda Calculus[pdf]. We won’t be doing anything too complicated with it, but you should understand the basic syntax and ideas behind it. If we want to understand Haskell’s execution, we should should first understand call-by-need evaluation of λ-calculus.

      In 1993 Launchbury wrote A Natural Semantics for Lazy Evaluation [pdf], which provided a simple operational semantics (I’ll explain what that is in a second) for understanding the lazy evaluation of an extended λ-calculus. The most important extension is the let statements: “lets are irreplaceable in lazy functional programming in that they may create cyclic structures in the heap. This cannot be achieved without them (or without something essentially equivalent), and cyclic structures have a huge implication for the amount of computation that may be performed” . He also includes constructors, case statements, and primitives in an extension to the base semantics. Operational semantics is very appropriately named: it provides semantics, or a meaning, to the code in terms of its operation on some abstract machine. What precisely is an abstract machine? Basically, it’s a mathematical model of a machine. Rather than create a physical machine with moving or electronic parts, you create a transition function. This function takes the current state (e.g. the current heap, stack, and the expression you want to reduce), and produces the next state (say, the heap remained the same, but you pushed something onto the stack and you’re evaluating some subexpression). Launchbury’s semantics are reasonably high-level and simple, but still capture sharing and heap usage.

      Before we continue, let’s remind ourselves how the λ-calculus is evaluated. There are several strategies for reducing a λ-term, which have different pros and cons. In normal order reduction, the outermost leftmost redex (reducible expression, i.e. something of the form (λx.y)z is reduced. This means that you don’t evaluate the arguments to a function until after they’re plugged into the function. Call-by-name is similar to normal order, except that you don’t reduce anything inside a λ-abstraction. Call-by-need is very similar to call-by-name, but with one key difference: call-by-name implements sharing (I’ll explain what that is momentarily). Call-b-name and call-by-need stand in contrast with call-by-value reduction, where you reduce the outermost redex only after you reduce its right hand side (i.e. the arguments to the function get evaluated before the function does).

      What does sharing mean? It means that arguments to a function are only evaluated once; the calculated values are “shared” between their uses. For example, consider the expression (λx. mult x x) (ackermann 4 1). Under call-by-value, you first reduce this to (λx. mult x x) 65533, and then to mult 65533 65533, then to 4,294,574,089. In call-by-name, you first reduce it to mult (ackermann 4 1) (ackermann 4 1). You then evaluate (ackermann 4 1) twice, because it appears twice. Call-by-need gets rid of this repeated evaluation: the first time you evaluate (ackermann 4 1), you replace both occurrences of it with 65533. Obviously, this is a very important optimization. It’s important to point out here that sharing doesn’t mean Common Subexpression Elimination; it only means that arguments are evaluated only once.

      Before you can apply the reduction rules, Launchbury’s semantics require a term to be normalized. A Launchbury-normalized term is one in which every name is unique (so scope isn’t important because something in an inner scope cannot shadow something in an outer scope), and which every application involves applying a variable to an expression. Normalization is simple: just replace every variable with a fresh variable (so everything becomes distinctly named), and replace constructs of the form “e1 e2″ with “let x = e2 in e1 x”. So, in our previous example, (λx. mult x x) (ackermann 4 1) is normalized to let z = expensive 4 1 in (λx. mult x x) z. Since we will store our let-bound variables on the heap, this means that implementing sharing is trivial: we update the reference on the heap the first time we evaluate it, and just use that value every other time.

      In order to help illustrate the semantics, I’ve written up a small interpreter which takes in a normalized expression and reduces it, returning both the reduced expression and a log of how it got there. I didn’t implement a normalizer; instead expressions need to be normalized by hand.

      Let’s go over the interpreter. I’ll leave out constructors, primitives and case statements, although they’re on hackage. I’ll also leave out the implementation of the various helper functions; their details aren’t important to understanding the semantics.

      To start out with, we should define some types to represent an expression:

      type Name = String 
       
      data Expr =  Lambda Name Expr
                 | Apply Expr Name
                 | Var Name
                 | Let Bindings Expr
                 deriving (Eq, Show)
       
      type Binding = (Name,Expr)
      type Bindings = [Binding])
       
      type Heap = [(Var, Expr)]
       
      type ReduceM a =  ...
       
      --record that current state
      data ReduceState = ...

      Expr is fairly simple: It represents the syntax tree of lambda expressions extended with let. We’ll represent variables as strings, because it’s convenient. A Heap, with respect to the paper, is “a partial function from variables to expressions”, however, you can also think of it as “an unordered set of variable/expression pairs, binding distinct variable names to expressions”. The latter interpretation is easier to work with, so we’ll be using it. I’m leaving out the implementation of Show, but the Hackage code includes an instance of show that outputs reasonable looking code. ReduceM keeps track of the state of the heap, the fresh variables, and the log, as well as allowing reduce to fail.

      Here are the functions to make working with ReduceM reasonable:

       
       
      rmRun :: ReduceM a → ReduceState → (Either String a, ReduceState)
       
      -- hides the implementation detail of failing; makes it easier to
      -- swap out the underlying monad.
      rmErr :: String → ReduceM Expr
       
      -- |Like sub, but for a list of things to substite
      -- useful for implementing recursive lets (i.e. letrec)
      subs :: [(Name,Name)](Expr → Expr)
       
      -- |e[x/y] in Launchbury's notation
      --  [x ↦ y]e in Pierce's notation in TaPL
      --  recursively descend expression tree to substitute a free variable
      sub ::  Name → Name → (Expr → Expr)
       
      -- |freshen takes an expression, and returns the same expression with every 
      -- bound variable substituted for a fresh variable. 
      freshen :: Expr → ReduceM Expr
       
      type ErrorOr a = Either String a
      type Log = String
       
      -- look up a binding on the heap
      heapLookup :: Name → ReduceM Expr
       
      -- remove a binding from the heap
      heapRemove :: Name → ReduceM ()
       
      -- create a new binding on the heap
      heapAdd :: Name → Expr → ReduceM ()
       
      --  get the next fresh variable.  Fresh variables are of the form $23, for random
      -- values of 23
      getFreshVar :: ReduceM Name
       
      -- reduces an expression.
      reduceM :: Expr → ReduceM Expr

      Freshen and sub are important, but the real work of the program is done in reduceM; that’s where the reduction actually happens. So what does the code for it look like?

      reduceM :: Expr → ReduceM Expr
      reduceM e =
          case e of
              Lambda e' x → return (Lambda e' x)  
              Apply e' x  →  do Lambda y' e'' ← reduceM e'
                                reduceM (sub y' x e'')
              Var x →       do e' ← heapLookup x
                               heapRemove x
                               z ← reduceM e'
                               heapAdd x z
                               freshen z
              Let bs e' →  do mapM_ (uncurry heapAdd) bs
                              reduceM e'

      Let’s go through each of these cases one by one.

      First: Lambda. The only sensible thing to do here is to just return the λ-abstraction, because we’re not allowed to reduce inside of a λ-abstraction.

      Second: Apply. In order to actually apply a variable to a subexpression, we need a function to apply it to. However, the subexpression might not be a λ-abstraction. Instead, it could be something that returns a λ-abstraction – a variable, a let, or even another apply! The only sensible thing to do here is to reduce our subexpression, and then apply the variable to it.

      Third: Variable. In order to reduce a variable x, we must first know what it is bound to on the heap. So, we look up the expression e that it is bound to. The next step is less obvious: we reduce e with x’s binding removed from the heap, before returning a freshened version of the result, to avoid unwanted variable capture. Is this correct?

      Obviously, if the expression isn’t recursive, we’re fine – it’s never going to look at its own binding. What happens in that case if x is recursive? Well, there are two possibilities – either the sub-expression depends on itself before it reduces, or it depends on itself after it reduces. If it depends on itself after it reduces, then we’re obviously fine – the binding will already be back on the heap. What about the other case? We’re actually still fine: If you depend on yourself before you reduce to a value, then you must be an infinite loop. In this case, we fail outright, because we’re trying to look up a binding which doesn’t exist. Failing outright and having an infinite loop are the same thing, semantically speaking, so we’re actually still fine.

      Here’s some examples of recursion, both good and bad. Assume the appropriate functions have been defined for the third example. It also hasn’t been properly normalized, but ignore that as well. Can you figure out what these will do? The answers will be at the bottom of the post

      let x = x in x 
      let f = λx.f x in f 2
      let len = λlist. if (empty? list) 0 (1 + (len (cdr list))) in len some-list
      

      Fourth: Let. Since everything is distinctly named, we don’t need to worry about scope; names will never clash because each name can only be used once. Therefore, it’s obviously correct to just add the bindings to the heap.

      Now that we’ve explained this, let’s look at the output of some runthroughs:

      Reducing let: let u = 3 + 2, v = u + 1 in v + v : {}

      How should we interpret this? Well, it’s fairly simple. In the beginning,we start to reduce the expression ‘let u = 3 + 2, v = u + 1 in v + v’, using an empty heap (denoted ‘{}’).

      |Reducing primitive: v + v : {v → u + 1, u → 3 + 2}

      We then add the bindings to the heap and take a look at the addition ‘v + v’.

      ||Reducing variable: v : {v → u + 1, u → 3 + 2}

      In order to reduce ‘v + v’, you first need to evaluate v.

      |||Reducing primitive: u + 1 : {u → 3 + 2}
      ||||Reducing variable: u : {u → 3 + 2}

      To calculate v, we need to evaluate u.

      |||||Reducing primitive: 3 + 2 : {}
      ||||||Returning constructor: 3 : {}
      ||||||Returning constructor: 2 : {}
      |||||Primitive evaluated to 5
      ||||Rebinding var u to 5
      ||||Returning constructor: 1 : {u → 5}
      |||Primitive evaluated to 6
      ||Rebinding var v to 6

      So we evaluate u, then evaluate v with the result of u.

      ||Reducing variable: v : {v → 6, u → 5}

      We’re re-grabbing v here for the second usage of v in v+v

      |||Returning constructor: 6 : {u → 5}
      ||Rebinding var v to 6
      |Primitive evaluated to 12
      Ans: 12

      Note that we access v two times in this, but we only access u a single time.

      For another two test cases, let’s take a look at two examples from Launchbury’s paper:

      First: let u = 2 + 3, f = \x.let v = u + 1 in v + x, a = 2, b = 3 in f a + f b

      and

      Second: let u = 2 + 3, f = let v = u + 1 in \x.v + x, a = 2, b = 3 in f a + f b

      Note that the expressions are the same except for f’s definition; in the first, f is “\x.let v = u + 1 in v + x,” and in the second f is “let v = u + 1 in \x.v + x”. Obviously, these two are going to evaluate to the same thing. However, what’s going to be the difference in terms of how they reduce? Does one share sub-computations more effectively than the other to reduce the amount of computation needed? Let’s take a look at the log produced by running them to figure that out.

      Reducing let: let u = 2 + 3, f = \x.let v = u + 1 in v + x, a = 2, b = 3 in f a + f b : {}
      |Reducing primitive: f a + f b : {b → 3, a → 2, f → \x.let v = u + 1 in v + x, u → 2 + 3}
      ||Reducing apply: f a : {b → 3, a → 2, f → \x.let v = u + 1 in v + x, u → 2 + 3}
      |||Reducing variable: f : {b → 3, a → 2, f → \x.let v = u + 1 in v + x, u → 2 + 3}
      ||||Returning lambda: \x.let v = u + 1 in v + x : {b → 3, a → 2, u → 2 + 3}
      |||Rebinding var f to \x.let v = u + 1 in v + x
      |||Reducing let: let $2 = u + 1 in $2 + a : {f → \x.let v = u + 1 in v + x, b → 3, a → 2, u → 2 + 3}
      ||||Reducing primitive: $2 + a : {$2 → u + 1, f → \x.let v = u + 1 in v + x, b → 3, a → 2, u → 2 + 3}
      |||||Reducing variable: $2 : {$2 → u + 1, f → \x.let v = u + 1 in v + x, b → 3, a → 2, u → 2 + 3}
      ||||||Reducing primitive: u + 1 : {f → \x.let v = u + 1 in v + x, b → 3, a → 2, u → 2 + 3}
      |||||||Reducing variable: u : {f → \x.let v = u + 1 in v + x, b → 3, a → 2, u → 2 + 3}
      ||||||||Reducing primitive: 2 + 3 : {f → \x.let v = u + 1 in v + x, b → 3, a - &gt; 2}
      |||||||||Returning constructor: 2 : {f → \x.let v = u + 1 in v + x, b → 3, a - &gt; 2}
      |||||||||Returning constructor: 3 : {f → \x.let v = u + 1 in v + x, b → 3, a - &gt; 2}
      ||||||||Primitive evaluated to 5
      |||||||Rebinding var u to 5
      |||||||Returning constructor: 1 : {u → 5, f → \x.let v = u + 1 in v + x, b → 3, a → 2}
      ||||||Primitive evaluated to 6
      |||||Rebinding var $2 to 6
      |||||Reducing variable: a : {$2 → 6, u → 5, f → \x.let v = u + 1 in v + x, b → 3, a → 2}
      ||||||Returning constructor: 2 : {$2 → 6, u → 5, f → \x.let v = u + 1 in v + x, b → 3}
      |||||Rebinding var a to 2
      ||||Primitive evaluated to 8
      ||Reducing apply: f b : {a → 2, $2 → 6, u → 5, f → \x.let v = u + 1 in v + x , b → 3}
      |||Reducing variable: f : {a → 2, $2 → 6, u → 5, f → \x.let v = u + 1 in v + x, b → 3}
      ||||Returning lambda: \x.let v = u + 1 in v + x : {a → 2, $2 → 6, u → 5, b → 3}
      |||Rebinding var f to \x.let v = u + 1 in v + x
      |||Reducing let: let $4 = u + 1 in $4 + b : {f → \x.let v = u + 1 in v + x, a - &gt; 2, $2 → 6, u → 5, b → 3}
      ||||Reducing primitive: $4 + b : {$4 → u + 1, f → \x.let v = u + 1 in v + x, a → 2, $2 → 6, u → 5, b → 3}
      |||||Reducing variable: $4 : {$4 → u + 1, f → \x.let v = u + 1 in v + x, a → 2, $2 → 6, u → 5, b → 3}
      ||||||Reducing primitive: u + 1 : {f → \x.let v = u + 1 in v + x, a → 2, $2 → 6, u → 5, b → 3}
      |||||||Reducing variable: u : {f → \x.let v = u + 1 in v + x, a → 2, $2 → 6, u → 5, b → 3}
      ||||||||Returning constructor: 5 : {f → \x.let v = u + 1 in v + x, a → 2, $2 → 6, b → 3}
      |||||||Rebinding var u to 5
      |||||||Returning constructor: 1 : {u → 5, f → \x.let v = u + 1 in v + x, a → 2, $2 → 6, b → 3}
      ||||||Primitive evaluated to 6
      |||||Rebinding var $4 to 6
      |||||Reducing variable: b : {$4 → 6, u → 5, f → \x.let v = u + 1 in v + x, a → 2, $2 → 6, b → 3}
      ||||||Returning constructor: 3 : {$4 → 6, u → 5, f → \x.let v = u + 1 in v + x, a → 2, $2 → 6}
      |||||Rebinding var b to 3
      ||||Primitive evaluated to 9
      |Primitive evaluated to 17
      Ans: 17
       
      Reducing let: let u = 2 + 3, f = let v = u + 1 in \x.v + x, a = 2, b = 3 in f a + f b : {}
      |Reducing primitive: f a + f b : {b → 3, a → 2, f → let v = u + 1 in \x.v + x , u → 2 + 3}
      ||Reducing apply: f a : {b → 3, a → 2, f → let v = u + 1 in \x.v + x, u → 2 + 3}
      |||Reducing variable: f : {b → 3, a → 2, f → let v = u + 1 in \x.v + x, u → 2 + 3}
      ||||Reducing let: let v = u + 1 in \x.v + x : {b → 3, a → 2, u → 2 + 3}
      |||||Returning lambda: \x.v + x : {v → u + 1, b → 3, a → 2, u → 2 + 3}
      |||Rebinding var f to \x.v + x
      |||Reducing primitive: v + a : {f → \x.v + x, v → u + 1, b → 3, a → 2, u → 2 + 3}
      ||||Reducing variable: v : {f → \x.v + x, v → u + 1, b → 3, a → 2, u → 2 + 3}
      |||||Reducing primitive: u + 1 : {f → \x.v + x, b → 3, a → 2, u → 2 + 3}
      ||||||Reducing variable: u : {f → \x.v + x, b → 3, a → 2, u → 2 + 3}
      |||||||Reducing primitive: 2 + 3 : {f → \x.v + x, b → 3, a → 2}
      ||||||||Returning constructor: 2 : {f → \x.v + x, b → 3, a → 2}
      ||||||||Returning constructor: 3 : {f → \x.v + x, b → 3, a → 2}
      |||||||Primitive evaluated to 5
      ||||||Rebinding var u to 5
      ||||||Returning constructor: 1 : {u → 5, f → \x.v + x, b → 3, a → 2}
      |||||Primitive evaluated to 6
      ||||Rebinding var v to 6
      ||||Reducing variable: a : {v → 6, u → 5, f → \x.v + x, b → 3, a → 2}
      |||||Returning constructor: 2 : {v → 6, u → 5, f → \x.v + x, b → 3}
      ||||Rebinding var a to 2
      |||Primitive evaluated to 8
      ||Reducing apply: f b : {a → 2, v → 6, u → 5, f → \x.v + x, b → 3}
      |||Reducing variable: f : {a → 2, v → 6, u → 5, f → \x.v + x, b → 3}
      ||||Returning lambda: \x.v + x : {a → 2, v → 6, u → 5, b → 3}
      |||Rebinding var f to \x.v + x
      |||Reducing primitive: v + b : {f → \x.v + x, a → 2, v → 6, u → 5, b → 3}
      ||||Reducing variable: v : {f → \x.v + x, a → 2, v → 6, u → 5, b → 3}
      |||||Returning constructor: 6 : {f → \x.v + x, a → 2, u → 5, b → 3}
      ||||Rebinding var v to 6
      ||||Reducing variable: b : {v → 6, f → \x.v + x, a → 2, u → 5, b → 3}
      |||||Returning constructor: 3 : {v → 6, f → \x.v + x, a → 2, u → 5}
      ||||Rebinding var b to 3
      |||Primitive evaluated to 9
      |Primitive evaluated to 17
      Ans: 17

      As you can immediately see, the first one has a larger heap than the second one does. The reason for that becomes clear when you look at it more closely: In the second one, when we reduce f we add the let bindings to the heap once, because it’s outside the enclosing lambda. In the first one, on the other hand, f doesn’t reduce at all, because the lambda is the outermost thing. On account of that, we end up adding “v = u + 1″ to the heap twice in the first case, which leads to performing that calculation twice. In the second expression, on the other hand, v is added to the heap the first time we call f, so both invocations share the same heap space for v, which means that u is only accessed a single time.

      Based off of those logs, it is reasonable to expect that the first expression will run faster. So, lets test that. Here are those two functions, translated straightforwardly into Haskell. We’ll run them in Hugs, since Hugs will print the number of heap cells used and reductions made. Smaller numbers for both are better.

      slowExprHaskell = let u = 3+2
                            f = \x → let v = u+1 in v + x
                        in f 2 + f 3
       
      fastExprHaskell = let u = 3+2
                            f = let v = u+1 in \x → v + x
                        in f 2 + f 3- results from Hugs:
      Main$ slowExprHaskell
      17
      (47 reductions, 86 cells)
      Main$ fastExprHaskell
      17
      (43 reductions, 78 cells)

      This is what we expected.

      What key lessons can you take from this to use in your everyday Haskell coding? First, always prefer to create a let outside of a lambda rather than inside of a lambda, to take greater advantage of sharing. Second, lazy evaluation is not black magic, it doesn’t evaluate things in an order based off of the phase of the moon; it is deterministic and follows fairly simple semantics. While tracing impure lazy computations is very difficult, it is not too hard to get a reasonable idea of how a pure lazy function will be evaluated.

      (The answers to the questions posed earlier are that the first expression halts because it uses itself before it has been rebound on the heap, the second expression is an infinite loop, and the third one is a normal recursive function that will reduce to the correct answer.)

      by pipoca at June 20, 2011 06:30 PM

      June 17, 2011

      Neil Brown

      Pieces of Yesod: Inverting a Haskell Function

      I’ve recently been working with web frameworks, and decided to have a play with what Haskell has to offer for web resources. One of the several frameworks available is Yesod. Web frameworks need some sort of mapping between URLs and internal logic, usually called routing. In Yesod, routing is done with a “pieces” mechanism. You might declare in your routes:

      /resources/#SortBy ResourcesR GET

      And this will mean that if you access example.com/resources/d, that will translate into a call that turns the string "d" into a value of type SortBy, which is then passed to getResourcesR. It also means that if you use the syntax “@{ResourceR Date}” in your template files, this will be translated into example.com/resources/d. This translation is done is with the “pieces” classes, for example the single piece class:

      class SinglePiece a where
        fromSinglePiece :: Text -> Maybe a
        toSinglePiece :: a -> Text

      So a value should always be translatable into a part of a URL (i.e. Text), and an arbitrary URL part may be a valid value. Here’s a simple sorting-order type:

      data SortBy = Date | Popularity | Title

      And here’s the trivial SinglePiece instance:

      instance SinglePiece SortBy where
        fromSinglePiece "d" = Just Date
        fromSinglePiece "p" = Just Popularity
        fromSinglePiece "t" = Just Title
        fromSinglePiece _ = Nothing
      
        toSinglePiece Date = "d"
        toSinglePiece Popularity = "p"
        toSinglePiece Title = "t"

      There’s several drawbacks with this code. The major drawback is obvious: the code is repetitive! It expresses the same simple mapping from sort order to text twice, once in each direction. This is tedious. Another drawback is that is a potential source of bugs if I make a typo. SinglePiece does have an obvious property for testing: fromSinglePiece . toSinglePiece == Just. I also need to make sure I cover all the possible sorting values. The nice thing about toSinglePiece (rather than using a lookup table) is that if I use the GHC flags -fwarn-incomplete-patterns -Werror, it will warn me if I miss out a case. So if I add a new sort order, say Author, and forget to change toSinglePiece, I’ll get a compile-time error as soon as I try to compile (which is quicker even than a test). But fromSinglePiece doesn’t have the same advantage: if I forget a new case for Author, I’ll only know about it because of my test.

      The way to fix my problems is to stop repeating myself. I can change the code to this:

      instance SinglePiece SortBy where
        fromSinglePiece = invert toSinglePiece
        toSinglePiece Date = "d"
        toSinglePiece Popularity = "p"
        toSinglePiece Title = "t"

      That invert function looks a bit magical! Broadly, its type is:

      invert :: (a -> b) -> (b -> Maybe a)

      Now, as it stands, the invert function is not implementable; there’s no way to arbitrarily invert a Haskell function. But we can brute force an inversion if we know the full range of inputs. That’s possible with the help of some automatically derivable Haskell type-classes:

      data SortBy = Date | Popularity | Title
        deriving (Eq, Enum, Bounded)
      
      invert :: (Enum a, Bounded a, Eq b) => (a -> b) -> (b -> Maybe a)
      invert f = flip lookup $ map (f &&& id) [minBound..maxBound]

      We use Enum and Bounded to get the list of all possible inputs, then form a lookup table which we can reference to find the inverse of a value. So now we’re able to write just toSinglePiece, and automatically derive fromSinglePiece. No repetition and less possible source of mistakes. The only problem is that invert might be a bit slow. Instinctively, it feels like the lookup table can’t be as fast as Haskell’s built-in case structure. To find out, let’s whip up a test using the Criterion benchmarking library.

      I’m testing the performance of fromSinglePiece on the three valid URLs and one invalid, i.e.

      main = defaultMain
        [bench "benchName" $ nf (map fromSinglePiece :: [Text] -> [Maybe SortBy])
           ["d", "p", "t", "x"]]
      

      I ran this once on the original fromSinglePiece that used a case statement (calling that “cases”), and once on the new fromSinglePiece that uses the lookup table (calling that “invert”), compiling with -O2 each time on GHC 7.0.3. Here’s the results:

      benchmarking cases
      collecting 100 samples, 23951 iterations each, in estimated 1.011024 s
      bootstrapping with 100000 resamples
      mean: 423.0417 ns, lb 421.8167 ns, ub 426.2270 ns, ci 0.950
      std dev: 9.136906 ns, lb 1.879618 ns, ub 16.88352 ns, ci 0.950
      found 2 outliers among 100 samples (2.0%)
        2 (2.0%) high severe
      variance introduced by outliers: 0.997%
      variance is unaffected by outliers
      
      benchmarking invert
      collecting 100 samples, 21664 iterations each, in estimated 1.011000 s
      bootstrapping with 100000 resamples
      mean: 466.8043 ns, lb 466.7298 ns, ub 466.9089 ns, ci 0.950
      std dev: 448.0250 ps, lb 340.4309 ps, ub 723.9564 ps, ci 0.950
      found 2 outliers among 100 samples (2.0%)
        1 (1.0%) high severe
      variance introduced by outliers: 0.990%
      variance is unaffected by outliers
      

      So the difference in speed is only around 10%. I can accept that cost in order to not repeat myself everywhere. Obviously, many types used with SinglePiece are not a trivial conversion between a type constructor and a text value, but for those that are, I found this trick useful.


      by Neil Brown at June 17, 2011 02:25 PM

      June 08, 2011

      Neil Brown

      Testing a Side-Effecting Haskell Monad

      In this post I’m going to continue discussing testing CHP, this time focusing on the CHP monad. The monad is a bit complex (and indeed, there is a subtle bug in the current version of CHP to do with poison-handling) because it has several semantic aspects:

      • it is an error-handling/short-circuiting monad (because it must obey the poison semantics),
      • it has some extra semantics to do with choice that treats the leading action differently,
      • underneath it all, there is the IO monad, which must be lifted correctly.

      CHP is a monad with side-effects: quite aside from the communication with parallel processes, the presence of lifted IO means that it can have arbitrary side-effects. This means, if my terminology is correct, that we need a check for operational equality rather than denotational. Let’s explore this by considering testing an error-handling monad.

      Testing An Error-Handling Monad

      The Either type can be used as an error-handling monad by using the Left constructor to hold an error, and the Right constructor to hold a successful return. One property that should hold is that Left e >>= k == Left e. That’s an actual Haskell equality; we can test that property for any function k, and use actual equality on the result.

      It’s also possible to use a similar error-handling monad transformer. The transformers library offers an ErrorT e m a type that allows you to wrap a monad “m”, using error type “e”, with successful returns of type “a” (with a function throwError :: e -> ErrorT e m a for introducing errors). We can test this using the Identity monad as type “m”, which would make sure the code was pure. But the semantics of this error transformer involve what actions occur in the error monad. For example, runErrorT (m >> throwError e >> n) should be equivalent to m >> return (Left e), but usually m (which is a monadic action) cannot be directly compared for equality.

      Setting this in the IO monad makes it all much clearer: runErrorT (lift (putStrLn "Hello") >> throwError e >> lift (putStrLn "Goodbye")) should print Hello on the screen, and then finish with an error — and not print Goodbye. Testing this is difficult because we need to have a harness that can check for which side-effects occurred, making sure the right ones (e.g. printing Hello) did happen, and the wrong ones (e.g. printing Goodbye) didn’t happen.

      This is exactly the situation that the CHP monad is in; we always have IO beneath the CHP monad (which we can’t swap out for a more easily testable monad!), so we must test equality of code by checking which side effects occurred. We need to check that liftIO_CHP (putStrLn "Hello") >> throwPoison >> liftIO_CHP (putStrLn "Goodbye") prints Hello then throws poison, and doesn’t print Goodbye. Printing text is a difficult side-effect to track in a test harness, but the nice thing about IO is that it has a whole host of side-effects to choose from! So I test by observing alterations to a Software Transactional Memory (STM) variable (TVar) — therefore my test inputs are of type TVar String -> CHP a.

      Testing CHP’s Error-Handling

      I use these two functions as a harness to execute the CHP code:

      runCHPEx :: CHP a -> IO (Either SomeException a)
      runCHPEx m = (Right <$> runCHP m) `C.catches`
          [Handler (\(e::UncaughtPoisonException) -> return $ Left $ toException e)
          ,Handler (\(e::ErrorCall) -> return $ Left $ toException e)]
      
      runCode :: (TVar String -> CHP a) -> IO (String, Either SomeException a)
      runCode m = do
        tv <- newTVarIO ""
        r <- runCHPEx (m tv)
        s <- atomically (readTVar tv)
        return (s, r)

      The runCode function gives back (in the IO monad) the String that was the final value of the TVar, and the result of running the CHP computation — either an uncaught poison/error exception or an actual return value. (One planned change to CHP semantics in version 3.0.0 is that uncaught poison gets translated into an exception, rather than giving runCHP a Maybe-wrapped return.) Here’s some example HUnit tests (with some useful helper functions):

      obsTestPoison :: Test
      obsTestPoison = TestList
        [("ab", Right 'b') ==* \tv -> tv!'a' >> tv!'b'
        ,("", Left UncaughtPoisonException) ==* const throwPoison
        ,("a", Left UncaughtPoisonException) ==* \tv -> tv!'a' >> throwPoison
        ]
        where
          (!) :: TVar String -> Char -> CHP Char
          (!) tv c = c <$ (liftIO_CHP $ atomically $
                             readTVar tv >>= writeTVar tv . (++[c]))
      
          (==*) :: (String, Either UncaughtPoisonException Char)
                -> (TVar String -> CHP Char) -> Test
          (==*) x m = TestCase $ runCode m >>= assertEqual "" (onLeftSnd Just x)
                        . onLeftSnd fromException
            where
              onLeftSnd :: (b -> c) -> (a, Either b d) -> (a, Either c d)
              onLeftSnd f (x, Left y) = (x, Left $ f y)
              onLeftSnd f (x, Right y) = (x, Right y)

      The left-hand side of the starred equality is the expected result. The right-hand side is the code (which is of type TVar String -> CHP Char) which should produce that result.

      So by observing deliberately-placed side-effects in the code, we can check for equality in a side-effecting monad. These unit tests aren’t the only way I’m testing my monad though — in my next post, I’ll build on this to form a more advanced property-based testing technique.


      by Neil Brown at June 08, 2011 01:26 PM

      Tillmann Vogt

      SVG-Fonts-0.4 released on Hackage

      This library has been laying around in an unusable state on my harddrive for a long time, but now it is finally good enough to be used. In contrast to a lot of graphics libraries on Hackage it is not a binding. That means there were a lot of algorithms that had to be implemented. Although Haskell is often propagated as a language to write correct programs, algorithmic errors are as common as in other languages, because they cannot be catched by the type system. Debug.Trace, debugging with ghci is no alternative because they have no visual output and a list of hundred of floats is not helpful. In the future I plan to have a visual debugger for this.

      But it works now. The interface should be easy to integrate into other libraries. Take a look at the following example:

      main = do
        args <- getArgs
        let str = if null args then "Haskell" else head args
            resolution = (300, 300)
      -- mode: INSIDE_V1_V2: the string is inside v1 v2 boundaries (height/length-relation not kept)
      -- mode: INSIDE_V1: stay inside v1 boundary, size of v2 adjusted to height/length-relation
      -- mode: INSIDE_V2: stay inside v2 boundary, size of v1 adjusted to height/length-relation
            mode = INSIDE_V2
      -- spacing: MONO: Use mono spacing between glyphs
      --          HADV: Every glyph has a unique constant horiz. advance
      --          KERN: Same as HADV but sometimes overridden by kerning: i.e. the horizontal advance
      --                in "VV" is bigger than in "VA"
            spacing = HADV
            gaw = makeOutlMap "../../../src/Test/GirlsareWeird.svg" resolution
            lin = makeOutlMap "../../../src/Test/LinLibertine.svg" resolution
            o = (0,0,1.1) -- origin
            v1 = (5,0,0) -- direction of char-advance
            v2 = (0,0,-1) -- height direction
            v3 = (0,0.2,0) -- extrusion
      
            f :: String -> [String]  -- assigning a property to every character by a unique string
            f str = replicate (length str) "p"
                                    -- data CharProp = Prop (FontData, OutlineMap) String Textured
            prop :: Map.Map String CharProp
            prop = Map.fromList [("p", Prop lin "to3d" False)]
            -- transformation of a Geoemtry Node (i.e. triangulation)
            transf :: Map.Map String (Geometry -> Geometry)
            transf = Map.fromList [("to3d", to3d)]
            -- to3d: not the perfect solution since some points are generated twice
            to3d geom = ( ((extrude (0,0.1,0)).deleteHoles) geom ) `atop`
                          ( tri (translate (0,0.1,0) geom) )
            tri = (triangulate ketTri).deleteHoles
      
            tex = makeTexMap resolution prop transf
            node = displayString str "node" resolution prop transf mode spacing o v1 v2 f tex
      
        genCollada (lightedScene node) emptyAnim
        putStrLn "Collada File generated"
      

      Line 30 shows that a integration into a drawing combinator library seems to be reasonable. I am a little bit proud that this library is lot smaller than freetype2 (something around 100KB against 4000KB)
      and the only things that it cannot do (yet!) is Rastered Images, Autohinting, Unicode, and the big amount of file formats supported by freetype2.
      Another difference is that the library consists of several small parts that can be used in many other places while a lot of C-libraries tend to be monolithic (like freetype2).
      I have to admit that parsing ttf and the other formats is maybe more work then the what has been done so far.
      Thanks to the developers of Blender who fixed a bug I described it is possible to load this file into Blender to generate a raytraced picture:

      Example Rendering with Blender




      by tillmannvogt at June 08, 2011 09:03 AM

      June 05, 2011

      Eric Kidd

      Screencast: Use Rails and RDF.rb to parse Best Buy product reviews

      In the past few years, many companies have been embedding machine-readable metadata in their web pages. Among these is Best Buy, which provides extensive RDFa data describing their products, prices and user reviews.

      The following 20-minute screencast shows how to use Ruby 1.9.2, Rails 3.1rc1, RDF.rb and my rdf-agraph gem to compare user ratings of the iPad and various Android Honeycomb tablets.

      Read More

      by Eric Kidd at June 05, 2011 07:07 PM

      June 04, 2011

      Conal Elliott

      A third view on trees

      A few recent posts have played with trees from two perspectives. The more commonly used I call "top-down", because the top-level structure is most immediately apparent. A top-down binary tree is either a leaf or a pair of such trees, and that pair can be accessed without wading through intervening structure. Much less commonly used are "bottom-up" trees. A bottom-up binary tree is either a leaf or a single such tree of pairs. In the non-leaf case, the pair structure of the tree elements is accessible by operations like mapping, folding, or scanning. The difference is between a pair of trees and a tree of pairs.

      As an alternative to the top-down and bottom-up views on trees, I now want to examine a third view, which is a hybrid of the two. Instead of pairs of trees or trees of pairs, this hybrid view is of trees of trees, and more specifically of bottom-up trees of top-down trees. As we’ll see, these hybrid trees emerge naturally from the top-down and bottom-up views. A later post will show how this third view lends itself to an in-place (destructive) scan algorithm, suitable for execution on modern GPUs.

      Edits:

      • 2011-06-04: "Suppose we have a bottom-up tree of top-down trees, i.e., t ∷ TB (TT a). Was backwards. (Thanks to Noah Easterly.)
      • 2011-06-04: Notation: "f ➶ n" and "f ➴ n".

      The post Parallel tree scanning by composition defines "top-down" and a "bottom-up" binary trees as follows (modulo type and constructor names):

      data TT a = LT a | BT { unBT  Pair (TT a) } deriving Functor

      data TB a = LB a | BB { unBB TB (Pair a) } deriving Functor

      So, while a non-leaf TT (top-down tree) has a pair at the top (outside), a non-leaf TB (bottom-up tree) has pairs at the bottom (inside).

      Combining these two observations leads to an interesting possibility. Suppose we have a bottom-up tree of top-down trees, i.e., t ∷ TB (TT a). If t is not a leaf, then t ≡ BB tt where tt is a bottom-up tree whose leaves are pairs of top-down trees, i.e., tt ∷ TB (Pair (TT a)). Each of those leaves of type Pair (TT a) can be converted to type TT a (single tree), simply by applying the BT constructor. Moreover, this transformation is invertible. For convenience, define a type alias for hybrid trees:

      type TH a = TB (TT a)

      Then the two conversions:

      upT    TH a  TH a
      upT = fmap BT ∘ unBB

      downT TH a TH a
      downT = BBfmap unBT

      Exercise: Prove upT and downT are inverses where defined.

      Answer:

        upT ∘ downT
      fmap BT ∘ unBB ∘ BBfmap unBT
      fmap BTfmap unBT
      fmap (BT ∘ unBT)
      fmap id
      id

      downT ∘ upT
      BBfmap unBT ∘ fmap BT ∘ unBB
      BBfmap (unBT ∘ BT) ∘ unBB
      BBfmap id ∘ unBB
      BBid ∘ unBB
      BB ∘ unBB
      id

      Consider a perfect binary leaf tree of depth n, i.e., an n-deep binary tree with each level full and data only at the leaves (where a leaf is depth 0 tree.) We can view such a tree as top-down, or bottom-up, or as a hybrid.

      Each of these three views is really n+1 views:

      • Top-down: a depth n tree, or a pair of depth n-1 trees, or a pair of pairs of depth n-2 trees, etc.
      • Bottom-up: a depth n tree, or a depth n-1 tree of pairs, or a depth n-2 tree of pairs of pairs, etc.
      • Hybrid: a depth n tree of depth 0 trees, or a depth n-1 tree of depth 1 trees, or, …, or a depth 0 tree of depth n trees.

      In the hybrid case, counting from 0 to n, the kth such view is a depth n-k bottom-up tree whose elements (leaf values) are depth k top-down trees. When k=n, we have a bottom-up tree whose leaves are all single-leaf trees, and when k=0, we have a single-leaf bottom-up tree containing a top-down tree. Imagine a horizontal line at depth k, dividing the bottom-up outer structure from the top-down inner structure. The downT function moves the dividing line downward, and the upT function moves the line upward. Both functions are partial.

      Generalizing

      The role of Pair in the tree types above is simple and regular. We can abstract out this particular type constructor, generalizing to an arbitrary functor. I’ll call this generalization "functor trees". Again, there are top-down and bottom-up versions:

      data FT f a = FLT a | FBT { unFBT  f (FT f a) } deriving Functor

      data FB f a = FLB a | FBB { unFBB FB f (f a) } deriving Functor

      And a hybrid version, with generalized versions of upT and downT:

      type FH f a = FB f (FT f a)

      upH Functor f FH f a FH f a
      upH = fmap FBT ∘ unFBB

      downH Functor f FH f a FH f a
      downH = FBBfmap unFBT

      These definitions specialize to the ones (for binary trees) by substituting Pair for the parameter f.

      Depth-typing

      The upward and downward view-changing functions above are partial, as they can fail at extreme tree views (at depth 0 or n). We could make this partiality explicit by changing the result type to Maybe (TH a) for binary hybrid trees and to Maybe (FH f a) for the functor generalization. Alternatively, make the tree sizes explicit in the types, as in a few recent posts, including A trie for length-typed vectors. (In those posts, I used the terms "right-folded" and "left-folded" in place of "top-down" and "bottom-up", reflecting the right- or left-folding of functor composition. The "folded" terms led to some confusion, especially in the context of data type folds and scans.) In the depth-typed versions, "leaves" are zero-ary compositions, and "branches" are (m+1)-ary compositions for some m.

      Top-down:

      data (➴)  (*  *)  *  (*  *) where
      ZeroT a (f ➴ Z) a
      SuccT IsNat n f ((f ➴ n) a) (f ➴ S n) a

      unZeroT (f ➴ Z) a a
      unZeroT (ZeroT a) = a

      unSuccT (f ➴ S n) a f ((f ➴ n) a)
      unSuccT (SuccT fsa) = fsa

      instance Functor f Functor (f ➴ n) where
      fmap h (ZeroT a) = ZeroT (h a)
      fmap h (SuccT fs) = SuccT ((fmap∘fmap) h fs)

      Bottom-up:

      data (➶)  (*  *)  *  (*  *) where
      ZeroB a (f ➶ Z) a
      SuccB IsNat n (f ➶ n) (f a) (f ➶ S n) a

      unZeroB (f ➶ Z) a a
      unZeroB (ZeroB a) = a

      unSuccB (f ➶ S n) a (f ➶ n) (f a)
      unSuccB (SuccB fsa) = fsa

      instance Functor f Functor (f ➶ n) where
      fmap h (ZeroB a) = ZeroB (h a)
      fmap h (SuccB fs) = SuccB ((fmap∘fmap) h fs)

      Hybrid:

      type H p q f a = (f ➶ p) ((f ➴ q) a)

      Upward and downward shift become total functions, and their types explicitly describe how the line shifts between (p+1)/q and p/(q+1):

      up    (Functor f, IsNat q)  H (S p) q f a  H p (S q) f a
      up = fmap SuccT ∘ unSuccB

      down (Functor f, IsNat p) H p (S q) f a H (S p) q f a
      down = SuccBfmap unSuccT

      So what?

      Why care about the multitude of views on trees?

      • It’s pretty.
      • A future post will show how these hybrid trees enable an elegant formulation of parallel scanning that lends itself to an in-place, GPU-friendly implementation.

      flattr this!

      by conal at June 04, 2011 02:46 AM

      June 03, 2011

      Mikhail Glushenkov

      GSoC 2011: Parallelising cabal-install

      This summer, I will be working on parallelising cabal-install under the aegis of the Google Summer of Code program. The aim of the project is to make Cabal utilise multiple threads for running the build process (a-la make -j). This means that Cabal will be able to make use of those shiny multi-core processors many developers now own. While initially the unit of granularity will be the package, in the second phase of the project I plan to add support for building even single modules in parallel (which will require interaction with ghc --make). Until my patches are accepted into the main repo, they’ll live on Darcden (not much there yet!).

      Thread communication in the prototype

      Thread communication in the prototype

      While I haven’t yet done much work on modifying the cabal-install proper, I’ve produced a small prototype that illustrates my approach to the problem. The prototype program consists of several threads which communicate via Chans. There are several worker threads, which compile the packages (threadDelay is used to simulate actual operations). A single control thread maintains the package graph and assigns tasks to the worker threads. A single logger thread prints out messages received from the worker threads. A single install thread installs the packages into the target directory (this is done serially, but can also be parallelized if deemed safe).

      After the install thread installs a package, it notifies the controller thread, which then updates the package graph and adds new tasks for the worker threads (if possible). The control thread terminates when the last package has been installed (which leads to the termination of all other threads).

      I still have exams until the 7th of June, but I’ve already posted my first patch. Though not directly related to the project, it helped me to smooth out some wrinkles in the workflow and get accustomed to the Darcs way of doing things.

      June 03, 2011 12:00 AM

      May 24, 2011

      Conal Elliott

      Parallel tree scanning by composition

      My last few blog posts have been on the theme of scans, and particularly on parallel scans. In Composable parallel scanning, I tackled parallel scanning in a very general setting. There are five simple building blocks out of which a vast assortment of data structures can be built, namely constant (no value), identity (one value), sum, product, and composition. The post defined parallel prefix and suffix scan for each of these five "functor combinators", in terms of the same scan operation on each of the component functors. Every functor built out of this basic set thus has a parallel scan. Functors defined more conventionally can be given scan implementations simply by converting to a composition of the basic set, scanning, and then back to the original functor. Moreover, I expect this implementation could be generated automatically, similarly to GHC’s DerivingFunctor extension.

      Now I’d like to show two examples of parallel scan composition in terms of binary trees, namely the top-down and bottom-up variants of perfect binary leaf trees used in previous posts. (In previous posts, I used the terms "right-folded" and "left-folded" instead of "top-down" and "bottom-up".) The resulting two algorithms are expressed nearly identically, but have differ significantly in the work performed. The top-down version does Θ(nlogn) work, while the bottom-up version does only Θ(n), and thus the latter algorithm is work-efficient, while the former is not. Moreover, with a very simple optimization, the bottom-up tree algorithm corresponds closely to Guy Blelloch’s parallel prefix scan for arrays, given in Programming parallel algorithms. I’m delighted with this result, as I had been wondering how to think about Guy’s algorithm.

      Edit:

      • 2011-05-31: Added Scan and Applicative instances for T2 and T4.

      Scanning via functor combinators

      In Composable parallel scanning, we saw the Scan class:

      class Scan f where
      prefixScan, suffixScan Monoid m f m (m, f m)

      Given a structure of values, the prefix and suffix scan methods generate the overall fold (of type m), plus a structure of the same type as the input. (In contrast, the usual Haskell scanl and scanr functions on lists yield a single list with one more element than the source list. I changed the interface for generality and composability.) The post gave instances for the basic set of five functor combinators.

      Most functors are not defined via the basic combinators, but as mentioned above, we can scan by conversion to and from the basic set. For convenience, encapsulate this conversion in a type class:

      class EncodeF f where
      type Enc f * *
      encode f a Enc f a
      decode Enc f a f a

      and define scan functions via EncodeF:

      prefixScanEnc, suffixScanEnc 
      (EncodeF f, Scan (Enc f), Monoid m) f m (m, f m)
      prefixScanEnc = second decode ∘ prefixScan ∘ encode
      suffixScanEnc = second decode ∘ suffixScan ∘ encode

      Lists

      As a first example, consider

      instance EncodeF [] where
      type Enc [] = Const () + Id × []
      encode [] = InL (Const ())
      encode (a : as) = InR (Id a × as)
      decode (InL (Const ())) = []
      decode (InR (Id a × as)) = a : as

      And declare a boilerplate Scan instance via EncodeF:

      instance Scan [] where
      prefixScan = prefixScanEnc
      suffixScan = suffixScanEnc

      I haven’t checked the details, but I think with this instance, suffix scanning has okay performance, while prefix scan does quadratic work. The reason is the in the Scan instance for products, the two components are scanned independently (in parallel), and then the whole second component is adjusted for prefixScan, while the whole first component is adjusted for suffixScan. In the case of lists, the first component is the list head, and second component is the list tail.

      For your reading convenience, here’s that Scan instance again:

      instance (Scan f, Scan g, Functor f, Functor g)  Scan (f × g) where
      prefixScan (fa × ga) = (af ⊕ ag, fa' × ((af ⊕) <$> ga'))
      where (af,fa') = prefixScan fa
      (ag,ga') = prefixScan ga

      suffixScan (fa × ga) = (af ⊕ ag, ((⊕ ag) <$> fa') × ga')
      where (af,fa') = suffixScan fa
      (ag,ga') = suffixScan ga

      The lop-sidedness of the list type thus interferes with parallelization, and makes the parallel scans perform much worse than cumulative sequential scans.

      Let’s next look at a more balanced type.

      Binary Trees

      We’ll get better parallel performance by organizing our data so that we can cheaply partition it into roughly equal pieces. Tree types allows such partitioning.

      Top-down trees

      We’ll try a few variations, starting with a simple binary tree.

      data T1 a = L1 a | B1 (T1 a) (T1 a) deriving Functor

      Encoding and decoding is straightforward:

      instance EncodeF T1 where
      type Enc T1 = Id + T1 × T1
      encode (L1 a) = InL (Id a)
      encode (B1 s t) = InR (s × t)
      decode (InL (Id a)) = L1 a
      decode (InR (s × t)) = B1 s t

      instance Scan T1 where
      prefixScan = prefixScanEnc
      suffixScan = suffixScanEnc

      Note that these definitions could be generated automatically from the data type definition.

      For balanced trees, prefix and suffix scan divide the problem in half at each step, solve each half, and do linear work to patch up one of the two halves. Letting n be the number of elements, and W(n) the work, we have the recurrence W(n)=2W(n/2)+cn for some constant factor c. By the Master theorem, therefore, the work done is Θ(nlogn). (Use case 2, with a=b=2, f(n)=cn, and k=0.)

      Again assuming a balanced tree, the computation dependencies have logarithmic depth, so the ideal parallel running time (assuming sufficient processors) is Θ(logn). Thus we have an algorithm that is depth-efficient (modulo constant factors) but work-inefficient.

      Composition

      A binary tree as defined above is either a leaf or a pair of binary trees. We can make this pair-ness more explicit with a reformulation:

      data T2 a = L2 a | B2 (Pair (T2 a)) deriving Functor

      where Pair, as in Composable parallel scanning, is defined as

      data Pair a = a :# a deriving Functor

      or even

      type Pair = Id × Id

      For encoding and decoding, we could use the same representation as with T1, but let’s instead use a more natural one for the definition of T2:

      instance EncodeF T2 where
      type Enc T2 = Id + PairT2
      encode (L2 a) = InL (Id a)
      encode (B2 st) = InR (O st)
      decode (InL (Id a)) = L2 a
      decode (InR (O st)) = B2 st

      Boilerplate scanning:

      instance Scan T2 where
      prefixScan = prefixScanEnc
      suffixScan = suffixScanEnc

      for which we’ll need an applicative instance:

      instance Applicative T2 where
      pure = L2
      L2 f <*> L2 x = L2 (f x)
      B2 (fs :# gs) <*> B2 (xs :# ys) = B2 ((fs <*> xs) :# (gs <*> ys))
      _ <*> _ = error "T2 (<*>): structure mismatch"

      The O constructor is for functor composition.

      With a small change to the tree type, we can make the composition of Pair and T more explicit:

      data T3 a = L3 a | B3 ((PairT3) a) deriving Functor

      Then the conversion becomes even simpler, since there’s no need to add or remove O wrappers:

      instance EncodeF T3 where
      type Enc T3 = Id + PairT3
      encode (L3 a) = InL (Id a)
      encode (B3 st) = InR st
      decode (InL (Id a)) = L3 a
      decode (InR st) = B3 st

      Bottom-up trees

      In the formulations above, a non-leaf tree consists of a pair of trees. I’ll call these trees "top-down", since visible pair structure begins at the top.

      With a very small change, we can instead use a tree of pairs:

      data T4 a = L4 a | B4 (T4 (Pair a)) deriving Functor

      Again an applicative instance allows a standard Scan instance:

      instance Scan T4 where
      prefixScan = prefixScanEnc
      suffixScan = suffixScanEnc

      instance Applicative T4 where
      pure = L4
      L4 f <*> L4 x = L4 (f x)
      B4 fgs <*> B4 xys = B4 (liftA2 h fgs xys)
      where h (f :# g) (x :# y) = f x :# g y
      _ <*> _ = error "T4 (<*>): structure mismatch"

      or a more explicitly composed form:

      data T5 a = L5 a | B5 ((T5Pair) a) deriving Functor

      I’ll call these new variations "bottom-up" trees, since visible pair structure begins at the bottom. After stripping off the branch constructor, B4, we can get at the pair-valued leaves by means of fmap, fold, or traverse (or variations). For B5, we’d also have to strip off the O wrapper (functor composition).

      Encoding is nearly the same as with top-down trees. For instance,

      instance EncodeF T4 where
      type Enc T4 = Id + T4Pair
      encode (L4 a) = InL (Id a)
      encode (B4 t) = InR (O t)
      decode (InL (Id a)) = L4 a
      decode (InR (O t)) = B4 t

      Scanning pairs

      We’ll need to scan on the Pair functor. If we use the definition of Pair above in terms of Id and (×), then we’ll get scanning for free. For using Pair, I find the explicit data type definition above more convenient. We can then derive a Scan instance by conversion. Start with a standard specification:

      data Pair a = a :# a deriving Functor

      And encode & decode explicitly:

      instance EncodeF Pair where
      type Enc Pair = Id × Id
      encode (a :# b) = Id a × Id b
      decode (Id a × Id b) = a :# b

      Then use our boilerplate Scan instance for EncodeF instances:

      instance Scan Pair where
      prefixScan = prefixScanEnc
      suffixScan = suffixScanEnc

      We’ve seen the Scan instance for (×) above. The instance for Id is very simple:

      newtype Id a = Id a

      instance Scan Id where
      prefixScan (Id m) = (m, Id ∅)
      suffixScan = prefixScan

      Given these definitions, we can calculate a more streamlined Scan instance for Pair:

        prefixScan (a :# b)
      {- specification -}
      prefixScanEnc (a :# b)
      {- prefixScanEnc definition -}
      (second decode ∘ prefixScan ∘ encode) (a :# b)
      {- (∘) -}
      second decode (prefixScan (encode (a :# b)))
      {- encode definition for Pair -}
      second decode (prefixScan (Id a × Id b))
      {- prefixScan definition for f × g -}
      second decode
      (af ⊕ ag, fa' × ((af ⊕) <$> ga'))
      where (af,fa') = prefixScan (Id a)
      (ag,ga') = prefixScan (Id b)
      {- Definition of second on functions -}
      (af ⊕ ag, decode (fa' × ((af ⊕) <$> ga')))
      where (af,fa') = prefixScan (Id a)
      (ag,ga') = prefixScan (Id b)
      {- prefixScan definition for Id -}
      (af ⊕ ag, decode (fa' × ((af ⊕) <$> ga')))
      where (af,fa') = (a, Id ∅)
      (ag,ga') = (b, Id ∅)
      {- substitution -}
      (a ⊕ b, decode (Id ∅ × ((a ⊕) <$> Id ∅)))
      {- fmap/(<$>) for Id -}
      (a ⊕ b, decode (Id ∅ × Id (a ⊕ ∅)))
      {- Monoid law -}
      (a ⊕ b, decode (Id ∅ × Id a))
      {- decode definition on Pair -}
      (a ⊕ b, (∅ :# a))

      Whew! And similarly for suffixScan.

      Now let’s recall the Scan instance for Pair given in Composable parallel scanning:

      instance Scan Pair where
      prefixScan (a :# b) = (a ⊕ b, (∅ :# a))
      suffixScan (a :# b) = (a ⊕ b, (b :# ∅))

      Hurray! The derivation led us to the same definition. A "sufficiently smart" compiler could do this derivation automatically.

      With this warm-up derivation, let’s now turn to trees.

      Scanning trees

      Given the tree encodings above, how does scan work? We’ll have to consult Scan instances for some of the functor combinators. The product instance is repeated above. We’ll also want the instances for sum and composition. Omitting the suffixScan definitions for brevity:

      data (f + g) a = InL (f a) | InR (g a)

      instance (Scan f, Scan g) Scan (f + g) where
      prefixScan (InL fa) = second InL (prefixScan fa)
      prefixScan (InR ga) = second InR (prefixScan ga)

      newtype (g ∘ f) a = O (g (f a))

      instance (Scan g, Scan f, Functor f, Applicative g) Scan (g ∘ f) where
      prefixScan = second (Ofmap adjustL ∘ zip)
      ∘ assocR
      ∘ first prefixScan
      unzip
      fmap prefixScan
      ∘ unO

      This last definition uses a few utility functions:

      zip  Applicative g  (g a, g b)  g (a,b)
      zip = uncurry (liftA2 (,))

      unzip Functor g g (a,b) (g a, g b)
      unzip = fmap fst &&& fmap snd

      assocR ((a,b),c) (a,(b,c))
      assocR ((a,b),c) = (a,(b,c))

      adjustL (Functor f, Monoid m) (m, f m) f m
      adjustL (m, ms) = (m ⊕) <$> ms

      Let’s consider how the Scan (g ∘ f) instance plays out for top-down vs bottom-up trees, given the functor-composition encodings above. The critical definitions:

      type Enc T2 = Id + PairT2

      type Enc T4 = Id + T4Pair

      Focusing on the branch case, we have Pair ∘ T2 vs T4 ∘ Pair, so we’ll use the Scan (g ∘ f) instance either way. Let’s consider the work implied by that instance. There are two calls to prefixScan, plus a linear amount of other work. The meanings of those two calls differ, however:

      • For top-down trees (T2), the recursive tree scans are in fmap prefixScan, mapping over the pair of trees. The first prefixScan is a pair scan and so does constant work. Since there are two recursive calls, each working on a tree of half size (assuming balance), plus linear other work, the total work Θ(nlogn), as explained above.
      • For bottom-up trees (T4), there is only one recursive recursive tree scan, which appears in first prefixScan. The prefixScan in fmap prefixScan is pair scan and so does constant work but is mapped over the half-sized tree (of pairs), and so does linear work altogether. Since there only one recursive tree scan, at half size, plus linear other work, the total work is then proportional to n+n/2+n/4+2n=Θ(n). So we have a work-efficient algorithm!

      Looking deeper

      In addition to the simple analysis above of scanning over top-down and over bottom-up, let’s look in detail at what transpires and how each case can be optimized. This section may well have more detail than you’re interested in. If so, feel free to skip ahead.

      Top-down

      Beginning as with Pair,

        prefixScan t
      {- specification -}
      prefixScanEnc t
      {- prefixScanEnc definition -}
      (second decode ∘ prefixScan ∘ encode) t
      {- (∘) -}
      second decode (prefixScan (encode t))

      Take T2, with T3 being quite similar. Now split into two cases for the two constructors of T2. First leaf:

        prefixScan (L2 m)
      {- as above -}
      second decode (prefixScan (encode (L2 m)))
      {- encode for L2 -}
      second decode (prefixScan (InL (Id m)))
      {- prefixScan for functor sum -}
      second decode (second InL (prefixScan (Id m)))
      {- prefixScan for Id -}
      second decode (second InL (m, Id ∅))
      {- second for functions -}
      second decode (m, InL (Id ∅))
      {- second for functions -}
      (m, decode (InL (Id ∅)))
      {- decode for L2 -}
      (m, L2 ∅)

      Then branch:

        prefixScan (B2 (s :# t))
      {- as above -}
      second decode (prefixScan (encode (B2 (s :# t))))
      {- encode for L2 -}
      second decode (prefixScan (InR (O (s :# t))))
      {- prefixScan for (+) -}
      second decode (second InR (prefixScan (O (s :# t))))
      {- property of second -}
      second (decode ∘ InR) (prefixScan (O (s :# t)))

      Focus on the prefixScan application:

        prefixScan (O (s :# t)) =
      {- prefixScan for (∘) -}
      ( second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan
      unzipfmap prefixScan ∘ unO ) (O (s :# t))
      {- unO/O -}
      ( second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan
      unzipfmap prefixScan ) (s :# t)
      {- fmap on Pair -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan ∘ unzip)
      (prefixScan s :# prefixScan t)
      {- expand prefixScan -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan ∘ unzip)
      ((ms,s') :# (mt,t'))
      where (ms,s') = prefixScan s
      (mt,t') = prefixScan t
      {- unzip -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan)
      ((ms :# mt), (s' :# t')) where
      {- first -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR)
      (prefixScan (ms :# mt), (s' :# t')) where
      {- prefixScan for Pair -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR)
      ((ms ⊕ mt, (∅ :# ms)), (s' :# t')) where
      {- assocR -}
      (second (Ofmap adjustL ∘ zip))
      (ms ⊕ mt, ((∅ :# ms), (s' :# t'))) where
      {- second -}
      ( ms ⊕ mt
      , (Ofmap adjustL ∘ zip) ((∅ :# ms), (s' :# t')) ) where
      {- zip -}
      ( ms ⊕ mt
      , (Ofmap adjustL) ((∅,s') :# (ms,t')) ) where
      {- fmap for Pair -}
      ( ms ⊕ mt
      , O (adjustL (∅,s') :# adjustL (ms,t')) ) where
      {- adjustL -}
      ( ms ⊕ mt
      , O (((∅ ⊕) <$> s') :# ((ms ⊕) <$> t')) ) where
      {- Monoid law (left identity) -}
      ( ms ⊕ mt
      , O ((id <$> s') :# ((ms ⊕) <$> t')) ) where
      {- Functor law (fmap id) -}
      ( ms ⊕ mt
      , O (s' :# ((ms ⊕) <$> t')) )
      where (ms,s') = prefixScan s
      (mt,t') = prefixScan t

      Continuing from above,

        prefixScan (B2 (s :# t))
      {- see above -}
      second (decode ∘ InR) (prefixScan (O (s :# t)))
      {- prefixScan focus from above -}
      second (decode ∘ InR)
      ( ms ⊕ mt
      , O (s' :# ((ms ⊕) <$> t')) )
      where (ms,s') = prefixScan s
      (mt,t') = prefixScan t
      {- definition of second on functions -}
      (ms ⊕ mt, (decode ∘ InR) (O (s' :# ((ms ⊕) <$> t')))) where
      {- (∘) -}
      (ms ⊕ mt, decode (InR (O (s' :# ((ms ⊕) <$> t'))))) where
      {- decode for B2 -}
      (ms ⊕ mt, B2 (s' :# ((ms ⊕) <$> t'))) where

      This final form is as in Deriving parallel tree scans, changed for the new scan interface. The derivation saved some work in wrapping & unwrapping and method invocation, plus one of the two adjustment passes over the sub-trees. As explained above, this algorithm performs Θ(nlogn) work.

      I’ll leave suffixScan for you to do yourself.

      Bottom-up

      What happens if we switch from top-down to bottom-up binary trees? I’ll use T4 (though T5 would work as well):

      data T4 a = L4 a | B4 (T4 (Pair a))

      The leaf case is just as with T2 above, so let’s get right to branches.

        prefixScan (B4 t)
      {- as above -}
      second decode (prefixScan (encode (B4 t)))
      {- encode for L2 -}
      second decode (prefixScan (InR (O t)))
      {- prefixScan for (+) -}
      second decode (second InR (prefixScan (O t)))
      {- property of second -}
      second (decode ∘ InR) (prefixScan (O t))

      As before, now focus on the prefixScan call.

        prefixScan (O t) =
      {- prefixScan for (∘) -}
      ( second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan
      unzipfmap prefixScan ∘ unO ) (O t)
      {- unO/O -}
      ( second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan
      unzipfmap prefixScan ) t
      {- prefixScan on Pair (derived above) -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan ∘ unzip)
      fmap (λ (a :# b) (a ⊕ b, (∅ :# a))) t
      {- unzip/fmap -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR ∘ first prefixScan)
      ( fmap (λ (a :# b) (a ⊕ b)) t
      , fmap (λ (a :# b) (∅ :# a)) t )
      {- first on functions -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR)
      ( prefixScan (fmap (λ (a :# b) (a ⊕ b)) t)
      , fmap (λ (a :# b) (∅ :# a)) t )
      {- expand prefixScan -}
      (second (Ofmap adjustL ∘ zip) ∘ assocR)
      ((mp,p'), fmap (λ (a :# b) (∅ :# a)) t)
      where (mp,p') = prefixScan (fmap (λ (a :# b) (a ⊕ b)) t)
      {- assocR -}
      (second (Ofmap adjustL ∘ zip))
      (mp, (p', fmap (λ (a :# b) (∅ :# a)) t))
      where
      {- second on functions -}
      (mp, (Ofmap adjustL ∘ zip) (p', fmap (λ (a :# b) (∅ :# a)) t))
      where
      {- fmap/zip/fmap -}
      (mp, O (liftA2 tweak p' t))
      where tweak s (a :# _) = adjustL (s, (∅ :# a))
      (mp,p') = prefixScan (fmap (λ (a :# b) (a ⊕ b)) t)
      {- adjustL, then simplify -}
      (mp, O (liftA2 tweak p' t))
      where tweak s (a :# _) = (s :# s ⊕ a)
      (mp,p') = prefixScan (fmap (λ (a :# b) (a ⊕ b)) t)

      Now re-introduce the context of prefixScan (O t):

        prefixScan (B4 t)
      {- see above -}
      second (decode ∘ InR) (prefixScan (O t))
      {- see above -}
      second (decode ∘ InR)
      (mp, O (liftA2 tweak p' t))
      where
      {- decode for T4 -}
      (mp, B4 (liftA2 tweak p' t))
      where p = fmap (λ (e :# o) (e ⊕ o)) t
      (mp,p') = prefixScan p
      tweak s (e :# _) = (s :# s ⊕ e)

      Notice how much this bottom-up tree scan algorithm differs from the top-down algorithm derived above. In particular, there’s only one recursive tree scan (on a half-sized tree) instead of two, plus linear additional work, for a total of Θ(n) work.

      Guy Blelloch’s parallel scan algorithm

      In Programming parallel algorithms, Guy Blelloch gives the following algorithm for parallel prefix scan, expressed in the parallel functional language NESL:

      function scan(a) =
      if #a ≡ 1 then [0]
      else
      let es = even_elts(a);
      os = odd_elts(a);
      ss = scan({e+o: e in es; o in os})
      in interleave(ss,{s+e: s in ss; e in es})

      This algorithm is nearly identical to the T4 scan algorithm above. I was very glad to find this route to Guy’s algorithm, which had been fairly mysterious to me. I mean, I could believe that the algorithm worked, but I had no idea how I might have discovered it myself. With the functor composition approach to scanning, I now see how Guy’s algorithm emerges as well as how it generalizes to other data structures.

      Nested data types and parallelism

      Most of the recursive algebraic data types that appear in Haskell programs are regular, meaning that the recursive instances are instantiated with the same type parameter as the containing type. For instance, a top-down tree of elements of type a is either a leaf or has two trees whose elements have that same type a. In contrast, in a bottom-up tree, the (single) recursively contained tree is over elements of type (a,a). Such non-regular data types are called "nested". The two tree scan algorithms above suggest to me that nested data types are particularly useful for efficient parallel algorithms.

      flattr this!

      by conal at May 24, 2011 08:31 PM

      May 23, 2011

      Paul Johnson

      Windows, Linux, ARM and Intel in a Zero Sum Game

      Microsoft is talking about putting Windows 8 on ARM. Intel is trying to play this down, talking about how Linux is already on ARM, and how Intel even supports Linux itself. None of this makes sense unless you understand the position of Microsoft and Intel in the PC market.

      People often speak of the "Wintel" duopoly, which of course is a misnomer. A duopoly is when two companies share a single market, at which point there is a risk of anti-competitive behaviour. However Microsoft and Intel are not a duopoly, they are two near-monopolies. Microsoft dominates the desktop operating system market, and Intel dominates the desktop CPU market. Although both OS and CPU are necessary components in a desktop computer, this doesn't make their manufacturers a duopoly because even in the best of worlds they don't compete: increased market share for Microsoft is not at the expense of Intel.

      However this doesn't mean that everything can be all cozy between them, because in a broader sense they do compete. Microsoft and Intel are players in the desktop PC "value chain" (actually, its a value network, or even more precisely, a value directed acyclic graph, and if you look closely you find it isn't even really acyclic because chip-makers buy PCs, but term "chain" is more often used so that is what I'll stick to). This describes the way that money flows from PC buyers through vendors to manufacturers to parts makers to raw materials.

      If you are a player in a value chain then its pretty much a zero sum game; every time someone buys a PC their money bubbles back up through the value chain and everyone grabs their bit. Your problem as a player is to get as much of this money as you can, which inevitably means that someone else gets less. Value chains are characterised by dysfunctional relationships between people who need each other but nevertheless hate each others guts.

      There are two big strategic goals in a value chain. The first is obvious, the second less so:
      1. Make your bit a monopoly, so the rest of the value chain has to come to you to produce the product. That way you can charge monopoly prices and hoover up all the money coming through the value chain.
      2. Make everyone else's bits into generic commodities so that they can only compete by keeping prices low. That way they can't charge monopoly prices, which leaves more money for you. That's the consequence of the zero sum game thing.
      Once you understand the second goal you understand everything about Microsoft and Intel.

      • Microsoft makes very sure that Windows runs well on as wide a variety of hardware as possible in order to keep PC hardware a commodity business.
      • Intel makes sure that Linux runs nicely on Intel processors in order to create a competitor for Microsoft, so Microsoft will have to reduce its prices, thereby leaving more money for Intel. I suspect Intel were also very supportive of Apple's move to Intel hardware for this reason, as well as the more obvious one of having a new channel.
      • Microsoft made sure that Windows runs well on AMD. However AMD have never really recovered from Intel's "Intel Inside" marketing coup (where they paid box vendors to make it sound like "Intel Inside" was a big selling point, so lots of customers thought it was). So now Microsoft need to create a new competitor for Intel, and have decided that ARM will fulfil this role nicely. In the past they also did this with the DEC Alpha for the same reason.
      ARM already plays nicely with Linux, which is quite a feat because ARM isn't a single processor; its an entire family. If ARM have any sense they will make sure that this continues. Microsoft are likely to try to pay ARM off to get it to remove support for Linux. If ARM have any sense they will resist this because it will enable Microsoft to entrench itself as a mobile monopoly, thereby decreasing the amount of money available for ARM to grab for itself.

      Edit: "zero sum gain" -> "zero sum game". That's what happens when you let your fingers do the thinking.

      by Paul Johnson (noreply@blogger.com) at May 23, 2011 07:09 PM

      May 18, 2011

      Alberto Ruiz

      Tutorial

      After a long delay, I have finally released a first version of a tutorial for easyVision. I know that this system is only useful to me, but trying to explain something to other people is the best way to find things that can be improved, have a poor design, or more probably, that I don't really understand. For instance, I noticed that the definition of camera combinators was too verbose. We can

      by Alberto (noreply@blogger.com) at May 18, 2011 09:45 PM

      mightybyte

      Looping and Control Flow in Heist

      One of the most frequent questions I get about Heist is how to do loops or control flow in Heist templates. I usually respond that we've tried to avoid introducing these constructs into templates to encourage a better separation between view and model/controller. In this post I want to discuss the issue in more detail so that my thoughts are stored and I can refer people here in the future.


      by mightybyte (noreply@blogger.com) at May 18, 2011 08:42 PM

      TypLAB

      Silk completes €320,000 ($475,000) funding round led by Atomico

      Investment will bring groundbreaking technology and product to market

      Amsterdam, The Netherlands – May 18, 2011 – Silk, the Amsterdam-based web startup, today announced that it has completed a €320,000 ($475,000) funding round led by Atomico, the venture capital firm led by Niklas Zennström, who also co-founded Skype.

      Other investors participating in the round are Floris Rost van Tonningen, co-founder of Hyves, the largest social network in the Netherlands, Han de Groot, founder of MetrixLab, Hans-Poul Veldhuyzen van Zanten and Mark de Lange, who is Founding Partner of Global Grid Capital.

      Silk is a web-based tool that allows content creators to provide their content in a more structured manner on the web. The tool enables users to choose the data they want from the mass of information available and to view and arrange it in ways that make sense to them.

      It is at the forefront of the “Web 3.0″ trend that helps users interact with the huge and growing volume of information online. A video explanation of the product is available at http://www.silkapp.com/.

      Silk last month won the “Best overall startup” award at The Next Web Conference, a leading European technology conference, in Amsterdam. It is in private beta at the moment, but people can request invites to get early access to the product at the company’s website.

      The new funds will be used to bring the product and technology to market and scale up development efforts.

      Salar al Khafaji, CEO and co-founder of Silk, said: “Silk allows content creators to present information to audiences in a more engaging way, and allows users to sort through information and visualize results in compelling ways.”

      “For example, publishers can use it to enhance the value of their content, and the service they offer to audiences, and we are already working with several publishers to create Silk powered web sites. We also have plans for other uses of the product.”

      Geoffrey Prentice, Partner at Atomico, said: “We are delighted to have led this investment round. It highlights the huge potential that Silk has, and our enthusiasm for working with talented co-investors. The caliber of people participating in this deal reflects the company’s outstanding products and management team.”

      Floris Rost van Tonningen, co-founder of Hyves, the largest social network in the Netherlands, said: “I’m very impressed with the vision of the Silk team who have developed a powerful technology that is bound to revolutionize the space of content creation and search.”

      Mark de Lange, an investor who is Founding Partner at Global Grid Capital, said: “As the amount of online content and data grows exponentially, and will continue to do so, publishers and consumers alike will need tools that enable them to derive real value from the available data. I believe Silk is uniquely positioned to be one of these tools.”

      Silk’s technology has been in development since 2009 and the company started rolling out its beta platform this year in co-operation with The Next Web, one of the largest tech blogs in the world, and NRC, a leading Dutch newspaper, who were early users.

      To stay up-to-date on Silk, please visit www.silkapp.com and follow it on Twitter at http://twitter.com/silkapp.

      Notes to editors

      For more information, see our about page, or get in touch with us.

      by Salar at May 18, 2011 07:00 AM

      May 17, 2011

      Alberto Ruiz

      metric rectification from circles

      The circular points encode the metric structure of the 2D plane. They are invariant to rotation, isotropic scaling and translation. If the plane undergoes a more general (e.g. affine or projective) transformation  they move to some unknown location. Metric rectification can be performed if we find the image of the circular points and move them back to their original location. These points are

      by Alberto (noreply@blogger.com) at May 17, 2011 05:43 PM

      May 08, 2011

      Martijn van Steenbergen

      Introducing JsonGrammar

      The first version of JsonGrammar has just been released on Hackage! JsonGrammar offers an API for converting between your own datatypes and JSON ASTs.

      “What, another JSON library? Don’t we have enough already?”

      It’s true that there are already a few JSON libraries out there. These libraries, however, require you to write fromJson and toJson separately.

      “Uhm, yes… is that bad?”

      Yes. It violates the DRY principle. If I show you an implementation of fromJson for a certain type, you can write a corresponding toJson without requiring any further information. Similarly, if I show you an implementation of toJson, you can write the accompanying fromJson. Writing down the same thing twice is tedious and opens up the possibility to make mistakes.

      “But most of these libraries offer Template Haskell support that does this work for you!”

      This is true, but they also make all the choices for you about how your datatypes should map to JSON. Usually they assume the names of your record fields map directly to JSON property names. The shapes of your family of datatypes need to correspond to how the objects in JSON are nested. These libraries give you the choice: either you write out fromJson and toJson by hand and have full control over the mapping, or you give up this control and let Template Haskell do all the work for you.

      JsonGrammar gives you the best of both worlds: it gives you full control over what the mapping should be, with an API that lets you define fromJson and toJson at the same time. It achieves this by separating the constructing/destructing of datatype constructors and its fields from the description of the JSON values. The former is derived by Template Haskell, the latter is provided by the programmer.

      An example

      Suppose we have these two datatypes describing people and their current location:

      data Person = Person
        { name   :: String
        , gender :: Gender
        , age    :: Int
        , lat    :: Float
        , lng    :: Float
        }
      
      data Gender = Male | Female
      

      Sadly, the JSON source we are communicating with is using JSON with Dutch property names and values, so we cannot use Template Haskell to derive the JSON mapping for us, like we would do with other JSON libraries. Neither do we want to use Dutch names for our record selectors; nobody would be able to understand our code anymore! Fortunately this isn’t a problem with JsonGrammar.

      The first step is to have Template Haskell derive the constructor-destructor pairs:

      person         = $(deriveIsos ''Person)
      (male, female) = $(deriveIsos ''Gender)
      

      For the latter to work, you need to enable -XNoMonoPatBinds.

      Then we write instances of the Json type class to define the mapping from/to Json. The order in which the properties are listed matches that of the fields in the datatype:

      instance Json Person where
        grammar = person . object
          ( prop "naam"
          . prop "geslacht"
          . prop "leeftijd"
          . prop "lat"
          . prop "lng"
          )
      
      instance Json Gender where
        grammar =  male   . litJson "man"
                <> female . litJson "vrouw"
      

      The . operator is from Control.Category. The <> is just another name for mappend from Data.Monoid and denotes choice.

      That’s all! We have just defined both fromJson and toJson in one simple definition. Here’s how you can use these grammars:

      ghci> let anna = Person "Anna" Female 36 53.0163038 5.1993053
      
      ghci> let Just annaJson = toJson anna
      
      ghci> annaJson
      Object (fromList [("geslacht",String "vrouw"),("lat",Number
      53.01630401611328),("leeftijd",Number 36),("lng",Number
      5.199305534362793),("naam",String "Anna")])
      
      ghci> fromJson annaJson :: Maybe Person
      Just (Person {name = "Anna", gender = Female, age = 36, lat = 53.016304,
      lng = 5.1993055})
      

      Show me the types!

      The library is based on partial isomorphisms:

      data Iso a b = Iso (a -> Maybe b) (b -> Maybe a)
      
      instance Category Iso
      instance Monoid (Iso a b)
      

      A value of type Iso a b gives you a function that converts an a into a Maybe b, and a function that converts a b into a Maybe a. This composes beautifully as a Category. The Monoid instance denotes choice: first try the left-hand conversion function, and if it fails, try the right-hand side.

      A JSON grammar for some type a is nothing more than a value of type Iso Value a, where Value is the type of a JSON AST from the aeson package. That is, it’s a pair of conversion functions between JSON trees and your own datatype. Building JSON grammars like the one above is about composing isomorphisms that translate between intermediate types.

      The isomorphisms person, male and female translate between constructors and their individual fields. For example:

      person :: Iso (String, Gender, Int, Float, Float) Person

      Converting from a constructor to its fields might fail, because the value that is passed to the conversion function might be a different constructor of the same datatype. This is why the Monoid instance is so useful: we can give multiple grammars, usually one for each constructor, and they will be tried in sequence. They are effectively composable pattern matches.

      Stack isomorphisms

      There is a problem with encoding the fields of such a constructor as an n-tuple: if we want to compose it with other isomorphisms that handle the individual fields, we have to use complicated tuple projections to select the fields that we’re interested in. Basically we have unwrapped the fields from one constructor only to wrap them in another one!

      The solution is to use heterogenous stacks of values. They are reminiscent of continuation-passing style, because in the way we use them they usually have a polymorphic tail:

      person :: Iso (String :- Gender :- Int :- Float :- Float :- t) (Person :- t)

      Read :- as ‘cons’, but then for types instead of values. Its definition is simple:

      data h :- t = h :- t

      The polymorphic tail says that person doesn’t care what’s on the stack below the two Floats; it will simply pass that part of the stack on to the right-hand side. And vice versa, if we’re working with the isomorphism in the opposite direction.

      Have you thought about what the types of male and female would be in the non-stack versions of the isomorphisms? They don’t have any fields; we would have to leave the first type parameter of Iso empty somehow, for example by choosing (). Stack isomorphisms have no such problem; we simply make the first type argument the polymorphic tail on its own, without any values on top:

      male   :: Iso t (Gender :- t)
      female :: Iso t (Gender :- t)
      

      Stack isomorphisms compose beautifully using ., often without needing any special projection functions. To get a feeling for it, try compiling the example Json grammars and looking at the types of the individual components.

      I lied when I wrote that grammars have type Iso Value a; they actually use stacks themselves, too. Here is the true definition of the Json type class:

      class Json a where
        grammar :: Iso (Value :- t) (a :- t)
      

      Different tree shapes

      Let’s take our Person example and make a small modification. We decide that because (lat, lng)-pairs are so common together, we’d like to put them together in their own datatype:

      data Coords = Coords { lat :: Float, lng :: Float }
        deriving (Eq, Show)
      
      data Person = Person
        { name     :: String
        , gender   :: Gender
        , age      :: Int
        , location :: Coords
        } deriving (Eq, Show)
      

      However, in this example we have no control over the JSON format and cannot change it to match our new structure. With JsonGrammar we can express mappings where the nesting is not one-to-one:

      instance Json Person where
        grammar = person . object
          ( prop "naam"
          . prop "geslacht"
          . prop "leeftijd"
          . coordsProps
          )
      
      coordsProps :: Iso (Object :- t) (Object :- Coords :- t)
      coordsProps = duck coords . prop "lat" . prop "lng"
      

      Here duck coords wraps (or unwraps, depending on the direction) the two matched Float properties in their own Coords constructor before continuing matching the other properties in an object. Function duck is a combinator that makes a grammar (coords in this case) work one element down the stack. Here it makes sure the top values can remain Objects, which is needed by prop to build/destruct JSON objects one property at a time.

      What is important to note here is that not only can we express mappings with different nestings, we can also capture this behaviour in its own grammar for reuse. JsonGrammar allows this level of modularity in everything it does.

      History and related work

      The ideas behind JsonGrammar go back a bit. They are based on Zwaluw, a library that Sjoerd Visscher and I worked on. The library aids in writing bidirectional parsers/pretty-printers for type-safe URLs, also in a DRY manner. Zwaluw, too, uses stacks to achieve a high level of modularity. In turn, Zwaluw was inspired by HoleyMonoid, which shows that the CPS-like manner of using polymorphic stack tails allows combinators to build up a list of expected arguments for use in printf-like functionality.

      The Iso datatype comes from partial-isomorphisms and is described in more detail in Invertible syntax descriptions: Unifying parsing and pretty printing by Tillmann Rendel and Klaus Ostermann. They also use stacks (in the form of nested binary tuples), but they are not using the trick with the polymorphic tail (yet?).

      Future work

      Although JsonGrammar is usable, there is still work to be done:

      • Supporting new use cases. JsonGrammar has not been used in the wild much yet. If you find any use cases that the library currently does not support, please let me know!
      • Benchmarking. No performance testing or memory usage profiling has been done yet.
      • Improved error messages. The Maybe return values indicate whenever conversion has failed, but never how it has failed. The aeson package gives nice error message when for example an expected property was not found. Such error reporting still has to be added to JsonGrammar.
      • Other experiments. Perhaps a library can be written on top of JsonGrammar that allows grammars to be specified that also compile to JSON Schema.

      If you have any questions, comments, ideas or bug reports, feel to leave a comment or open a ticket on GitHub.

      by Martijn at May 08, 2011 09:11 PM

      May 04, 2011

      Bas van Gijzel

      A small handout on TTTAS (Typed Transformations of Typed Abstract Syntax)

      This period I did the course Advanced Functional Programming and I had to choose a subject from a set list of subjects to present. I chose typed transformations of typed abstract syntax (TTTAS). (See this webpage for the technical report and associated library by Arthur Baars, Doaitse Swierstra and Marcos Viera.)

      I already expected the subject to be one of the harder ones and I also had a restriction for the presentation due to practical issues, namely we could only use handouts (or your own laptop) and not a beamer because of multiple presenters.

      Thus I made some handouts in a tutorial like fashion. It might save you some typing of the code found in the paper and hopefully get you interested in the subject. You might also find it useful to get some motivation for the use of GADTs and transformations of EDSLs in general.

      The handouts:
      The PDF file is here.
      The lhs can be found here.

      by Nebasuke (noreply@blogger.com) at May 04, 2011 03:26 PM

      Slides about Martin-Löf type theory

      It's been a while since my last update. Anyway, last year, I participated in the seminar Dependently typed programming given by Andres Löh and Doaitse Swierstra. I gave a presentation, or more of a lecture really, about Martin-Löf type theory, the Curry-Howard correspondence and some of its connections with Agda.

      Since I thought it would still be useful to share, here are the slides:
      Type Systems<iframe class="scribd_iframe_embed" data-aspect-ratio="1.33115468409586" data-auto-height="true" frameborder="0" height="600" id="doc_58782" scrolling="no" src="http://www.scribd.com/embeds/54313539/content?start_page=1&amp;view_mode=slideshow&amp;access_key=key-28q3ghwna8b0fbsszx24" width="100%"></iframe><script type="text/javascript">(function() { var scribd = document.createElement("script"); scribd.type = "text/javascript"; scribd.async = true; scribd.src = "http://www.scribd.com/javascripts/embed_code/inject.js"; var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(scribd, s); })();</script>

      Download the slides here.

      by Nebasuke (noreply@blogger.com) at May 04, 2011 03:17 PM

      May 02, 2011

      Eric Kow (kowey)

      untangling a cabal install problem

      I sometimes have trouble translating abstract general explanations to my particular concrete cases.  I hope that by sharing a very concrete situation I experienced, other users may recognise themselves and get unstuck on their own problems.

      I finally untangled out a cabal install problem that's been bugging me for some time, almost driving me to use cabal-dev on all my packages (which seems like it might be a bit inconvenient)

      So I have a fairly standard setup (at least, it was standard when I wrote this post), GHC 6.12.3 with the latest released Haskell Platform.  I'm working on two packages, GenI and nltg-hillwalking simultaneously.  Switching from one to the other is painful.  When I try to install GenI typing "cabal install" results in this horribly disheartening sequence, where it installs random, haskell98, cpphs, haskell-src-exts, derive and finally GenI.  If I then switch back to working on hillwalking, I then get this another discouraging sequence involving random (again!), QuickCheck, test-framework, ntlg-hillwalking.  And going back to working on GenI, I go through the same pain again.

      It took me a while to work out that the problem was just the interaction between these two packages.  Having had a chance to chat about this with Duncan and Ian, I got a bit of a clue about what the problem might be.  Indeed, when I ran  "cabal install --dry-run -v2", this little bit of output caught my eye:

      In order, the following would be installed:
      random-1.0.0.3 (reinstall) changes: time-1.1.4 -> 1.1.2.4
      haskell98-1.0.1.1 (reinstall)
      cpphs-1.11 (new package)
      haskell-src-exts-1.10.2 (new package)
      derive-2.4.2 (new package)
      GenI-0.21 (new package)

      See that little arrow?  It says that the reason random, the cause of all my heartache, is being reinstalled because of it wants to depend on an older version of time.  Why on earth would it want to do that?  ... Oh, because I told it to.  Apparently, some past version of myself decided to put this dependency in GenI.cabal: time == 1.1.2.4

      Oops!

      I think the problem looks like this.  GenI uses the derive package, which triggers a chain of dependencies all the way down to random and time.  Unfortunately, GenI also directly depends on time but now we have an issue.  I'm not entirely clear on why this causes a recompile as opposed to the more usual "this will likely cause an error" output (maybe the latter is only appropriate for direct dependencies, ie. if derive depended on time itself?).


      By forcing GenI to use this old version of time, I was indirectly forcing it to install a version of  the random package that depends on this old version.  In doing so, I would clobber the version of the random package that QuickCheck uses.

      Fixing the issue in GenI was relatively straightforward.  Did I really need to be using such a constrained version of time?  It turns out that time == 1.1.* works perfectly fine (taking advantage of the PVP promise of backwards compatibility in all A.B.* versions of a package).  Just one little dependency and everything works a lot more smoothly.

      So what did I learn from this?
      1. take a deep breath - I think when I'm faced with these issues, I'm feeling really impatient to get on with my work.  But solving the issue involves recognising just some silly little problem, which can be hard to do when I'm being impatient.  So part of the trick is to defocus somehow and shift to poking mode.
      2. use cabal install --dry-run -v2 and study the end part : what packages are we trying to install and why?  The -v2 is important because it tells you why packages are being installed.
      3. ???  hunt for the offending dependency - for me this was a simple case of staring at GenI.cabal.  What if GenI depended on some library which in turn depended on time-1.1.2.4?  I guess the answer would lie in the list of packages that cabal-install says it would install.  The dependency must lie *somewhere* in the chain.
      If I understand correctly, this may actually an improvement over the pre GHC 6.12 days before the ABI hash was introduced.  I don't actually know, but I could imagine there's something that'd make one random not-quite-compatible with the other, even if they're both version 1.0.0.3 and silently swapping one out for the other would cause subtle breakage.  At least now, we know if something is wrong and we can fix it relatively easily by just reinstalling the missing package.

      This dependency stuff must be really tricky!  It looks like there may be some work that could make life better, for example, a Nix-like approach where both versions of random 1.0.0.3 could co-exist.  But we should be glad in the meantime that Duncan et al have not torn their hair out yet.  (Just think of the pre-Cabal-install days if it helps, life's much better now, isn't it?)

      by kowey (noreply@blogger.com) at May 02, 2011 10:44 PM

      Lennart Augustsson

      More points for lazy evaluation

      In a recent blog post Bob Harper shows one use of laziness, but I think he misses the real import points of laziness. So I will briefly enumerate what I think are the important points of lazyness (excuse me for not coming up with any kind of pun about points).

      First, I'd like to say that I don't think strict or lazy matters that much in practice; they both work fine. I have programmed in regular non-strict Haskell as well as strict Haskell, and most things carry over well to a strict Haskell.  Furthermore, when I say strict language I mean a strict language with at least non-termination as an effect; total languages is another matter.

      Lazy bindings

      I like to tell Haskell beginners that any subexpression can be named and "pulled out", modulo name capture, of course. For instance
          ... e ...
      
      is the same as
          let x = e
          in  ... x ...
      

      The key thing is that
          ... e ... e ...
      
      is the same as
          let x = e
          in  ... x ... x ...
      
      so that common subexpressions can be given a name.

      This is (in general) just wrong in a strict language, of course. Just take the simple example
          if c then error "BOO!" else 0
      
      Which is not the same as
          let x = error "BOO!"
          in  if c then x else 0
      

      In this case you can easily fix the problem by delaying the computation with a lambda (a common theme).
          let x () = error "BOO!"
          in  if c then x () else 0
      

      But for a slightly more complicated example this simple technique goes wrong. Consider
          map (\ a -> a + expensive) xs
      
      where expensive does not depend on a. In this case you want to move the expensive computation out of the loop (cf. loop invariant code motion in imperative languages). Like so
          let x = expensive
          in  map (\ a -> a + x) xs
      
      In a lazy language x will be evaluated exactly zero times or once, just as we want. Using the delaying trick doesn't work here:
          let x () = expensive
          in  map (\ a -> a + x ()) xs
      
      since expensive will get evaluated once for every list element.

      This is easy enough to remedy, by introducing an abstraction for lazy computations (which will contain an assignment to compute the value just once). The signature for the abstract type of lazy values is something like
          data Lazy a
          delay :: (() -> a) -> Lazy a
          force :: Lazy a -> a
      
      Note that the delay needs to take a function to avoid the a being evaluated early.
      (This is probably what Bob would name a benign effect and is easily programmed using unsafePerformIO, which means it needs careful consideration.)

      And so we get
          let x = delay (\ () -> expensive)
          in  map (\ a -> a + force x) xs
      
      This isn't exactly pretty, but it works fine. In a language with macros the ugliness can be hidden better.

      Lazy functions

      Even strict languages like ML and C have some lazy functions even if they don't call them that, like SML's if, andthen, and orelse. You really need the if construct to evaluate the condition and then one of the branches depending on the condition. But what if I want to define my own type with the same kind of functions? In ML I can't, in C I would have to use a macro.

      The ability to define new functions that can be used as control constructs is especially important when you want to design embedded domain specific languages. Take the simple example of the when (i.e., one-arm if) function in Haskell.
          when :: (Monad m) => Bool -> m () -> m ()
      
      A quite common use of this function in monadic code is to check for argument preconditions in a function, like
          f x = do
              when (x < 0) $
                  error "x must be >= 0"
              ...
      
      If the when function is strict this is really bad, of course, since the call to error will happen before the when is called.

      Again, one can work around this by using lazy values, like
          myAnd :: MyBool -> Lazy MyBool -> MyBool
          ...
          ... myAnd x (delay (\ () -> y)) ...
      
      But in my opinion, this is too ugly to even consider. The intent of the function is obscured by the extra noise to make the second argument lazy.

      I think every language needs a mechanism for defining some form of call-by-name functions. And many languages have this in the form of macros (maybe built in like in Lisp, or as a preprocessor like C).
      If a language cannot define lazy function it simply lacks in abstraction power. I think any kind of fragment of the language should be nameable and reusable. (Haskell lacks this ability for patterns and contexts; a wart in the design.) So if you notice a repeated expression pattern, like
          if c then t else False
      
      and cannot give this a name, like
          and c t = if c then t else False
      
      and then use it with the same effect as the orginal expression, well, then your language is lacking.

      For some language constructs the solution adopted by Smalltalk (and later Ruby), i.e., a very lightweight way on constructing closures is acceptable. So, for instance, I could accept writing
          ... myAnd x {y} ...

      (In SML you could make something using functors, but it's just too ugly to contemplate.)

      Lazy constructors

      Lazy constructors were sort of covered in what Bob claimed to be the point of laziness, so I'll just mention them for completeness.  Sometimes you need them, but in my experience it's not very often.

      Cyclic data structures

      This is related to the last point.

      Sometimes you really want cyclic data structures. An example are the Haskell data types in Data.Data that describe data types and constructors. A data type descriptor needs to contain a list of its constructors and a constructor descriptor needs to contain the data type descriptor.
      In Haskell this can be described very naturally by having the two descriptors reference each other.
      In SML this is not possible. You will have to break the cycle by somthing like a reference (or a function).
      In OCaml you can define cyclic data structures in a similar fashion to Haskell, so this isn't really a problem with strict languages, but rather a feature that you can have if you like. Of course, being able to define cyclic data leads to non-standard elements in your data types, like
          data Nat = Zero | Succ Zero
          omega :: Nat
          omega = Succ omega
      
      So having the ability to define cyclic data structures is a double edged sword.
      I find the lack of a simple way to define cyclic data a minor nuisance only.

      Reuse

      I've saved my biggest gripe of strict evaluation for last. Strict evaluation is fundamentally flawed for function reuse.
      What do I mean? I will illustrate with and example.
      Consider the any function is Haskell:
      any :: (a -> Bool) -> [a] -> Bool
      any p = or . map p
      
      It's quite natural to express the any function by reusing the
      map and or functions.  Unfortunately, it doesn't
      behave like we would wish in a strict language.  The any function should scan the list from the head forwards and as soon as an
      element that fulfills the predicate is found it should return true and stop
      scanning the list.  In a strict language this would not happen, since
      the predicate will be applied to every element before the or
      examines the elements.
      So we are forced to manually fuse the two functions, doing so we get:
      any :: (a -> Bool) -> [a] -> Bool
      any p = foldr False (\ x r -> p x || r)
      
      
      or :: [Bool] -> Bool
      or = foldr False (||)
      
      
      
      
      foldr :: (a -> b -> b) -> b -> [a] -> b
      foldr f z [] = z
      foldr f z (x:xs) = f x (foldr f z xs)
      
      
      
      But the misery doesn't end here. This still doesn't do the right thing, because the strict language will recurse all the way down the list since it will call foldr before f. So we either have to fuse again, or invent a new version of foldr that delays the recursive call.
      One more fusion gets us to
      any p []     = False
      any p (y:ys) = y || any p ys
      
      
      So where's the function reuse?  Nowhere in sight.
      

      With strict evaluation you can no longer with a straight face tell people: don't use recursion, reuse the recursion patterns in map, filter, foldr, etc. It simply doesn't work (in general).

      Using macros doesn't really save us this time, because of the recursive definitions. I don't really know of any way to fix this problem short of making all (most?) functions lazy, because the problem is pervasive.  I.e., in the example it would not be enough to fix foldr; all the functions involved need to be lazy to get the desired semantics.

      I find this the biggest annoyance with strict evaluation, but at the same time it's just an annoyance, because you can always rewrite the code to make it work. But strict evaluation really, fundamentally stops you from reusing functions the way you can with laziness.

      As an aside, the definition of any doesn't work in SML for another reason as well, namely the value restriction. But that's just a language wart, like the monomorphism restriction in Haskell.

      Complexity

      Another complaint Bob Harper had about lazy evaluation is the difficulty of finding out the complexity of functions. I totally agree that the space complexity is very messy in a lazy language. For (sequential) time complexity I don't see any need to worry.
      If strict a function has O(f(n)) complexity in a strict language then it has complexity O(f(n)) in a lazy language as well.  Why worry? :)

      Summing up

      One of the most important principles in all software design is DRY, i.e., Don't Repeat Yourself. This means that common patterns that we can find in a program should be abstractable so you can reuse them. In a lazy language you can use functions for abstracting control structures, which a strict language does not permit (at least not in a convenient way). This can be mitigated by providing some other abstraction mechanism, like macros (hopefully some kind of sane macros).
      For the reasons above, I find it more convenient to program in a lazy language than a strict one. But a strict language with the ability to define lazy bindings, lazy functions, and lazy constructors is good enough that I find it usable.

      by augustss (noreply@blogger.com) at May 02, 2011 06:28 PM

      April 20, 2011

      Douglas M. Auclair (geophf)

      Attributed Variables: their uses and one implementation

      Here's a problem that comes up enough, computationally: find a value within some range. You don't know what the value is at the moment, but you will know it, eventually, as the range becomes further restricted by refinements of the constraints or data.

      Now, in general, this is an easy problem for Haskell to solve, and Haskell provides more than one way to approach the solution. For example, if we have some x in some domain D, one way to look at an arbitrary x until we find the one, true x we are looking for is the following code snippet:

      > [ x | x <- D, ... further constraints ]

      More variables? Interrelated ones? No problem:

      > [ (x, y, z) | x <- D, y <- D, z <- D, x < y, ... further constraints ]

      Okay, let’s take it to the next level. What if the interrelation is such that one variable consumes from the domain, so that value is now unavailable to other variables. Pithily: the variables’ values are mutually exclusive.

      Well, one way to go about that is to embed a guard for each variable into the list compression:

      > [ (x, y, z) | x <- D, y <- D, y /= x, z <- D, z `notElem` [x,y], ...

      The drawback here is that this kind of guarded coding becomes tedious and prone to errors. On top of that we see that the guard follows the select, causing unnecessary backtracking through the search space,[1] especially given that we know the guarded values a priori.

      Well, then, we can automate guarded selection with a state (transformer) monad:[2]

      > ...
      > f = do x <- choose
      > y <- choose
      > z <- choose
      > ...
      > runStateT f [0..9]


      Okay, excellent! All of this works and works well … for a fixed and predetermined variable set.

      The Problem



      Things start to get tricky for me, programmatically, however, when one set of constrained values spill over onto another set of constrained values, such as the results of one generation in a cellular automata set or genetic (algorithm) pool affect the next set:

      > evalStateT (gen row1guards) domain >>= \ans1 ->
      > evalStateT (gen (row2guards ans1)) domain >>= \ans2 ->
      > evalStateT (gen (row3guards ans1 ans2)) domain >>= \ans3 ->
      > ... etc


      The state transformer isn’t enough, so it needs to be some RWS monad, perhaps pushing the interim results into the Reader monad and the accumulated ones into Writer while the State monad handles the flow of the computation?

      It seems to me that the work becomes more about monad management and less about the computation itself.[3]

      Then there’s the even more general computation. What if we do not know, a priori, how many values we are generating in our problem solving? The problem space I’m looking at is something like:
      For a variable number of variables, map selection and guards for each variable.

      How do I write that as a list compression expression, or as a monadic one? The problem with Haskell variables is that they aren't (variables), and so something so easily accomplished in a Prolog program:

      kakuro([], Vars, Functor) :-
      Fn =.. [Functor, Vars],
      call(Fn).
      kakuro([H|T], Acc, Functor) :-
      assign(H),
      kakuro(T, [H|Acc], Functor).


      called with:

      ?- kakuro([A, B, C, D], [], sum_equals_10).

      or:

      ?- kakuro([X, Y, Z], [], sum_equals_10).

      ... or with a list of any other length isn't so easily written out in Haskell when I go to attack the problem. After all, what is the type of a list of unbound variables? Haskell doesn't (so easily) allow this, and I was stuck: I didn't know how to proceed using monadic programming or list compression.

      A solution



      Fortunately for me, I've been studying constraint-based programming, and that research has let me to some papers on attributed variables[4] which faciliate that programming style.

      Attributed variables are variables ranging over a domain that can have attributes attached to them that restrict (or constrict) the range of values. It's a very simple step to go from a set of unbound attributed variables to a (simple or façile) constraint solver by adding (constraining) attributes to the variables.

      More formally, I declare (denotationally) an attributed variable as:

      > data Attribute a = Attribute [Constraint a] (Domain a)

      where Constraint and Domain are declared as:

      > newtype Constraint a = Constraint { resolve :: a -> Bool }

      > type Domain a = [a]


      Or, to put words to it, an Attributed Variable is an object that has a set of (constraining) functions over an assigned domain.[5]

      The above denotational declaration of attributed variables works, but I find, operationally, grounding an attributed variable once it's reduced to a single value to be useful:

      > data Attribute a = Attribute [Constraint a] (Domain a)
      > | Ground a


      So, that's my declaration.[8] How is this implemented in practice?

      Implementation



      The approach I take is that an attributed variable is created in a free state over its domain:

      > makeAttrib :: Domain a -> Attribute a
      > makeAttrib list = Attribute [] list


      And then, as constraints are added to the attributed variable, the system 'tries to solve' the attributed variable over the set of the constraints:

      > constrain (Attribute constraints dataset) constraint
      > = solve (Attribute (Constraint constraint:constraints) dataset)
      > constrain (Ground x) _ = Ground x


      And if the solution results in a singleton list, then the attributed variable becomes Ground:

      > solve attr@(Attribute _ []) = attr
      > solve attr@(Attribute constraints list@(h:t))
      > = let result = solve' constraints list
      > in if length result == 1
      > then Ground (head result)
      > else Attribute constraints result


      Of course, the 'solution' to a Ground attributed variable is itself:

      > solve (Ground x) = Ground x

      The above code is simply a dispatch along the binary type. The worker-bee, solve' is simply (again) an implementation of the dual of filter:

      > solve' :: [Constraint a] -> [a] -> [a]
      > solve' _ [] = []
      > solve' constraints (h:t) =
      > if (solveConstraints constraints h)
      > then (h:solve' constraints t)
      > else solve' constraints t

      > solveConstraints :: [Constraint a] -> a -> Bool
      > solveConstraints [] _ = True
      > solveConstraints (f:rest) x = resolve f x && solveConstraints rest x


      So something from Control.Applicative to reduce that to a one-liner? I leave that as an exercise to the reader.

      And there you have it, that is a working implementation of attributed variables. I have test functions for groundedness, solvability and the current state of the domain of an active attributed variable, but those are simple helper functions, easily implemented, and provided in the attached source code.

      Uses



      So, now, what do we have?

      Now we have variables over which we may specify a domain ('loosely' typing a variable) and then constrain. How can we use such a system?

      One way is that we can write constraint logic programs in Haskell. The illustration from Prolog (above) has a nearly direct transliteration into Haskell (with the additional benefit of strong typing):

      > kakuro :: [Int] -> Int -> [[Int]]
      > kakuro list total = nub $ summer (sort $ map trib list) [] total

      > trib :: Int -> Attribute Int
      > trib x | x == 0 = makeAttrib [1..9]
      > | otherwise = Ground x

      > summer :: [Attribute Int] -> [Int] -> Int -> [[Int]]
      > summer [] list total = do
      > guard $ sum list == total
      > return $ sort list
      > summer (h:t) list total = do
      > a <- domain $ constrain h (flip notElem list)
      > summer t (a:list) total


      And with the above code, we can now do the following:

      *Kakuro> kakuro [0,0,0] 10
      [[1,2,7],[1,3,6],[1,4,5],[2,3,5]]
      *Kakuro> kakuro [0,0,0,0] 10
      [[1,2,3,4]]


      Note that I use sort above, but attributed variables are not (denotationally) ordered types. Operationally, that is: in logical programming, it makes sense to eliminate early, so if we have Ground values, we wish to consume those first. So, to that end, I declare the following instances:

      > instance (Eq a) => Eq (Attribute a) where
      > Ground x == Ground y = x == y
      > _ == _ = False

      > instance (Ord a) => Ord (Attribute a) where
      > Ground x <= Ground y = x <= y
      > Ground _ <= _ = True
      > _ <= _ = False


      And with those instance declarations, sort automagically works. Neat!

      Interlude: Haskell is "Neat"



      I don't know about you, but I'm guessing that many of you are taking the gift that Haskell is for granted. Or I suppose I should say 'the way of thinking that using Haskell provides.' Over and over again in industry (yes: stone knives and bearskins) I hear 'that can't be done.' And I find myself puzzled. What's the showstopper here?

      For example, this past week at work, we needed to convert a data mapping document in Excel (stop snickering, please) into an equivalently flat XML schema representation with a accompanying XML sample file. So fields of the form:

      <root>/Field/Name="BusinessAddress"
      <root>/Field/Value="plaintext"

      <root>/Field/Name="BusinessAddress"
      <root>/Field/Value="encrypted"


      And there were enumerated values as well:

      <root>/Field/Name="PropertyOccupancyTypeOwnerOccupied"
      <root>/Field/Value="plaintext"

      <root>/Field/Name="PropertyOccupancyTypeRentedToTenant"
      <root>/Field/Value="plaintext"

      <root>/Field/Name="PropertyOccupancyTypeVacant"
      <root>/Field/Value="plaintext"


      There were over 1,000 of these element names. I mentioned I would write a parser to generate the XSD and XML and was told by the development team point-blank that that was impossible and the only way to go about this transcription was manually, line-by-line, by hand.

      So I wrote the script. I suppose I could have used Perl, but as I'm on a Windows box, the shell isn't very inviting for Perl-like piping, so I used Haskell, stripping the metainformation and pushing the parsed names into a map (or, to be precise, a Data.Map) where enumerated (mutally exclusive instance) types went into a list named by their type name and other values were singletons. Then for the XSD I simply iterated over the elements of the map (recursively), and for the sample XML file, I just chose an element (in this case, the first element) of each type.

      The net result was that all the elements analysis team compiled into an Excel spreadsheet by hand over a six week period I was able to deliver in two days as a full XSD schema and a comprehensive XML sample file.

      Two members from the development team assisted me on this task, one of them was the lead developer. He was intrigued and asked me: "What does this script do?" after being impressed that he could generate a schema definition file and a sample file in seconds. So I explained Haskell as a 'data-flow' language and I traced the flow of the input file through the monad and function composition showing the translation of the internal representation into XSD and then, in a different context, into XML. I'm not sure he got my explanation, but he did walk away impressed with what Haskell can do.

      Yes, these are simple things: parsing, mapping, iterating. But for most of the software engineering community in industry, it is easier and faster to go through, line-by-line, by hand, than to write a piece of code to automate that process ("Can that even be done?" "No, it's impossible, just do it manually").

      I have encountered this over and over again in industry. Teams are willing to expend more than two hundred person hours to a reoccuring task and are willing to accept the associated fatigue errors to schedule systems by hand. Then they are shocked that I build a functional or logic-based system that autogenerates a pristine result in sub-second time.

      Where does this leave us?

      What I see as status quo is that industry is resigned to slog through repetitive tasks manually, because it's too hard in Language X to consider tackling the problem: more time would be spent writing, compiling and fixing a program to do the task than to do the task by hand.

      On the academic side, I see researchers isolated from industry, feeling their advances have no practical use as there appears to be no interest from industry. I attended a talk on dependent types where the thesis was that this typing makes coding more declarative, more powerful and less error prone. I approached the lecturer and asked how I could implement such a system and was asked what application dependent types had for industry.

      Hm. Let me think on that for a while.

      Why would industry be interested in coding more declaratively, more powerfully, and with less errors? I was flabbergasted that the researcher saw no benefit for industry in his presentation.

      We have a powerful tool in Haskell. We can do more, cleaner, safer, and do it faster than the mainstream alternatives industry picks over and over again by default.

      I've found, however, that talking about typing, or monads, or pure functional programming only causes the eyes to glaze.

      What lights a fire in management's eyes? When you rescue an overdue project, delivering before the deadline, and you educate other workers to be as productive ... perhaps not writing Haskell themselves (I've been asked: "Is it like Pascal?" ... and that's after a code walkthrough), but using the tools you've used to deliver these 'tremendous' results, and earning their trust that when you say, 'just let me write a Haskell script to do that: I'll take a day to write it, and we'll start getting results right away,' that that is actually what happens. I've found that 'we should use Haskell because of [all the wonderful arguments put forward]' does not convince anybody. Delivering 'incredible' and 'impossible' results, significantly faster than anticipated (two days instead of two months for the XML project)? That moves mountains.

      Where I work, Haskell is installed on two lead developer's systems other than my own, and I'm given permission to code in Haskell, on a 'we only allow Language X' project.

      Other Uses



      So attributed variables can be used to help one solve kakuro puzzles, or to write a sudoku solver. Fine.

      And I've been inching my way toward attributed variables, so it can be applied to symbolic logic puzzles as well, replacing my Whatever type with attributed variables over the enumerated domain.

      So the domain is not restricted to Number, but these are not very interesting or pragmatic examples in and of themselves. Not very gripping.

      So how about this example?

      *Stocks> makeAttribute [Buy, Sell, Hold]

      with Constraints being input feeds filtered through weighted moving averages and other indicators. Now this is a system of keen interest for me and 'currently under active and intense research.'

      Will such a system work or not work? I'm not at liberty to say,[9] but here is an area that appears to fit well in the language space of constraint programming.

      As problem spaces attacked by software projects — workflow analysis, data mining, planning and scheduling, budgeting, forecasting, (war)gaming, modelling and simulation — many seem good candidates to use constraint-based reasoning. Attributed variables give Haskell programmers another way to express these problem domains.



      Endnotes:



      [1] Y’all don’t mind if I speak logically, and in the logic-programming paradigm, do you?

      [2] See my article in The Monad.Reader, issue 11, “MonadPlus: What a Super Monad!” for an implementation of choose, et al.

      [3] No, I’m not speaking disparagingly of the monadic programming style. I rather like that style, and use it in my domain language of predicate logic that I layer on top of ‘regular’ Haskell programming. The beauty of monads, for me, is that they are invisible (in coding them) and powerful (in their effect). When the code becomes more about monads and less of what they are there to do is when I see this as an opportunity to examine things in a different light.

      [4] http://www.swi-prolog.org/pldoc/doc_for?object=section(2,'6.1',swi('/doc/Manual/attvar.html'))

      [5] My representation of Domain is straightforward and workable for most cases, but it hides more than a few compromises (or 'sell-outs') beneath the surface. For example, it makes the (gross) assumption that the range of the domain is enumerable. If Domain is in ℜ that no longer holds (at all), so then Domain shouldn't have a list representation but a pair (or bookending) bounds. But then what if Domain extends beyond ℜ and goes into the surreal numbers?[6] Then we have infintesimals such as '*' (pron: 'star') where:

      * > 0, * < 0, * + * = 0

      and:

      * || 0 (that is: 'star' is incomparable to 0)

      So how does one bounds-check on incomparables?

      But these concerns are, in the main, not ones to lose one's head over. For the most part, simple linear programming solves a vast majority of the World's problems,[7] and those interesting cases are exactly that: interesting cases.

      [6] http://en.wikipedia.org/wiki/Surreal_number

      [7] The preface of How to Solve It: Modern Heuristics by Zbigniew Michalewicz and David B. Fogel makes this assertion, from hard-won experience in the field. Seconded.

      [8] This is one approach to attributed variables, that is: constraints or attributes are applied to the variable over the domain until it is reduced to a single value, at which point the constraints and even the domain become superfluous and simply go away. An advantage to this approach is that a test for groundedness becomes trivial. A disadvantage is that once grounded one loses the path or information that got one there except through the data flow through the algorithm (the stack trace, as it were). Another disadvantage is that an attributed variable, once grounded, automatically rejects constraints. This may lead to unexpected program behavior, because AV grounded to 5 with a new constraint of x > 23 should fail, but my implementation forwards the 5 value in a success state.

      [9] An audit trail of at least a year of data is necessary before any claims can be made.

      by geophf (noreply@blogger.com) at April 20, 2011 10:59 AM

      April 19, 2011

      Eric Kow (kowey)

      why darcs users care about consistency

      In the Darcs community, we've been discussing the recent blog posts saying that Git is inconsistent, that it cannot be made to be consistent.

      With Darcs being the foil to Git for the purposes of this discussion, I thought it would be useful if I cleared up a few points, particularly this first one:

      consistency is a usability issue

      When people say they like Darcs, they don't generally talk about it having a beautiful or elegant theory. Instead, they talk about how easy and simple it is to use, about how they never really had to grapple with a learning curve or feel stupid for doing something wrong.

      What makes Darcs so simple to use? Did it hit the right notes by accident or through David Roundy's good taste? Or is usability merely in the eye of the beholder? Some of these explanations may be true, but I think what lies at the heart of Darcs' usability is that it supports a very simple way of understanding a repository:

      a darcs repository is a set of patches

      This mental model may not be suitable for everybody, and in the long run Darcs may need to improve its support for history tracking.  But if you want to understand why, for all its current shortcomings, people continue to use and develop Darcs, you must appreciate how refreshingly simple the set-of-patches mental model can be.  As a Darcs user you are freed from a lot of the artefacts of worrying about commit order.  Collaborating with people is just question of shuffling patches around, with no merge states, no rebases, way fewer spurious dependencies to worry about.

      But simplicity is hard.  In order to make this simple world view possible, Darcs has to guarantee a property that any ordering of patches allowed by Darcs commutation rules is equivalent. If Darcs gives you the option of skipping a patch, it has to work hard to make sure that if you include the patch later on, that the repository you get is equivalent. That's what the patch theory fuss is about.  While it's useful that Darcs tends to attract purists and math geeks, we're really not engaged in the pursuit of some sort of ivory tower theoretical elegance for its own sake.  Ultimately what we're after is usability.

      A good user interface minimises work for the user, be it cognitive, memory or physical work. The joy of Darcs is being able to focus cognitive work on our real jobs, and not on babysitting version control systems.  So when Russell O'Connor says that merges ought to be associative, he's not saying this to tick some sort of mathematical box, what I think he's really saying is as a Darcs user, he doesn't want to worry about the difference between pushing patches one at a time vs all in one go. Consistency is a usability issue.

      darcs is imperfect

      Darcs is very much a work in progress.  Some users have felt let down by Darcs: whenever performance grew to be unacceptable for their repositories, when they hit one exponential merge too many, or when Darcs just plain did something wrong. Even our much vaunted usability has cracks at the edges, a confirmation prompt too many, an inconsistent flag set, a non-reversible operation or two.

      I particularly want to make sure I'm very clear about this point:

      darcs patch theory is incomplete

      We still don't know how to cope with complicated conflicts. Moreover the implementation of our first two theories is somewhat buggy. Darcs copes well enough with most every day conflicts, but if a conflict gets hairy enough, Darcs will crash and emit a nasty message.  This is one of the reasons why we don't recommend Darcs for large repositories.

      Our version of "don't do that" is not to maintain long term feature branches without merging back to the trunk on a regular basis. This is not acceptable for bigger projects, but for smaller projects like Darcs itself, the trade-off between a simple user interface in the general case, and the occasional hairy conflict can be worth it. In the long run, we have to fix this. We are revising our patch theory again, this time taking a much more rigorous and systematic approach to the problem.

      In the interim, we will be gaining some powerful new tools to help work around the problem, namely a new "darcs rebase" feature that will allow users to smooth away conflicts rather than letting them get out of hand. This will be a crucial bridging tool while we continue to attack the patch theory problem.

      patch theory is simple at heart

      I am in the awkward position of being a non-expert maintainer, having to defer a lot of thinking about software engineering and patch theory to the rest of the Darcs team. In a way, this is healthy for Darcs, because we have long suffered from an excess concentration of expertise. Inverting the pie so that you basically have the number one Darcs Fan as the maintainer is useful because it forces everybody else to break things down into words an Eric can understand.

      The good news is that basic patch theory is one of these things an Eric can understand: patches have inverses and may sometimes be commuted.  Just learning the core theory teaches you how merging and cherry picking works, why you can trust the set-of-patches abstraction and most importantly, how simple Darcs is. So we're not after some kind of magical AI here, nor are we trying to guess user intention. The things we do with patches are much more mechanical, systematically adjusting patches to context, one at a time, click-clack on the abacus until the merge is complete.

      patch vs snapshot is not so important

      We think it's important to continue working on Darcs because we are exploring territory that no other version control system is looking at - patch-based version control. That said, patches and snapshots are duals of each other. We think that things that Darcs can do are possible in snapshot based version control and we would be very interested to see work in that direction.

      The secret to Darcs merging is that it replaces guesswork (fuzz factor) with history. A darcs patch only exists in the context of its predecessors, and if we want to apply a patch to a different context, we mechanically transform the patch to fit. We think this sort of history-aware merging could be implemented in Git. In fact, we would be excited to see somebody taking up the challenge. Git fans! How about stealing history-aware merging from us?

      exponential merges still exist but there are fewer of them

      We have developed two versions of patch theory. The second version avoids a lot of the common causes of exponential merge blowups, but it is still possible to trigger them. Recent Darcs repositories are created using version 2 of the theory. For compatibility's sake, repositories created before Darcs 2 came along tend to still be using version 1 of the theory (we only recommend converting if conflicts become a problem).

      The most well-known remaining cause of blowups in theory 2 is the problem of "conflict fights" where one side of the conflict resolves the conflict and gets on with their life without propagating the resolution back to the other side. What tends to happen there is that we not only encounter the conflict again in the future, but we also conflict with the resolution!

      So life is definitely better with Darcs 2. We've given the exponential merge problem a good knock on the head, but it's still staggering around and we're working our way to the finishing blow.

      performance is improving

      I think that when people complain about Darcs being slow, they're not talking about the exponential merge problem. They're mostly referring to day-to-day issues like the time it takes to check out a repository. Our recent focus has been to solve a lot of these pedestrian performance issues. For example, the upcoming Darcs 2.8 is like to use a new "packs" feature which makes it possible to fetch a repository in the form of two larger tarballs rather than thousands of little patch files. This makes a big difference!

      Another improvement we hope to bring to Darcs 2.8 is the performance of the darcs annotate command (cf. git blame).  Annotate has neglected for a while, and to make things better, we've basically reimplemented the command from scratch with more readable output to boot.  As an example of something fixed along the way, one misfeature of the old annotate is that would work by applying all the patches relevant to a given file, building it up from the very beginning.  But if you think about it, annotating a file is really about annotating its current state; we don't care about ancient history! So one of the Darcs hackers had the sort of idea that’s obvious in hindsight: rather than applying patches forwards from the beginning of history, we simply unapply them from the end.  Much faster.

      We're not yet trying to compete with Git when working on these performance issues. We admire the performance that Git can deliver and we agree that getting speed right is a usability issue (too slow and your user loses their train of thought).  But we've been picking a lot of low hanging fruit lately, solving problems that make Darcs faster with very little cost. We hope you'll like the results!

      by kowey (noreply@blogger.com) at April 19, 2011 11:55 PM

      April 18, 2011

      Nicolas Wu

      Do The Hokey Cokey

      A recent blog post by Jeremy Gibbons, Morality and Temptation, left the exercise of finding two definitions of the arrow with the signature \(μF → νF\), and showing that they are equivalent. This post shows my solution to this problem, and while I’m here, I too have a confession to make: when dealing with the ins and outs of initial algebras and final coalgebras I sometimes get the Hokey Cokey playing in my mind. For those of you who don’t know this children’s song, it goes something like this:

      You put your left foot in,
      Your left foot out,
      In, out, in, out,
      You shake it all about.
      You do the Hokey Cokey,
      And you turn around,
      That’s what it’s all about!

      Now that’s out of the way, let’s focus on the problem at hand. I’ll start by summarising some of the basics of what we’re tackling.

      Your F-Algebra In

      Given a functor \(F : \C → \C\), an F-Algebra is a pair \((X, f)\), where \(X ∈ \C\), and \(f ∈ \C(F X, X)\). F-Algebras are objects in the category written \(Alg_F\), where an arrow in this category is given by an \(h ∈ \C\) such that the following property holds between two objects \((X, f)\), and \((Y, g)\): \[ h . f = g . F h \] The construction of a \(fold\) is given by looking at the initial object in this category, written as \((μF, in)\). Initial objects have a unique arrow to every object in the category; in this category the unique arrow to an object \((Y, g)\) is given by \(\fold{g}\), and it makes the following commute (you’ll have to excuse the ASCII diagrams, I have yet to find a good way of rendering commutative diagrams for the web):

              F(|g|)
         F μF ─——————> F Y
          │             │
       in │             │ g
          │             │
          V             V
          μF ---------> Y
               (|g|)
      

      Following the arrows of this diagram gives us most of the ingredients needed to state the Universal Property of a \(fold\), which is as follows: \[ h . in = g . F h ⇔ \fold{g} = h \] In other words, \(\fold{g}\) is the unique solution that satisfies this commuting diagram.

      Your F-Coalgebra Out

      Given a functor \(F : \C → \C\), an F-coalgebra is the categorical dual of an F-algebra and inhabits the \(Coalg_F\) category. Here, objects are pairs \((X, f)\), where \(X ∈ \C\), and \(f ∈ \C(X, F X)\).

      Unsurprisingly, we are interested in the final object, written \((νF, out)\), of this category, where the unique arrow from every object \((X, f)\) to the final object is written \(\unfold{f}\), and the following commutes:

             F[(f)]
        F X ────────> F νF
         ^             ^
         │             │
       f │             │ out
         │             │
         │             │
         X ----------> νF
              [(f)]
      

      The universal property also dualises, and in this case we have: \[ h = \unfold{f} ⇔ F h . f = out . h \] This property simply states that \(\unfold{f}\) is the unique equation satisfying the commuting diagram.

      In, Out, In, Out

      Our goal is to find two definitions of an arrow of type \(μF → νF\), and to show that these definitions are equivalent. Looking at the definition of an initial F-algebra, it’s pretty obvious that we can produce such an arrow by taking the fold of an arrow with type \(F(νF) → νF\). Using the inverse of \(out\), which we’ll write as \(out°\), is a fairly obvious choice and produces the following diagram:

               F(|out°|)
          F μF ─———————> F νF
           │              │
           │              │
        in │              │ out°
           │              │
           V              V
           μF ----------> νF
               (|out°|)
      

      By dualizing, the alternative definition of this arrow becomes quite apparent, where the unfold of \(in°\) is used instead:

               F[(in°)]
          F μF ─———————> F νF
           ^              ^
           │              │
       in° │              │ out
           │              │
           │              │
           μF ----------> νF
               [(in°)]
      

      So, we now have two definitions for the arrow in question, \(\fold{out°}\), and \(\unfold{in°}\). All that is left is to show that these two definitions are equivalent.

      Shake It All About

      The universal properties of folds and unfolds are a useful way of encoding lots of information about their structure in one place. Consequently, these properties are very useful tools for proving properties of folds and unfolds.

      By plugging our two arrows into the equations that describe the universal properties of folds and unfolds, we get the following: \[ \unfold{in°} . in = out° . F \unfold{in°} ⇔ \fold{out°} = \unfold{in°} ⇔ F \fold{out°} . in° = out . \fold{out°} \] I think this equation is marvellous, I don’t know if it has a name, but for the purpose of this post I’ll call it the Hokey Cokey Law. The left hand side comes from instantiating the universal property for folds with an unfold, and the right hand side comes from instantiating the universal property for unfolds with a fold.

      Do the Hokey Cokey

      We still haven’t proved that \(\fold{out°} = \unfold{in°}\), but proving either the left hand side or the right hand side of the equation is all that is required. Let’s start with what is known: the universal property of \(fold\) applied to \(\fold{out°}\):

      \(\fold{out°} . in = out° . F \fold{out°}\)

      \(\hint{⇒}{Leibniz, twice}\)

      \(out . \fold{out°} . in . in° = out . out° . F \fold{out°} . in°\)

      \(\hint{⇒}{Identity, twice}\)

      \(out . \fold{out°} = F \fold{out°} . in°\)

      \(\hint{⇔}{Hokey Cokey}\)

      \(\fold{out°} = \unfold{in°}\)

      So there you have it, the Hokey Cokey in all its glory. Of course, we could take this proof and turn around, since everything dualises: we could have equally picked the universal of \(unfold\) applied to \(\unfold{in°}\), and would have arrived at the left hand side of our equivalence. That’s what it’s all about!

      Woah! The Hokey Cokey!
      Woah! The Hokey Cokey!
      Woah! The Hokey Cokey!
      Knees bent, arms stretched, ra ra ra!

      April 18, 2011 12:00 AM

      April 08, 2011

      Tillmann Vogt

      GSoC proposal

      A few days ago I made a short proposal for a GSoC project: trac

      I hope to explain a little bit better why a 3d user interface for hackage is desirable und how to do it. Chrome, Firefox and Webkit (Apple) now support WebGL meaning that the following pictures should roughly be possible inside a browser:

      A picture from a motorola campaign

      A picture produced by the company gamr7: www.gamr7.com (with modifications)

      The second picture was modified to illustrate the potential of visualization. The yellow lines show code and dependencies. Similar things have been done by others (I had this idea on my own): codecity . I want to do this in a very general way. I was able to convince the advisor of my diploma thesis to let me do half of it about visualization of flow networks. This was hard but he finally agreed (maybe because Haskell programmers are rare and he had a plan to translate flow networks into Haskell). In the visualization part I layed foundations to make a visualization like the above. I use symmetry classes (based on the book “the symmetries of things” ) and a way to produce smallest deviations of symmetry.

      In what ways will this project benefit the wider Haskell community?  

      At the end of the 3 month period:

      • The visualization will let programmers quicker find a library they need (see changes, etc. ), because the visualization algorithm is like reversed pattern recognition to produce most recognizable shapes.
      • The code size, popularity, dependencies, whatever the community demands can be integrated or left away
      • A general way to extend or change the visualization

      => Haskell could have an impressive package repository.

      The people who implement WebGL in Browsers (like Google) could have a poster child application

      After GSoC:

      • A framework could emerge that lets you show visualizations near to the library, the code, the code-line. This allows more freedom than opening a separate graphics window. I.e. it could be helpful sometimes to see how example data is transformed by a function
      • A visualization of data structures could be on the channels of flow networks as a first step and help debugging programs. I personally would enjoy it if I could find the bug in the triangulation algorithm of one of my libraries. That is very hard with a command line
      • Having an alternative to bindings to UI-libraries

      What deliverables do you think are reasonable targets? Can you outline an approximate schedule of milestones?

      Milestone 1

      • Change Sourcegraph to parse all libraries (not only one), the result should be a big graph of all code
      • Experiment with a spring layout for the placement of libraries. The springs could be dependencies
      • Maybe somewhere in the hierarchy Graphviz could be used for the layout: Convert the graph with Graphviz and parse the annotated dot files with language-dot
      • A data structure is generated from various data sources and then transformed into a collada file with my library collada-output

      Milestone 2

      • Use a WebGL engine to display the collada file. The problems here are most likely the picking, zooming and following of a link
      • Integration into Hackage2 as an optinal link
      • Helping to finish Hackage2
      • Documentation, everything I have forgot

      Why do you think you would be the best person to tackle this project?

      I did the relevant libraries to prepare this, among it a library that is a complete reprogramming of Freetype2 (as mentioned above it is hard to find algorithmic errors). Other people who did shape generation (also here in the Haskell community) haven’t considered symmetry as thorougly as me. Maybe Cabal and GHC are more important but this visualization project could in the long term help to understand Cabal and GHC and there is also a lot to do.


      by tillmannvogt at April 08, 2011 08:36 AM

      April 05, 2011

      André Pang (ozone)

      Immutability and Blocks, Lambdas and Closures [UPDATE x2]

      I recently ran into some “interesting” behaviour when using lambda in Python. Perhaps it’s only interesting because I learnt lambdas from a functional language background, where I expect that they work a particular way, and the rest of the world that learnt lambdas through Python, Ruby or JavaScript disagree. (Shouts out to you Objective-C developers who are discovering the wonders of blocks, too.) Nevertheless, I thought this would be blog-worthy. Here’s some Python code that shows the behaviour that I found on Stack Overflow:

      <script src="https://gist.github.com/899755.js?file=python_lambda_scope.py"></script>

      Since I succumb to reading source code in blog posts by interpreting them as “blah”1, a high-level overview of what that code does is:

      1. iterate over a list of strings,
      2. create a new list of functions that prints out the strings, and then
      3. call those functions, which prints the strings.

      Simple, eh? Prints “do”, then “re”, then “mi”, eh? Wrong. It prints out “mi”, then “mi”, then “mi”. Ka-what?

      (I’d like to stress that this isn’t a theoretical example. I hit this problem in production code, and boy, it was lots of fun to debug. I hit the solution right away thanks to the wonders of Google and Stack Overflow, but it took me a long time to figure out that something was going wrong at that particular point in the code, and not somewhere else in my logic.)

      The second answer to the Stack Overflow question is the clearest exposition of the problem, with a rather clever solution too. I won’t repeat it here since you all know how to follow links. However, while that answer explains the problem, there’s a deeper issue. The inconceivable Manuel Chakravarty provides a far more insightful answer when I emailed him to express my surprise at Python’s lambda semantics:

      This is a very awkward semantics for lambdas. It is also probably almost impossible to have a reasonable semantics for lambdas in a language, such as Python.

      The behaviour that the person on SO, and I guess you, found surprising is that the contents of the free variables of the lambdas body could change between the point in time where the closure for the lambda was created and when that closure was finally executed. The obvious solution is to put a copy of the value of the variable (instead of a pointer to the original variable) into the closure.

      But how about a lambda where a free variable refers to a 100MB object graph? Do you want that to be deep copied by default? If not, you can get the same problem again.

      So, the real issue here is the interaction between mutable storage and closures. Sometimes you want the free variables to be copied (so you get their value at closure creation time) and sometimes you don’t want them copied (so you get their value at closure execution time or simply because the value is big and you don’t want to copy it).

      And, indeed, since I love being categorised as a massive Apple fanboy, I found the same surprising behaviour with Apple’s blocks semantics in C, too:

      <script src="https://gist.github.com/899755.js?file=blocks_scope_closure_unexpected.c"></script>

      You can see the Gist page for this sample code to see how to work around the problem in Objective-C (basically: copy the block), and also to see what it’d look like in Haskell (with the correct behaviour).

      In Python, the Stack Overflow solution that I linked to has an extremely concise way of giving the programmer the option to either copy the value or simply maintain a reference to it, and the syntax is clear enough—once you understand what on Earth what the problem is, that is. I don’t understand Ruby or JavaScript well enough to comment on how they’d capture the immediate value for lambdas or whether they considered this design problem. C++0x will, unsurprisingly, give programmers full control over lambda behaviour that will no doubt confuse the hell out of people. (See the C++0x language draft, section 5.1.2 on page 91.)

      In his usual incredibly didactic manner, Manuel then went on to explain something else insightful:

      I believe there is a deeper issue here. Copying features of FP languages is the hip thing in language design these days. That’s fine, but many of the high-powered FP language features derive their convenience from being unspecific, or at least unconventional, about the execution time of a piece of code. Lambdas delay code execution, laziness provides demand-dependent code execution plus memoisation, continuations capture closures including their environment (ie, the stack), etc. Another instance of that problem was highlighted by Joe Duffy in his STM retrospective.

      I would say, mutability and flexible control flow are fundamentally at odds in language design.

      Indeed, I’ve been doing some language exploration again lately as the lack of static typing in Python is really beginning to bug me, and almost all the modern languages that attempt to pull functional programming concepts into object-oriented land seem like a complete Frankenstein, partially due to mutability. Language designers, please, this is 2011: multicore computing is the norm now, whether we like it or not. If you’re going to make an imperative language—and that includes all your OO languages—I’ll paraphrase Tim Sweeney: in a concurrent world, mutable is the wrong default! I’d love a C++ or Objective-C where all variables are const by default.

      One take-away point from all this is to try to keep your language semantics simple. I love Dan Ingall’s quote from Design Principles Behind Smalltalk: “if a system is to serve the creative spirit, it must be entirely comprehensible to a single individual”. I love Objective-C partially because its message-passing semantics are straightforward, and its runtime has a amazingly compact API and implementation considering how powerful it is. I’ve been using Python for a while now, and I still don’t really know the truly nitty-gritty details about subtle object behaviours (e.g. class variables, multiple inheritance). And I mostly agree with Guido’s assertion that Python should not have included lambda nor reduce, given what Python’s goals are. After discovering this quirk about them, I’m still using the lambda in production code because the code savings does justify the complexity, but you bet your ass there’s a big comment there saying “warning, pretentous code trickery be here!”

      1. See point 13 of Knuth et al.’s Mathematical Writing report.

      UPDATE: There’s a lot more subtlety at play here than I first realised, and a couple of statements I’ve made above are incorrect. Please see the comments if you want to really figure out what’s going on: I’d summarise the issues, but the interaction between various language semantics are extremely subtle and I fear I’d simply get it wrong again. Thank you to all the commenters for both correcting me and adding a lot of value to this post. (I like this Internet thing! Other people do my work for me!)

      Update #2

      I’ve been overwhelmed by the comments, in both the workload sense and in the pleasantly-surprised-that-this-provoked-some-discussion sense. Boy, did I get skooled in a couple of areas. I’ve had a couple of requests to try to summarise the issues here, so I’ll do my best to do so.

      Retrospective: Python

      It’s clear that my misunderstanding of Python’s scoping/namespace rules is the underlying cause of the problem: in Python, variables declared in for/while/if statements will be declared in the compound block’s existing scope, and not create a new scope. So in my example above, using a lambda inside the for loop creates a closure that references the variable m, but m’s value has changed by the end of the for loop to “mi”, which is why it prints “mi, mi, mi”. I’d prefer to link to the official Python documentation about this here rather than writing my own words that may be incorrect, but I can’t actually find anything in the official documentation that authoritatively defines this. I can find a lot of blog posts warning about it—just Google for “Python for while if scoping” to see a few—and I’ve perused the entire chapter on Python’s compound statements, but I just can’t find it. Please let me know in the comments if you do find a link, in which case I’ll retract half this paragraph and stand corrected, and also a little shorter.

      I stand by my assertion that Python’s for/while/if scoping is slightly surprising, and for some particular scenarios—like this—it can cause some problems that are very difficult to debug. You may call me a dumbass for bringing assumptions about one language to another, and I will accept my dumbassery award. I will happily admit that this semantics has advantages, such as being able to access the last value assigned in a for loop, or not requiring definitions of variables before executing an if statement that assigns to those variables and using it later in the same scope. All language design decisions have advantages and disadvantages, and I respect Python’s choice here. However, I’ve been using Python for a few years, consider myself to be at least a somewhat competent programmer, and only just found out about this behaviour. I’m surprised 90% of my code actually works as intended given these semantics. In my defence, this behaviour was not mentioned at all in the excellent Python tutorials, and, as mentioned above, I can’t a reference for it in the official Python documentation. I’d expect that this behaviour is enough of a difference vs other languages to at least be mentioned. You may disagree with me and regard this as a minor issue that only shows up when you do crazy foo like use lambda inside a for loop, in which case I’ll shrug my shoulders and go drink another beer.

      I’d be interested to see if anyone can come up an equivalent for the “Closures and lexical closures” example at http://c2.com/cgi/wiki?ScopeAndClosures, given another Python scoping rule that assignment to a variable automatically makes it a local variable. (Thus, the necessity for Python’s global keyword.) I’m guessing that you can create the createAdder closure example there with Python’s lambdas, but my brain is pretty bugged out today so I can’t find an equivalent for it right now. You can simply write a callable class to do that and instantiate an object, of course, which I do think is about 1000x clearer. There’s no point using closures when the culture understands objects a ton better, and the resulting code is more maintainable.

      Python summary: understand how scoping in for/while/if blocks work, otherwise you’ll run into problems that can cost you hours, and get skooled publicly on the Internet for all your comrades to laugh at. Even with all the language design decisions that I consider weird, I still respect and like Python, and I feel that Guido’s answer to the stuff I was attempting would be “don’t do that”. Writing a callable class in Python is far less tricky than using closures, because a billion times more people understand their semantics. It’s always a design question of whether the extra trickiness is more maintainable or not.

      Retrospective: Blocks in C

      My C code with blocks failed for a completely different reason unrelated to the Python version, and this was a total beginner’s error with blocks, for which I’m slightly embarrassed. The block was being stack-allocated, so upon exit of the for loop that assigns the function list, the pointers to the blocks are effectively invalid. I was a little unlucky that the program didn’t crash. The correct solution is to perform a Block_copy, in which case things work as expected.

      Retrospective: Closures

      Not all closures are the same; or, rather, closures are closures, but their semantics can differ from language to language due to many different language design decisions—such as how one chooses to define the lexical environment. Wikipedia’s article on closures has an excellent section on differences in closure semantics.

      Retrospective: Mutability

      I stand by all my assertions about mutability. This is where the Haskell tribe will nod their collective heads, and all the anti-Haskell tribes think I’m an idiot. Look, I use a lot of languages, and I love and hate many things about each of them, Haskell included. I fought against Haskell for years and hated it until I finally realised that one of its massive benefits is that things bloody well work an unbelievable amount of the time once your code compiles. Don’t underestimate how much of a revelation this is, because that’s the point where the language’s beauty, elegance and crazy type system fade into the background and, for the first time, you see one gigantic pragmatic advantage of Haskell.

      One of the things that Haskell does to achieve this is the severe restriction on making things immutable. Apart from the lovely checkbox reason that you can write concurrent-safe algorithms with far less fear, I truly believe that this makes for generally more maintainable code. You can read code and think once about what value a variable holds, rather than keep it in the back of your mind all the time. The human mind is better at keeping track of multiple names, rather than a single name with different states.

      The interaction of state and control flow is perhaps the most complex thing to reason about in programming—think concurrency, re-entrancy, disruptive control flow such as longjmp, exceptions, co-routines—and mutability complicates that by an order of magnitude. The subtle difference in behaviour between all the languages discussed in the comments is exemplar that “well-understood” concepts such as lexical scoping, for loops and closures can produce a result that many people still don’t expect; at least for this simple example, these issues would have been avoided altogether if mutability was disallowed. Of course mutability has its place. I’m just advocating that we should restrict it where possible, and at least a smattering of other languages—and hopefully everyone who has to deal with thread-safe code—agrees with me.

      Closing

      I’d truly like to thank everyone who added their voice and spent the time to comment on this post. It’s been highly enlightening, humbling, and has renewed my interest in discussing programming languages again after a long time away from it. And hey, I’m blogging again. (Though perhaps after this post, you may not think that two of those things are good things.) It’s always nice when you learn something new, which I wouldn’t have if not for the excellent peer review. Science: it works, bitches!

      by André Pang (ozone) (ozone@algorithm.com.au) at April 05, 2011 05:55 AM

      April 03, 2011

      The York Haskell Compiler

      Yhc is dead

      Someone recently asked on the Yhc mailing list if Yhc was dead. The answer is yes, noone has been working on the compiler for several years. The nhc98 compiler, from which Yhc originally forked, is still maintained. Since this is the end of Yhc, I thought I'd share a few random thoughts about Yhc:


      • Writing a Haskell compiler is a big undertaking, and the work required to compile a moderate number of programs from Hackage is immense. Too many libraries rely on something GHC specific - either language or runtime features. The GHC team have gone out of their way to ensure that their compiler is by default a standard Haskell compiler, which has kept Haskell as a viable language separate from GHC. But without widely-used competing implementations, programs tend to end up depending on GHC.

      • There are still other non-GHC Haskell compilers, and I wish them all well. Many offer features missing in GHC (compile to Javascript, tiny runtime system, extreme portability, code mobility etc.) - there are lots of interesting ideas floating around.

      • Yhc started in a fairly haphazard way, and became a fork of nhc long before anyone had noticed. Had Yhc's contributions been rolled back into nhc they'd probably have had more lasting impact.

      • One of the big draws for Yhc was it's ability to take a Haskell program, and produce a Core complete program. Despite all of Yhc's other weaknesses, Yhc Core drew several people to the project - I hope something equivalent appears for other Haskell compilers.

      • If you don't understand monads, you aren't yet ready to write a Haskell compiler.

      • If you have a group of Haskell programmers in the same place, you should try a group project. It's fun.

      • The biggest challenge for Yhc was the build system - we ended up with 10,000 lines of Python Scons scripts. Without a robust build system nothing else matters. When our sole Python hacker left the team that was the beginning of the end.



      Working on Yhc was fun, and I learned a lot. Thanks to everyone who was involved.

      by Neil Mitchell (noreply@blogger.com) at April 03, 2011 08:52 PM

      Christopher Lane Hinson

      ANN: Roguestar 0.6.0.0

      Roguestar 0.6.0.0 is now up on hackage.  You can install and run with:

      $ cabal install roguestar roguestar-glut roguestar-engine
      & ~/.cabal/bin/roguestar
      

      The latest version brings:

      • New monsters: Hellions and Dust Vortexes
      • Underground dungeons
      • Power-ups (hidden in the aforementioned dungeons) that serve to level-up the player’s character.
      • Improved walking animation.
      • It’s no longer possible to spawn with zero sight range.
      • Fixed some of the worst of the user interface glitches.

      Under the hood, I’ve also split RSAGL into a few topic-specific libraries and implemented value recursion in rsagl-frp.  There is an experimental GTK-based client, but I don’t recommend it just yet, which is why it isn’t on hackage.

      I’ve also migrated the project to github, which includes source code, the manual, and the issue tracker.


      by Christopher Lane Hinson at April 03, 2011 02:04 AM

      April 02, 2011

      Gergely Patai

      Elerea documentation rewritten

      I noticed that there have been some misunderstandings about the capabilities and other characteristics of Elerea, so I sat down today and updated the documentation. Also, I deprecated the variant with the automatic delays, because it’s a difficult to handle beast. The new version is already live on Hackage. I still need to update the examples not to use the legacy interfaces, but they shouldn’t look very different after the rewrite.

      If you have any advice or comment regarding the renewed docs, don’t hesitate to share it with me.

      by Patai Gergely (noreply@blogger.com) at April 02, 2011 08:41 PM

      Ulisses Costa

      From LALR(1) to LL(k)

      I, Pedro Faria and Jose Pedro Silva have been working on a LALR to LL conversor. Our idea was to discover the problems involved in this type of automatic conversion. Our main goal here was to receive an LALAR(1) grammar and produce an LL(k) one. As a test case we receive an Yacc and produce [...]

      by Ulisses Araújo Costa at April 02, 2011 02:05 PM

      Permutations in pure functional ActionScript

      Explanation of permutation function from a Functional Programming definition to ActionScript

      by Ulisses Araújo Costa at April 02, 2011 12:22 AM

      March 24, 2011

      Sean Leather

      Draft: "Type-Changing Program Improvement with Type and Transform Systems"

      Johan Jeuring, Andres Löh,and I have submitted a paper to ICFP 2011.

      Type-Changing Program Improvement with Type and Transform Systems

      Structured program improvement is monotonous and error-prone and should be automated whenever possible. Typical approaches such as refactoring are too restrictive, allowing for only minimal changes to types in the program. Other improvement techniques that allow type changes are few and somewhat limited.

      We are interested in type-driven program transformations that spread virally throughout code, infecting both function definitions and applications. Such transformations are beneficial for software evolution or migrating a program to a new library interface. An example transformation is the optimization of list-based strings to Hughes' lists, a transformation that can potentially infect every string in a program.

      We present type and transform systems, an approach to type-safe, automatic program improvement in which complex transformations can be expressed by simple inference rules that define a relation between source and target programs. In this paper, we describe the relations, their properties, how to design transformations, and a prototype implementation. We also demonstrate automatic string optimization, which, to the best of our knowledge, has not been shown before.

      The prototype code is also available for a test drive.

      by Sean Leather (noreply@blogger.com) at March 24, 2011 02:07 PM

      March 16, 2011

      Haskell Web News

      Hackage Stats: The Past Year

      It’s been a while since I’ve crunched the Hackage / Cabal logs. Here’s an update, covering March 2010 – Feb 2011.

      Hackage

      By March 2011, there were 2894 packages on Hackage. There were 2.02 million “cabal installs” in the 12 months to March 2011 (doubling the download rate of the previous year). There have been just shy of 4 million total “cabal installs” total, since Hackage went live. By 2011, Hackage was averaging over 200k downloads a month, more than double the same period a year ago.

      Hackage continues to grow very quickly, averaging 18 package releases a day in March 2011, compared with 12 a day a year ago.

      The Long Tail

      The long tail of package popularity is visible on the log/log scale of downloads versus package rank, where the top 10% of packages, get 80% of the downloads

      .

      Libraries

      The top 15 libraries in 2010 were:

      1. parsec (30.3k downloads)
      2. HTTP (27.1k)
      3. network (26.8k)
      4. mtl (26.1k)
      5. binary (23.9k)
      6. zlib (23.5k)
      7. transformers (20.7k)
      8. utf8-string (19.7k)
      9. QuickCheck (19.4k)
      10. Cabal (17.5k)
      11. text (16.6k)
      12. haskell-src-exts (15.2k)
      13. deepseq (15k)
      14. regex-base (14.5k)
      15. hslogger (14k)

      Apps

      The top 15 apps in 2010

      1. cabal-install (25.9k downloads
      2. haddock (12.5k)
      3. xmonad (10.4k)
      4. cpphs (10.4k)
      5. happy (9.7k)
      6. alex (8.3k)
      7. pandoc (6.8k)
      8. darcs (6.4k)
      9. snap (5.5k)
      10. hlint (5k)
      11. leksah (4.5k)
      12. happstack (4k)
      13. xmobar (3.9k)
      14. yesod (3.1k)
      15. cabal2arch (2.6k)

      Some noticeable changes: the 3 core web frameworks are way up in the list.

      • xml is the most popular xml library (12k) vs HaXml (7.6k), hexpat (4.3k), hxt (3.1k).
      • For graphics, cairo is the clear winner (12.5k), gtk (8k), wxcore (3k)
      • vector is the most popular arrays library (10k)
      • cereal is closing in on binary (9.7k)
      • attoparsec is climbing (8.3k)
      • HDBC remains the most popular datbase layer (6.5k)

      The yearly totals for each package are available here.


      by dons00 at March 16, 2011 09:50 PM

      March 13, 2011

      Christopher Lane Hinson

      The One Function per Typeclass Rule

      After about five years programming in Haskell, I think we need a rule:  Only put one function in a typeclass.

      Why?  Because inevitably someone comes along with a data type for which one or the other function of a typeclass is perfectly suited, and yet another function of the same typeclass is not implementable.

      Here are some examples of consequences of breaking the rule:

      1. The Infamous Set Monad, which requires splitting Monad in half. (Monad)
      2. All of the abstract ways to construct Nothing: fail (Monad), mempty (Monoid), mzero (MonadPlus), empty (Alternative).  Not surprisingly, all of these typeclasses are subtly related.
      3. The natural numbers, which have a minBound, but not a maxBound (Bounded) . . .
      4. . . . and which support addition and multiplication, but aren’t closed under subtraction and for which the concept of a sign does not exist (Num).
      5. My own memoizable message type, which would like to implement Applicative, but needs a monadic computation to implement pure. (Applicative)

      It’s a little extra typing to write multiple “class . . . where” clauses for each type that needs to implement a large number of type-indexed functions, but it’s quite easy to combine related typeclasses when appropriate, as follows:

      class Foo a where
          foo :: a -> b
      
      class Bar a where
          bar :: a -> b
      
      class (Foo a,Bar a) => FooBar a where
          {this space intentionally left blank}
      

      In conclusion, you should definitely follow this rule if I have convinced you that it is a good idea to follow it.


      by Christopher Lane Hinson at March 13, 2011 09:59 PM

      March 10, 2011

      Simon Meier

      Efficient UTF-8 Encoding

      Belgien beer and UTF-8 encoding: what a coincidence. Well, this post is just a quick post about the results for UTF-8 encoding 100'000'000 characters to /dev/null using the various options that our current Haskell ecosystem offers. It happensto be written after a very nice Belgian evening spent in the awesome Minimum climbing gym in Zurich Wollishofen. Here's the code and the corresponding Makefile. The following results are the wall-time results obtained on my Core 2 Duo laptop with 2GB RAM using GHC 7.0.2 on a 32bit Linux 2.6.32-29 installation.

      text         1.08s
      blaze        1.70s
      base         2.63s
      via-text     5.97s
      utf8-string 47.99s
      utf8-light  40.18s

      The result for text is the time it takes to UTF-8 encode a nicely packed lazy text value of length 100'000'000. The other times for blaze, base, via-text, and utf8-string measure how long it takes to UTF-8 encode a String of length 100'000'000 using blaze-builder, GHC 7.0.2's base library, packing to a lazy text value and encoding it, and using the utf8-string library. Sadly, utf8-light supports only strict ByteStrings and, hence, it's not really usable to UTF-8 encode a String that long. Therefore, the time given for utf8-light is the time it takes to encode a String of length 1'000'000 a hundred times.

      My conclusions after this experiment are the following:

      1. Using [Word8] list based encoding implementations is likely to result in suboptimal performance.
      2. It is not worth packing a String first to a Text value, if it is encoded right away again.
      3. The current work I'm spending on integrating blaze-builder with the bytestring library is really worth the effort. Compared to the text benchmark, which uses a better (packed) representation of Char sequences, we are only a factor 1.6 slower. Moreover, it might even be worth a try to replace the String encoding functions in the base library by according Builders to gain these additional 50 percent of speed. As an additional benefit, we could even think of executing Builders directly on the buffer associated to a Handle and, therefore, output byte streams denoted by Builders without any buffer allocations.
      4. It would be very interesting to see how well we fare against other languages. If a reader would implement the same benchmark in C, C++, Java, Python, ... I'd be very glad to publish the obtained results here.

      by Simon Meier (noreply@blogger.com) at March 10, 2011 11:43 PM

      February 22, 2011

      Alson Kemp

      ExtJS 4 Models + node.js

      Finally starting to play with node.js.   Also, getting back into developing with the lovely ExtJS.  ExtJS 4 added strong support for client side models.

      I was thinking that it’d be nice to share a lot of code for models between the client and server.   Turns out that it’s not that difficult.   Super quick-n-dirty code below.  Now the question is: how much duplication can be removed from client and server models? I don’t want to include all server-side code in client-side code, so might do something like:

      • /common/models/user.js – ExtJS model;
      • /client/models/user.js – tune the model for client side (e.g. add a REST connection to the server);
      • /server/models/user.js – includes client/models/user.js;  overrides critical bits (e.g. the Proxy); adds a bunch of server specific code.

      If all of my models are in model.*, then I can probably iterate through them and auto-generate Mongoose models when the server boots…  Fun.

      This is definitely a hack, but isn’t as frighteningly ugly as I expected:

      fs = require('fs');
       
      // stub out a fake browser for ext-core-debug.js
      navigator = {};
      window = {
        navigator : 'Linux',
        attachEvent: function() {return false;}
      };
      navigator = {'userAgent' : 'node'};
      document = {
        documentElement:'',
        getElementsByTagName : function () {return false;}};
      // Helper function 
      function injectJS(f) {
        eval(fs.readFileSync(f, encoding="ascii"));
      }
       
      //Pull in ExtJS components
      injectJS('./ext-core-debug.js');
      injectJS('./src/util/Filter.js');
      injectJS('./src/util/Sorter.js');
      injectJS('./src/util/Observable.js');
      injectJS('./src/data/Connection.js');
      injectJS('./src/Ajax.js');
      injectJS('./src/util/Stateful.js');
      injectJS('./src/util/Inflector.js');
      injectJS('./src/util/MixedCollection.js');
      injectJS('./src/data/ResultSet.js');
      injectJS('./src/data/Batch.js');
      injectJS('./src/data/Reader.js');
      injectJS('./src/data/JsonReader.js');
      injectJS('./src/data/Writer.js');
      injectJS('./src/data/JsonWriter.js');
      injectJS('./src/data/Errors.js');
      injectJS('./src/data/Operation.js');
      injectJS('./src/data/Proxy.js');
      injectJS('./src/data/ServerProxy.js');
      injectJS('./src/data/AjaxProxy.js');
      injectJS('./src/data/RestProxy.js');
      injectJS('./src/data/validations.js');
      injectJS('./src/util/Date.js');
      injectJS('./src/data/SortTypes.js');
      injectJS('./src/data/Association.js');
      injectJS('./src/data/Types.js');
      injectJS('./src/util/Observable.js');
      injectJS('./src/util/HashMap.js');
      injectJS('./src/AbstractManager.js');
      injectJS('./src/PluginMgr.js');
      injectJS('./src/data/Field.js');
      injectJS('./src/data/BelongsToAssociation.js');
      injectJS('./src/data/HasManyAssociation.js');
      injectJS('./src/data/PolymorphicAssociation.js');
      injectJS('./src/data/Model.js');
      injectJS('./src/ModelMgr.js');
       
      // Register the model
      Ext.regModel('models.User', {
            fields: [
                {name: 'name',  type: 'string'},
                {name: 'age',   type: 'int'},
                {name: 'phone', type: 'string'},
                {name: 'alive', type: 'boolean', defaultValue: true}
          ],
          validations: [
              {type: 'presence',  field: 'age'},
              {type: 'length',    field: 'name',     min: 2}
          ],
          changeName: function() {
              var oldName = this.get('name'),
              newName = oldName + " The Barbarian";
              this.set('name', newName);
          }
      });
       
      // Create an instance
      var user = Ext.ModelMgr.create({
            name : 'Conan',
              age  : 24,
              phone: '555-555-5555'
      }, 'models.User');
       
      // Use the instance
      user.changeName();
      user.get('name'); //returns "Conan The Barbarian"
      user.validate();
      user.addEvents('changed');
      user.events;
      user.fireEvent('changed', 'my hair');
       
      repl = require("repl");
      repl.start('ExtJS&gt; ');

      by alson at February 22, 2011 06:50 PM