# The new Stackage Server

I made the first commit to the Stackage Server code base a little over a year ago. The goal was to provide a place to host package sets which both limited the number of packages from Hackage available, and modified packages where necessary. This server was to be populated by regular Stackage builds, targeted at multiple GHC versions, and consisted of both inclusive and exclusive sets. It also allowed interested individuals to create their own package sets.

If any of those details seem surprising today, they should. A lot has happened for the Stackage project in the past year, making details of what was initially planned irrelevant, and making other things (like hosting of package documentation) vital. We now have LTS Haskell. Instead of running with multiple GHC versions, we have Stackage Nightly which is targeted at a single GHC major version. To accomodate goals for GPS Haskell (which unfortunately never materialized), Stackage no longer makes corrections to upstream packages.

I could go into lots more detail on what is different in project requirements. Instead, I'll just summarize: I've been working on a simplified version of the Stackage Server codebase to address our goals better, more easily ensure high availability, and make the codebase easier to maintain. We also used this opportunity to test out a new hosting system our DevOps team put together. The result is running on beta.stackage.org, and will replace the official stackage.org after a bit more testing (which I hope readers will help with).

## The code

All of this code lives on the simpler branch of the stackage-server code base, and much to my joy, resulted in quite a bit less code. In fact, there's just about a 2000 line reduction. The rest of this post will get into how that happened.

## No more custom package sets

One of the features I mentioned above was custom package sets. This fell out automatically from the initial way Stackage Server was written, so it was natural to let others create package sets of their own. However, since release, only one person actually used that feature. I discussed with him, and he agreed with the decision to deprecate and then remove that functionality.

So why get rid of it now? Two powerful reasons:

• We already host a public mirror of all packages on S3. Since we no longer patch upstream packages, it's best if tooling is able to just refer to that high-reliability service.
• We now have Git repositories for all of LTS Haskell and Stackage Nightly. Making these the sources of package sets means we don't have two (possibly conflicting) sources of data. That brings me to the second point

We had some complicated logic to allow users to upload package sets. It started off simple, but over time we added Haddock hosting and other metadata features, making the code more complex. Actually, it ended up having two parallel code paths for this. So instead, we now just upload information on the package sets to the Git repositories, and leave it up to a separate process (described below) to clone these repositories and make the data available to the server.

After generating a snapshot, the Haddocks used to be tarred and compressed, and then uploaded as a compressed bundle to S3. Then, Stackage Server would receive a request for files, unpack them, and serve them. This presented some problems:

• Users would have to wait for a first request to succeed during the unpacking
• With enough snapshots being generated, we would eventually run out of disk space and need to clear our temp directory
• Since we run our cluster in a high availabilty mode with multiple horizontally-scaled machines, one machine may have finished unpacking when another didn't, resulting in unstyled content (see issue #82).

Instead, we now just upload the files to S3 and redirect there from stackage-server (though we'll likely switch to reverse proxying to allow for nicer SSL urls). In fact, you can easily view these docs, at URLs such as http://haddock.stackage.org/lts-2.9/ or https://s3.amazonaws.com/haddock.stackage.org/nightly-2015-05-21/index.html.

These Haddocks are publicly available, and linkable from projects beyond Stackage Server. Each set of Haddocks is guaranteed to have consistent internal links to other compatible packages. And while some documentation doesn't generate due to known package bugs, the generation is otherwise reliable.

I've already offered access to these docs to Duncan for usage on Hackage, and hope that will improve the experience for users there.

Previously, information on snapshots was stored in a PostgreSQL database that was maintained by Stackage Server. This database also had package metadata, like author, homepage, and description. Now, we have a completely different process:

• The all-cabal-metadata from the Commercial Haskell Special Interest Group provides an easily cloneable Git repo with package metadata, which is automatically updated by Travis.
• We run a cron job on the stackage-build server that updates the lts-haskell, stackage-nightly, and all-cabal-metadata repos and generates a SQLite database from them with all of the data that Stackage Server needs. You can look at the Stackage.Database module for some ideas of what this consists of. That database gets uploaded to Amazon S3, and is actually publicly available if you want to poke at it
• The live server downloads a new version of this file on a regular basis

I've considered spinning off the Stackage.Download code into its own repository so that others can take advantage of this functionality in different contexts if desired. Let me know if you're interested.

At this point, the PostgreSQL database is just used for non-critical functionality, such as social features (tags and likes).

## Slightly nicer URLs

When referring to a snapshot, there are "official" short names (slugs), of the form lts-2.9 and nightly-2015-05-22. The URLs on the new server now reflect this perfectly, e.g.: https://beta.stackage.org/nightly-2015-05-22. We originally used hashes of the snapshot content for the original URLs, but that was fixed a while ago. Now that we only have to support these official snapshots, we can always (and exclusively) use these short names.

As a convenience, if you visit the following URLs, you get automatic redirects:

• /nightly redirects to the most recent nightly
• /lts to the latest LTS
• /lts-X to the latest LTS in the X.* major version (e.g., today, /lts-2 redirects to /lts-2.9)

This also works for URLs under that hierarchy. For example, consider https://beta.stackage.org/lts/cabal.config, which is an easy way to get set up with LTS in your project (by running wget https://beta.stackage.org/lts/cabal.config).

## ECS-based hosting

While not a new feature of the server itself, the hosting cluster we're running this on is brand new. Amazon recently released EC2 Container Service, which is a service for running Docker containers. Since we're going to be using this for the new School of Haskell, it's nice to be giving it a serious usage now. We also make extensive use of Docker for customer projects, both for builds and hosting, so it's a natural extension for us.

This ECS cluster uses standard Amazon services like Elastic Load Balancer (ELB) and auto-scaling to provide for high availability in the case of machine failure. And while we have a lot of confidence in our ability to keep Stackage Server up and running regularly, it's nice that our most important user-facing content is provided by these external services:

This provides for a pleasant experience in both browsing the website and using Stackage in your build system.

A special thanks to Jason Boyer for providing this new hosting cluster, which the whole FP Complete team is looking forward to putting through its paces.

## May 21, 2015

### Neil Mitchell

Summary: The development version of ghcid seemed to have some problems with terminating when Control-C was hit, so I investigated and learnt some things.

Given a long-running/interactive console program (e.g. ghcid), when the user hits Control-C/Ctrl-C the program should abort. In this post I'll describe how that works in Haskell, how it can fail, and what asynchronous exceptions have to do with it.

#### What happens when the user hits Ctrl-C?

When the user hits Ctrl-C, GHC raises an async exception of type UserInterrupt on the main thread. This happens because GHC installs an interrupt handler which raises that exception, sending it to the main thread with throwTo. If you install your own interrupt handler you won't see this behaviour and will have to handle Ctrl-C yourself.

There are reports that if the user hits Ctrl-C twice the runtime will abort the program. In my tests, that seems to be a feature of the shell rather than GHC itself - in the Windows Command Prompt no amount of Ctrl-C stops an errant program, in Cygwin a single Ctrl-C works.

#### What happens when the main thread receives UserInterrupt?

There are a few options:

• If you are not masked and there is no exception handler, the thread will abort, which causes the whole program to finish. This behaviour is the desirable outcome if the user hits Ctrl-C.
• If you are running inside an exception handler (e.g. catch or try) which is capable of catching UserInterrupt then the UserInterrupt exception will be returned. The program can then take whatever action it wishes, including rethrowing UserInterrupt or exiting the program.
• If you are running with exceptions masked, then the exception will be delayed until you stop being masked. The most common way of running while masked is if the code is the second argument to finally or one of the first two arguments to bracket. Since Ctrl-C will be delayed while the program is masked, you should only do quick things while masked.

#### How might I lose UserInterrupt?

The easiest way to "lose" a UserInterrupt is to catch it and not rethrow it. Taking a real example from ghcid, I sometimes want to check if two paths refer to the same file, and to make that check more robust I call canonicalizePath first. This function raises errors in some circumstances (e.g. the directory containing the file does not exist), but is inconsistent about error conditions between OS's, and doesn't document its exceptions, so the safest thing is to write:

# Generalized numerics

It used to be that diagrams were hard-coded to use Double. As of version 1.3, Double is no longer baked in: diagrams are now parameterized by a suitable numeric type. It's too early to tell what the full implications of this will be, but in theory it opens up opportunities for things like automatic differentiation, constraint solving, and using diagrams in conjunction with deeply embedded DSLs.

This feature in particular was a tough nut to crack and is the fruit of a lot of labor. I want to especially highlight the work of Jan Bracker, Allan Gardner, and Frank Staals, all of whom did a lot of work attempting to generalize diagrams in this way. Although their code ultimately did not get merged, we learned a lot from their attempts! The fourth attempt, by Chris Chalmers, actually stuck. A big factor in his success was to simultaneously replace the vector-space package with linear, which turns out to work very nicely with diagrams and with generalized numeric types in particular.

Note that this is a Very Breaking Change as the types of almost everything changed. Anything which used to take a single type representing a vector space (such as R2) as an argument now takes two arguments, one for the structure/dimension of the vector space (e.g. V2) and one for the numeric/scalar type. See the migration guide for more specific information and help upgrading!

# And lots more...

Of course, there are lots of other miscellaneous improvements, added type class instances, new lenses and prisms, bug fixes, and the like. For a full rundown see the release notes. There is also lots more exciting stuff in the pipeline!

# Contributors

We now have a team of 5 people working very actively on diagrams (Chris Chalmers, Daniel Bergey, Jeff Rosenbluth, Ryan Yates, and myself), with many, many more contributing here and there. As a way to say thank you for all the great contributions and hard work by many in the community, below is a (probably incomplete) list of people who have contributed to diagrams in some way — all 67 of them! We welcome involvement from anyone, regardless of experience. If you'd like to get involved, come chat with us on the #diagrams IRC channel on freenode, or send a message to the mailing list.

Diagrams contributors:

Alexis Praga / Allan Gardner / Andy Gill / Anthony Cowley / Bartosz Nitka / Ben Gamari / Brent Yorgey / Carlos Scheidegger / Carter Tazio Schonwald / Chris Mears / Christopher Chalmers / Claude Heiland-Allen / Conal Elliott / Daniel Bergey / Daniel Kröni / Daniel Wagner / Daniil Frumin / Deepak Jois / Denys Duchier / Dominic Steinitz / Doug Beardsley / Felipe Lessa / Florent Becker / Gabor Greif / Hans Höglund / Heinrich Apfelmus / Ian Ross / Jan Bracker / Jeffrey Rosenbluth / Jeremy Gibbons / Jeroen Bransen / Jim Snavely / Joachim Breitner / Joel Burget / John Lato / John Tromp / Jonas Haag / Kanchalai Suveepattananont / Kaspar Emanuel / Konrad Madej / Konstantin Zudov / Luite Stegeman / Michael Sloan / Michael Thompson / Moiman / Niklas Haas / Peter Hall / Pontus Granström / Robbie Gleichman / Robert Vollmert / Ryan Scott / Ryan Yates / Sam Griffin / Scott Walck / Sergei Trofimovich / sleepyMonad / soapie / Steve Sprang / Steven Smith / Tad Doxsee / Taneb / Taru Karttunen / Tillmann Vogt / Tim Docker / Vilhelm Sjöberg / Vincent Berthoux / Yiding Jia

# Building ghc-7.8.4 for Debian 7 (wheezy) with Stackage

This article is a recipe for building ghc-7.8.4 from source, along with the other tools you need to use Haskell. Why would you do this rather than use haskell-platform? Well, the latest haskell-platform is now over a year old, and ships with ghc-7.8.3, so perhaps you want or need the latest compiler in the 7.8 series. Or perhaps you're just curious as to how it might be done.

We're also going to ensure that any packages we install to get things going will be the ones from the current Stackage LTS.

So far, I've only tried this recipe on a Debian 7 (wheezy) server, although apart from the very first step it should be the same for any distro.

## Install a binary ghc

The very first step is to install a functioning binary build of haskell-platform from somewhere... presumably your distro's packages:

# apt-get install haskell-platform


(Throughout this recipe, I will prefix commands to be run as root with # and commands to be run as any non-root user with $. For the root commands, you can of course either use a root shell, or prefix each command with sudo.) We're also going to need some other bits and pieces to build a new ghc. On a pretty minimal Debian 7, all I needed was this: # apt-get install make libncurses5-dev  ## Build and install ghc This is all straight forward. We want the default installation paths under /usr/local so there is nothing to pass to configure. Build as some non-root user: $ wget https://downloads.haskell.org/~ghc/7.8.4/ghc-7.8.4-src.tar.xz
$tar xfJ ghc-7.8.4-src.tar.xz$ cd ghc-7.8.4
$sh configure$ make -j5


In the last line, the number 5 should be 1 more than the CPU cores you have available.

Now install as root:

# cd /home/toby/ghc-7.8.4
# make install


## Bootstrap cabal

This is the tricky part, that took me a while to work out. There is a very loose coupling between ghc and cabal, but they do need to agree on some things: cabal needs to install libraries to where ghc is going to look for them! The distro supplied cabal-1.14 won't work properly with our newly installed ghc as it uses the wrong value for libsubdir. The symptom of this is that any library including C code will install happily, but anything depending on such a library will then produce errors like these:

/usr/bin/ld: cannot find -lHStf-random-0.5-ghc7.8.4
/usr/bin/ld: cannot find -lHSprimitive-0.6-ghc7.8.4
/usr/bin/ld: cannot find -lHSrandom-1.1-ghc7.8.4


The cleanest way to fix this is to perform a two-stage boot of cabal: use cabal-1.14 to install a “stage1” cabal-1.18, then clean everything out and use that stage1 cabal to install a fully functional “stage2” cabal-1.18. The particular version of cabal we are using will be the one from the current LTS stackage:

# cd
# cabal update
# cabal install cabal-install==1.18.0.8
# cp .cabal/bin/cabal .
# rm -r .ghc .cabal
# ./cabal update


The last line creates a new cabal configuration file that just needs a couple of tweaks. First, we want to reset the user-install flag. You could do that in your favourite text editor, or with this handy sed script:

# sed -i '/user-install/auser-install: False' .cabal/config


And we want to append the LTS stackage global file to the end of the file:

# u='https://www.stackage.org/lts/cabal.config?global=true'
# wget -q -O- \$u >> .cabal/config


Now we're ready to perform the second stage install:

# ./cabal install cabal-install
# rm ./cabal
# hash -r # sacrifice goat to the great god Bash


## Install other tools

Finally, to finish off, install the other basic tools:

# cabal install alex
# cabal install happy