Planet Haskell
http://planet.haskell.org/
enPlanet Haskell - http://planet.haskell.org/Tweag I/O: Announcing the <br>GHC DevOps Grouphttp://www.tweag.io/posts/2017-10-19-ghc-devops-group.html
http://www.tweag.io/posts/2017-10-19-ghc-devops-group.html
<div>Manuel M T Chakravarty</div><p>As Haskell is increasingly used in production environments, the Haskell toolchain is of critical importance to a growing number of people and organisations. At the heart of this toolchain is <a href="https://www.haskell.org/ghc/">GHC (The Glasgow Haskell Compiler)</a>. Conceived nearly
thirty years ago as a research project by <a href="https://www.microsoft.com/en-us/research/people/simonpj/">Simon Peyton Jones</a>, GHC has long existed at the intersection of research and commercial use, due to the increasing popularity of Haskell. To provide stability and a basic level of support, the project was generously supported by Microsoft Research for many years. However, as more and more companies and others are using GHC in mission-critical ways, the bar is going up. We need a solid, reliable, well-engineered, predictable GHC toolchain; and we need to achieve that without compromising GHC innovation and vitality.</p>
<h2>Shared responsibility</h2>
<p>In response to this situation, GHC HQ and several stakeholder organisations decided to work together to improve the situation and to provide shared leadership and a broader pool of resources for the DevOps aspects of GHC development. As a concrete first step in this new partnership, we announced the formation of the <em>GHC DevOps Group</em> at the 2017 Haskell Implementors’ Workshop in Oxford.</p>
<h2>Our mission</h2>
<p>As set out in the group’s <a href="https://ghc.haskell.org/trac/ghc/wiki/DevOpsGroupCharter">charter</a>, the mission of the GHC DevOps Group is threefold:</p>
<ul>
<li>to take leadership of the DevOps aspects of GHC,</li>
<li>to resource it better, and</li>
<li>to broaden the sense of community ownership and control of GHC.</li>
</ul>
<p>As important as the goals are the group’s non-goals. The GHC DevOps group is exclusively concerned with the processes and tools for code development, community contributions, and regular, reliable & well-tested releases. It is about <strong>how</strong> and <strong>when</strong> features get into GHC and are shipped to users. In contrast, it is <em>not</em> concerned with the choice of <strong>which</strong> features go into GHC, let alone the content of the libraries it ships with. This is the responsibility of the <a href="https://github.com/ghc-proposals/ghc-proposals/#ghc-proposals">GHC Steering Committee and proposals process</a> and the <a href="https://wiki.haskell.org/Core_Libraries_Committee">Core Libraries Committee</a>, respectively.</p>
<h2>Our current goals</h2>
<p>During its formation, the group identified two initial goals:</p>
<ol>
<li>moving to two calendar-based GHC releases per year, and</li>
<li>lowering the barrier to contributing to GHC.</li>
</ol>
<h3>Quality calendar-based releases every six months</h3>
<p>As Ben Gamari documented in detail in his <a href="https://ghc.haskell.org/trac/ghc/blog/2017-release-schedule">Reflections on GHC’s release schedule</a>, actual GHC release dates are hard to predict and initial release quality is often low due to critical bugs. These are the typical symptoms that modern DevOps processes, such as continuous and automated integration, testing, and delivery, are designed to eliminate. Hence, we are working towards reliable continuous building and testing of GHC as well as pre-merge commit testing. Automating everything leads to the entire set of release artefacts being built and provided during that same process.</p>
<p>The main choice that we are currently facing is whether to build our own solution and maintain our own infrastructure based on <a href="https://jenkins.io">Jenkins</a> or whether to use 3rd party services, such as <a href="https://circleci.com">CircleCI</a> and <a href="https://www.appveyor.com">AppVeyor</a>, instead. Jenkins provides more flexibility (especially with respect to the supported architectures and operating systems), but at the expense of greater development and maintenance costs — that is, developer time that could otherwise be invested in improving GHC itself. In principle, the use of CircleCI and AppVeyor minimises the work needed to set up and maintain the infrastructure, but at the cost of being dependent on those 3rd parties, including their choice of directly supported architectures and operating systems.</p>
<p>Other key requirements involve security and the ability for anyone that forks GHC to also trivially fork and then modify the build infrastructure at will. For a summary of the requirements as well as the pros and cons of the two alternative approaches, have a look at the <a href="https://ghc.haskell.org/trac/ghc/wiki/ContinuousIntegration">CI Trac Wiki page</a>.</p>
<h3>Lowering the barrier to entry</h3>
<p>Although only a limited number of developers will feel confident in, say, extending GHC’s type checker, there are many parts of the compiler and associated tools and libraries that are well within reach of any competent Haskell developer. Hence, we feel that —just like other prominent open-source projects— we should make it as easy as possible to build, modify, and contribute to GHC. For better or worse, today, the gold standard for source control and processing code contributions is GitHub — if only, because virtually every professional developer, especially if engaged in open source work, is already familiar with it.</p>
<p>In contrast, GHC currently requires contributors to use the <a href="https://ghc.haskell.org/trac/ghc/wiki/Phabricator">Phabricator</a> tool. This is not only unfamiliar to most, but also requires the local installation of the command line tool Arcanist. In addition, Phabricator requires maintaining our own Phabricator installation and is known to be cumbersome to integrate with CI tool chains (tying this in with the first goal).</p>
<p>Lowering the barrier to entry involves other changes too of course (e.g., documentation structure). But to start with, what we are discussing is an attempt at encouraging more contributions while simultaneously lowering our set up and maintenance costs. This may well involve moving from Phabricator to GitHub.</p>
<h2>Transparency and contributions</h2>
<p>If you are interested in the details of the discussion, please have a look at the <a href="https://mail.haskell.org/pipermail/ghc-devops-group/"><code>ghc-dev-ops@haskell.org</code> archives</a>. To provide transparency to the wider GHC user and developer community, all discussions of the GHC DevOps Group will be recorded on that mailing list.</p>
<p>Please let us know what you think. Feel free to approach any <a href="https://ghc.haskell.org/trac/ghc/wiki/DevOpsGroupCharter#Membership">member</a> of the GHC DevOps Group with feedback or suggestions. For a broader discussion, you may want to follow up on the general <a href="https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs"><code>ghc-devs</code></a> mailing list.</p>
<p>All this requires resources: time and money. To make this work, we will need significant contributions from companies that get value from GHC (which is itself free). But in exchange we get something valuable: a GHC ecosystem that we can rely on. If you think your company could help, in cash or kind, <a href="mailto:manuel.chakravarty@tweag.io">please get in touch</a>.</p>Thu, 19 Oct 2017 00:00:00 +0000Gabriel Gonzalez: Advice for Haskell beginnerstag:blogger.com,1999:blog-1777990983847811806.post-7179176354240283462
http://www.haskellforall.com/2017/10/advice-for-haskell-beginners.html
<html xmlns="http://www.w3.org/1999/xhtml"><head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <meta content="text/css" http-equiv="Content-Style-Type"/> <meta content="pandoc" name="generator"/> <title></title> <style type="text/css">code{white-space: pre;}</style></head><body><p>This post summarizes advice that I frequently give to Haskell beginners asking how to start out learning the language</p><p>First, in general I recommend reading the <a href="http://haskellbook.com/">Haskell Programming from first principles</a> book, mainly because the book teaches Haskell without leaving out details and also provides plenty of exercises to test your understanding. This is usually good enough if you are learning Haskell as your first language.</p><p>However, I would like to give a few additional tips for programmers who are approaching Haskell from other programming languages.</p><h2 id="learn-haskell-for-the-right-reasons">Learn Haskell for the right reasons</h2><p>Some people learn Haskell with the expectation that they will achieve some sort of programming enlightenment or nirvana. You will be disappointed if you bring these unrealistic expectations to the language. Haskell is not an achievement to unlock or a trophy to be won because learning is a never-ending process and not a finish line.</p><p>I think a realistic expectation is to treat Haskell as a pleasant language to use that lets you focus on solving real problems (as opposed to wasting your time fixing silly self-induced problems like null pointers and "undefined is not a function").</p><h2 id="avoid-big-design-up-front">Avoid big-design-up-front</h2><p>Haskell beginners commonly make the mistake of trying to learn as much of the language as possible before writing their first program and overengineering the first draft. This will quickly burn you out.</p><p>You might come to Haskell from a dynamically typed background like JavaScript, Python, or Ruby where you learned to avoid refactoring large code bases due to the lack of static types. This aversion to refactoring promotes a culture of "big-design-up-front" where you try to get your project as close to correct on the first try so that you don't have to refactor your code later.</p><p>This is a <em>terrible</em> way to learn Haskell, for two reasons. First, Haskell has a much higher ceiling than most other programming languages, so if you wait until you hit that ceiling before building something you will wait a looooooong time. Second, refactoring is cheap in Haskell so you don't need to get things right the first time.</p><p>You will accelerate your learning process if you get dirty and make mistakes. Write really ugly and embarrassing code and then iteratively refine your implementation. There is no royal road to learning Haskell.</p><h2 id="avoid-typeclass-abuse">Avoid typeclass abuse</h2><p>Specifically, avoid creating new typeclasses until you are more comfortable with the language.</p><p>Functional programming languages excel because many language features are "first class". For example, functions and effects are first class in Haskell, meaning that you can stick them in a list, add them, nest them, or pass them as arguments, which you can't (easily) do in imperative languages.</p><p>However, typeclasses are not first-class, which means that if you use them excessively you will quickly depend on advanced language features to do even simple things. Programming functionally at the term-level is much simpler and more enjoyable than the type-level Prolog that type-classes encourage.</p><p>Begin by learning how to solve problems with ordinary functions and ordinary data structures. Once you feel like you understand how to solve most useful problems with these simple tools then you can graduate to more powerful tools like typeclasses. Typeclasses can reduce a lot of boilerplate in proficient hands, but I like to think of them as more of a convenience than a necessity.</p><p>You can also take this approach with you to other functional languages (like Elm, Clojure, Elixir, or Nix). You can think of "functions + data structures" as a simple and portable programming style that will improve all the code that you write, Haskell or not.</p><h2 id="build-something-useful">Build something useful</h2><p>Necessity is the mother of invention, and you will learn more quickly if you try to build something that you actually need. You will quickly convince yourself that Haskell is useless if you only use the language to solve Project Euler exercises or Sudoku puzzles.</p><p>You are also much more likely to get a Haskell job if you have a portfolio of one or two useful projects to show for your time. These sorts of projects demonstrate that you learned Haskell in order to build something instead of learning Haskell for its own sake.</p><h2 id="conclusion">Conclusion</h2><p>Hopefully these tips will help provide some guard rails for learning the language for the first time. That's not to say that Haskell is perfect, but I think you will enjoy the language if you avoid these common beginner pitfalls.</p></body></html>Mon, 16 Oct 2017 14:45:05 +0000noreply@blogger.com (Gabriel Gonzalez)Ken T Takusagawa: [duimuqdd] Compressing prime numberstag:blogger.com,1999:blog-6757805.post-4259559515531214240
http://kenta.blogspot.com/2017/10/duimuqdd-compressing-prime-numbers.html
<p dir="ltr">Given a large prime number p not of any special form, can one data-compress it to specify it in less than log_2(p) bits?</p><p dir="ltr">Straightforward is to say it is the m-th prime. Expressing m will take about log_2(p/log(p)) bits by the prime number theorem. This is the theoretical optimum by some measure, because the m's map 1-to-1 with the primes. The number of bits saved is log_2(p)- log_2(p/log(p)) = log_2(p) - [log_2(p) - log_2(log(p))] = log_2(log(p)) which, because of the twice-iterated logarithm, is a very small savings. For 370-bit primes around exp(256), the savings is 8 bits, or a reduction of 1 part in 46. For larger primes, the relative savings is even less. Around exp(2^16), the reduction is 1 part in 5909.</p><p dir="ltr">Despite the mostly uselessness of this endeavor, we charge forward.</p><p dir="ltr">Computing m is not practical for large primes. For a practical method, we form a new number x = 2^k * floor(p / 2^k), equivalent to setting k lower bits of p to zero. Then we can encode p as the n-th prime larger than x. For small k, n is easy to compute, and it is easy to recover p from n and x (e.g., with a prime sieve starting from x). Storing n instead of the lower bits of p results in some data savings because primes are less dense. Curiously, the number of bits saved on average is log_2(log(p)), which is the theoretical maximum, and not dependent on k. However, we also need to store k, which eats up some of those savings. Because the number of bits saved is constant, this method paradoxically works less and less well for larger k. However, with small k, n might vary significantly from the average, which could be good or bad. <a href="http://web.mit.edu/kenta/www/three/compress-primes/duimuqdd/">Here is some Haskell code to compute n.</a> Here is some example output on the prime 3^233 + 176, which is around exp(2^8). We can see there is generally around 8 bits of savings, except for 2^21 where we got lucky.</p><p>input 1476564251485392778927857721313837180933869708288569663932077079002031653266328641356763872492873429131586567699<br />
prime # 1 after 2^ 9 * 2883914553682407771343472111941088244011464274001112624867338044925843072785798127649929438462643416272630015<br />
prime # 3 after 2^ 10 * 1441957276841203885671736055970544122005732137000556312433669022462921536392899063824964719231321708136315007<br />
prime # 8 after 2^ 11 * 720978638420601942835868027985272061002866068500278156216834511231460768196449531912482359615660854068157503<br />
prime # 12 after 2^ 12 * 360489319210300971417934013992636030501433034250139078108417255615730384098224765956241179807830427034078751<br />
prime # 24 after 2^ 13 * 180244659605150485708967006996318015250716517125069539054208627807865192049112382978120589903915213517039375<br />
prime # 59 after 2^ 14 * 90122329802575242854483503498159007625358258562534769527104313903932596024556191489060294951957606758519687<br />
prime # 129 after 2^ 15 * 45061164901287621427241751749079503812679129281267384763552156951966298012278095744530147475978803379259843<br />
prime # 252 after 2^ 16 * 22530582450643810713620875874539751906339564640633692381776078475983149006139047872265073737989401689629921<br />
prime # 526 after 2^ 21 * 704080701582619084800652371079367247073111395019802886930502452374473406441845246008283554312168802800935<br />
prime # 8789 after 2^ 22 * 352040350791309542400326185539683623536555697509901443465251226187236703220922623004141777156084401400467<br />
</p><p dir="ltr">Future exploration: "Luck" above is easily explained by long internal strings of zeroes in the binary representation of p: the least significant 21 bits of p are 000011111111000010011. We could also try to take advantage of long internal strings of ones. More generally, sieving for primes works well for regions defined by arithmetic progressions. We might search generally for representations of p as the "a-th prime number of the form b*x + c larger/smaller than d*e^k". Above we explored b=1, c=0, "larger", e=2, d unrestricted. We could also explore anchor expressions more complicated than d*e^k.</p><p>Previously <a href="http://kenta.blogspot.com/2014/04/bqjglune-encoding-large-random-prime.html">a completely different way to encode large prime numbers in a small amount of space</a>.</p>Mon, 16 Oct 2017 09:26:11 +0000noreply@blogger.com (Ken)Mark Jason Dominus: Counting increasing sequences with Burnside's lemmatag:,2017:/math/increasing-sequences-2
https://blog.plover.com/math/increasing-sequences-2.html
<p>[ I started this article in March and then forgot about it. Ooops! ]</p>
<p>Back in February I posted <a href="https://blog.plover.com/math/increasing-sequences.html">an article about how there are exactly 715
nondecreasing sequences of 4
digits</a>.
I said that <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%2810%2c%204%29%24" /> was the set of such sequences and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24C%2810%2c%0a4%29%24" /> was the number of such sequences, and in general $$C(d,n) =
\binom{n+d-1}{d-1} = \binom{n+d-1}{n}$$ so in particular $$C(10,4) =
\binom{13}{4} = 715.$$</p>
<p>I described more than one method of seeing this, but I didn't mention
the method I had found first, which was to use the
<a href="https://en.wikipedia.org/wiki/Burnside%27s_lemma">Cauchy-Frobenius-Redfeld-Pólya-Burnside counting
lemma</a>.
<a href="https://blog.plover.com/math/polya-burnside.html">I explained
the lemma in detail</a> some time ago,
with beautiful illustrated examples, so I won't repeat the explanation
here. The Burnside lemma is a kind of big hammer to use here, but I
like big hammers. And the results of this application of the big
hammer are pretty good, and justify it in the end.</p>
<p>To count the number of distinct sequences of 4 digits, where some
sequences are considered “the same” we first identify a symmetry group
whose orbits are the equivalence classes of sequences. Here the
symmetry group is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S_4%24" />, the group that permutes the elements of the
sequence, because two sequences are considered “the same” if they have
exactly the same digits but possibly in a different order, and the
elements of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S_4%24" /> acting on the sequences are exactly what you want
to permute the elements into some different order.</p>
<p>Then you tabulate how many of the 10,000 original sequences are left
fixed by each element <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24p%24" /> of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S_4%24" />, which is exactly the number of
cycles of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24p%24" />. (<a href="https://blog.plover.com/math/fixpoints.html">I have also discussed cycle classes of permutations
before</a>.) If <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24p%24" /> contains <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%24" />
cycles, then <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24p%24" /> leaves exactly <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%2410%5en%24" /> of the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%2410%5e4%24" /> sequences fixed.</p>
<p></p><table cellpadding="6" align="center" cellspacing="0">
<tbody><tr><td align="center"><span style="font-size: 120%;">Cycle class</span>
</td><td>Number<br />of cycles</td><td>How many<br />permutations?</td><td>Sequences<br />left fixed
</td></tr><tr><td><img src="https://pic.blog.plover.com/math/increasing-sequences-2/fp4-1111.jpg" /></td><td>4</td><td align="right">1
</td><td align="right">10,000
</td></tr><tr><td><img src="https://pic.blog.plover.com/math/increasing-sequences-2/fp4-211.jpg" /></td><td>3
</td><td align="right">6
</td><td align="right">1,000
</td></tr><tr><td>
<img src="https://pic.blog.plover.com/math/increasing-sequences-2/fp4-22.jpg" /><br />
<img src="https://pic.blog.plover.com/math/increasing-sequences-2/fp4-31.jpg" />
</td><td>2
</td><td align="right">3 + 8 = 11
</td><td align="right">100
</td></tr><tr><td><img src="https://pic.blog.plover.com/math/increasing-sequences-2/fp4-4.jpg" />
</td><td>1
</td><td align="right">6
</td><td align="right">10
</td></tr><tr style="border: black medium solid;"><td>
</td><td></td><td align="right"><span style="font-size: 130%;">24</span>
</td><td align="right"><span style="font-size: 130%;">17,160</span>
</td></tr></tbody></table><p></p>
<p>(Skip this paragraph if you already understand the table. The
four rows above are an abbreviation of the full table, which has 24
rows, one for each of the 24 permutations of order 4. The “How many
permutations?” column says how many times each row should be
repeated. So for example the second row abbreviates 6 rows, one for
each of the 6 permutations with three cycles,
which each leave 1,000 sequences fixed, for a total of 6,000 in the
second row, and the total for all 24 rows is 17,160. There are two
different types of permutations that have two cycles, with 3 and 8
permutations respectively, and I have collapsed these into a single
row.)</p>
<p>Then the magic happens: We average the number left fixed by each
permutation and get <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac%7b17160%7d%7b24%7d%20%3d%20715%24" /> which we already know
is the right answer.</p>
<p>Now suppose we knew how many permutations there were with each number
of cycles. Let's write
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cdef%5cst%231%232%7b%5cleft%5c%5b%7b%231%5catop%20%232%7d%5cright%5d%7d%5cst%20nk%24" /> for the number of
permutations of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%24" /> things that have exactly <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24k%24" /> cycles. For example, from
the table above we see that
$$\st 4 4 = 1,\quad
\st 4 3 = 6,\quad
\st 4 2 = 11,\quad
\st 4 1 = 6.$$</p>
<p>Then applying Burnside's lemma we can conclude
that $$S(d, n) = \frac1{n!}\sum_i \st ni d^i .\tag{$\spadesuit$}$$ So for example the table above
computes <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%2810%2c4%29%20%3d%20%5cfrac1%7b24%7d%5csum_i%20%5cst%204i%2010%5ei%20%3d%20715%24" />.</p>
<p>At some point in looking into this I noticed that
$$\def\rp#1#2{#1^{\overline{#2}}}%
\def\fp#1#2{#1^{\underline{#2}}}%
S(d,n) =
\frac1{n!}\rp dn$$ where <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5crp%20dn%24" /> is the so-called “<a href="https://en.wikipedia.org/wiki/Rising_power">rising
power</a>” of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24d%24" />: $$\rp dn = d\cdot(d+1)(d+2)\cdots(d+n-1).$$
I don't think I had a proof of this; I just noticed that <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%28d%2c%201%29%20%3d%0ad%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%28d%2c%202%29%20%3d%20%5cfrac12%28d%5e2%2bd%29%24" /> (both obvious), and the Burnside's
lemma analysis of the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%3d4%24" /> case had just given me <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%28d%2c%204%29%20%3d%0a%5cfrac1%7b24%7d%28d%5e4%20%2b6d%5e3%20%2b%2011d%5e2%20%2b%206d%29%24" />. Even if one doesn't immediately
recognize this latter polynomial it looks like it ought to factor and
then on factoring it one gets <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24d%28d%2b1%29%28d%2b2%29%28d%2b3%29%24" />. So it's easy to
conjecture <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%28d%2c%20n%29%20%3d%20%5cfrac1%7bn%21%7d%5crp%20dn%24" /> and indeed, this is easy to
prove from <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28%5cspadesuit%29%24" />: The <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cst%20n%20k%24" /> obey the recurrence
$$\st{n+1}k = n \st nk + \st n{k-1}\tag{$\color{green}{\star}$}$$ (by an easy combinatorial argument<a href="http://#fn1"><sup>1</sup></a>)
and it's also easy to show that the
coefficients of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5crp%20nk%24" /> obey the same recurrence.<a href="http://#fn2"><sup>2</sup></a></p>
<p>In general <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5crp%20nk%20%3d%20%5cfp%7b%28n%2bk%2d1%29%7dk%24" /> so we have
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%28d%2c%20n%29%20%3d%20%5crp%20dn%20%3d%20%5cfp%7b%28n%2bd%2d1%29%7dn%20%3d%0a%5cbinom%7bn%2bd%2d1%7dd%20%3d%20%5cbinom%7bn%2bd%2d1%7d%7bn%2d1%7d%24" /> which ties the knot with the formula from <a href="https://blog.plover.com/math/increasing-sequences.html">the
previous article</a>. In particular, <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24S%2810%2c4%29%20%3d%20%5cbinom%7b13%7d9%24" />.</p>
<p>I have a bunch more to say about this but this article has already
been in the oven long enough, so I'll cut the scroll here.</p>
<hr />
<p><a name="fn1">[1]</a> The combinatorial argument that justifies
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28%5ccolor%7bgreen%7d%7b%5cstar%7d%29%24" /> is as follows:
The Stirling number <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cst%20nk%24" /> counts the number of permutations of
order <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%24" /> with exactly <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24k%24" /> cycles.
To get a permutation of order <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%2b1%24" /> with exactly <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24k%24" /> cycles, we
can take one of the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cst%20nk%24" /> permutations of order <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%24" /> with <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24k%24" />
cycles and insert the new element into one of the existing cycles
after any of the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%24" /> elements. Or we can take on of the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cst%0an%7bk%2d1%7d%24" /> permutations with only <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24k%2d1%24" /> cycles and add the new element
in its own cycle.)</p>
<p><a name="fn2">[2]</a> We want to show that the coefficients of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5crp%0ank%24" /> obey the same recurrence as
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28%5ccolor%7bgreen%7d%7b%5cstar%7d%29%24" />. Let's say that the coefficient of the
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%5ei%24" /> term in <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5crp%20nk%24" /> is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c_i%24" />.
We have $$\rp n{k+1} = \rp nk\cdot (n+k) = \rp
nk \cdot n + \rp nk \cdot k $$ so the
coefficient of the
the <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24n%5ei%24" /> term on the left is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c_%7bi%2d1%7d%20%2b%20kc_i%24" />.</p>Sun, 15 Oct 2017 23:09:00 +0000mjd@plover.com (Mark Dominus)Dan Piponi (sigfpe): A tail we don't need to wagtag:blogger.com,1999:blog-11295132.post-6611224798233016626
http://blog.sigfpe.com/2017/10/a-tail-we-dont-need-to-wag.html
<br /><b>Introduction</b><p>I've been reading a little about <a href="https://en.wikipedia.org/wiki/Concentration_inequality">concentration inequalities</a> recently. I thought it would be nice to see if you can use the key idea, if not the actual theorems, to reduce the complexity of computing the probability distribution of the outcome of stochastic simulations. Examples might include <a href="https://en.wikipedia.org/wiki/Random_walk">random walks</a>, or <a href="https://en.wikipedia.org/wiki/Queueing_theory">queues</a>. </p><p><br />The key idea behind concentration inequalities is that very often most of the probability is owned by a small proportion of the possible outcomes. For example, if we toss a fair coin enough (say <img src="https://chart.googleapis.com/chart?cht=tx&chl=N" style="vertical-align: middle;" />) times we expect the number of heads to lie within <img src="https://chart.googleapis.com/chart?cht=tx&chl=2%5Csqrt%7BN%7D" style="vertical-align: middle;" /> of the mean about 99.99% of the time despite there being <img src="https://chart.googleapis.com/chart?cht=tx&chl=N%2B1" style="vertical-align: middle;" /> different total numbers possible. The probable outcomes tend to concentrate around the expectation. On the other hand, if we consider not the total number of heads, but the possible sequences of <img src="https://chart.googleapis.com/chart?cht=tx&chl=N" style="vertical-align: middle;" /> tosses, there are <img src="https://chart.googleapis.com/chart?cht=tx&chl=2%5EN" style="vertical-align: middle;" /> possibilities, all equally likely. In this case there is no concentration. So a key ingredient here is a reduction operation: in this case reducing a sequence of tosses to a count of the number that came up heads. This is something we can use in a computer program. </p><p><br />I (and many others) have written about the <a href="http://blog.sigfpe.com/2007/03/monads-vector-spaces-and-quantum.html">"vector space" monad</a> that can be used to compute probability distributions of outcomes of simulations and I'll assume some familiarity with that. Essentially it is a "weighted list" monad which is similar to the list monad except that in addition to tracking all possible outcomes, it also propagates a probability along each path. Unfortunately it needs to follow through every possible path through a simulation. For example, in the case of simulating <img src="https://chart.googleapis.com/chart?cht=tx&chl=N" style="vertical-align: middle;" /> coin tosses it needs to track <img src="https://chart.googleapis.com/chart?cht=tx&chl=2%5EN" style="vertical-align: middle;" /> different possiblities, even though we're only interested in the <img src="https://chart.googleapis.com/chart?cht=tx&chl=N%2B1" style="vertical-align: middle;" /> possible sums. If, after each bind operation of the monad, we could collect together all paths that give the same total then we could make this code much more efficient. The catch is that to collect together elements of a type the elements need to be comparable, for example instances of <tt>Eq</tt> or <tt>Ord</tt>. This conflicts with the type of <tt>Monad</tt> which requires that we can use the <tt>>>= :: m a -> (a -> m b) -> m b</tt> and <tt>return :: a -> m a</tt> functions with any types <tt>a</tt> and <tt>b</tt>. </p><p><br />I'm going to deal with this by adapting a technique presented by Oleg Kiselyov for <a href="http://okmij.org/ftp/Haskell/set-monad.html">efficiently implementing the Set monad</a>. Instead of <tt>Set</tt> I'm going to use the <tt>Map</tt> type to represent probability distributions. These will store maps saying, for each element of a type, what the probability of that element is. So part of my code is going to be a direct translation of that code to use the <tt>Map</tt> type instead of the <tt>Set</tt> type. </p><p><br /></p><pre><br />> {-# LANGUAGE GADTs, FlexibleInstances #-}<br />> {-# LANGUAGE ViewPatterns #-}<br /><p><br /><br />> module Main where<br /></p><p><br /><br />> import Control.Monad<br />> import Control.Arrow<br />> import qualified Data.Map as M<br />> import qualified Data.List as L<br /></p><p><br /><br /></p></pre>The following code is very similar to Oleg's. But for first reading I should point out some differences that I want you to ignore. The type representing a probability distribution is <tt>P</tt>: <p><br /></p><pre><br />> data P p a where<br />> POrd :: Ord a => p -> M.Map a p -> P p a<br />> PAny :: p -> [(a, p)] -> P p a<br /><p><br /><br /></p></pre>But note how the constructors take two arguments - a number that is a probability, in addition to a weighted <tt>Map</tt> or list. For now pretend that first argument is zero and that the functions called <tt>trimXXX</tt> act similarly to the identity: <p><br /></p><pre><br />> instance (Ord p, Num p) => Functor (P p) where<br />> fmap = liftM<br /><p><br /><br />> instance (Ord p, Num p) => Applicative (P p) where<br />> pure = return<br />> (<*>) = ap<br /></p><p><br /><br />> instance (Ord p, Num p) => Monad (P p) where<br />> return x = PAny 0 [(x, 1)]<br />> m >>= f = <br />> let (e, pdf) = unP m<br />> in trimAdd e $ collect $ map (f *** id) pdf<br /></p><p><br /><br />> returnP :: (Ord p, Num p, Ord a) => a -> P p a<br />> returnP a = POrd 0 $ M.singleton a 1<br /></p><p><br /><br />> unP :: P p a -> (p, [(a, p)])<br />> unP (POrd e pdf) = (e, M.toList pdf)<br />> unP (PAny e pdf) = (e, pdf)<br /></p><p><br /><br />> fromList :: (Num p, Ord a) => [(a, p)] -> M.Map a p<br />> fromList = M.fromListWith (+)<br /></p><p><br /><br />> union :: (Num p, Ord a) => M.Map a p -> M.Map a p -> M.Map a p<br />> union = M.unionWith (+)<br /></p><p><br /><br />> scaleList :: Num p => p -> [(a, p)] -> [(a, p)]<br />> scaleList weight = map (id *** (weight *))<br /></p><p><br /><br />> scaleMap :: (Num p, Ord a) => p -> M.Map a p -> M.Map a p<br />> scaleMap weight = fromList . scaleList weight . M.toList<br /></p><p><br /><br /></p></pre>This is a translation of Oleg's crucial function that allows us to take a weighted list of probability distributions and flatten them down to a single probability distribution: <p><br /></p><pre><br />> collect :: Num p => [(P p a, p)] -> P p a<br />> collect [] = PAny 0 []<br />> collect ((POrd e0 pdf0, weight) : rest) =<br />> let wpdf0 = scaleMap weight pdf0<br />> in case collect rest of<br />> POrd e1 pdf1 -> POrd (weight*e0+e1) $ wpdf0 `union` pdf1<br />> PAny e1 pdf1 -> POrd (weight*e0+e1) $ wpdf0 `union` fromList pdf1<br />> collect ((PAny e0 pdf0, weight) : rest) =<br />> let wpdf0 = scaleList weight pdf0<br />> in case collect rest of<br />> POrd e1 pdf1 -> POrd (weight*e0+e1) $ fromList wpdf0 `union` pdf1<br />> PAny e1 pdf1 -> PAny (weight*e0+e1) $ wpdf0 ++ pdf1<br /><p><br /><br /></p></pre>But now I really must explain what the first argument to <tt>POrd</tt> and <tt>PAny</tt> is and why I have all that "trimming". <p><br />Even though the <tt>collect</tt> function allows us to reduce the number of elements in our PDFs, we'd like to take advantage of concentration of probability to reduce the number even further. The <tt>trim</tt> function keeps only the top <img src="https://chart.googleapis.com/chart?cht=tx&chl=N" style="vertical-align: middle;" /> probabilities in a PDF, discarding the rest. To be honest, this is the only point worth taking away from what I've written here :-) </p><p><br />When we throw away elements of the PDF our probabilities no longer sum to 1. So I use the first argument of the constructors as a convenient place to store the amount of probability that I've thrown away. The <tt>trim</tt> function keeps the most likely <img src="https://chart.googleapis.com/chart?cht=tx&chl=N" style="vertical-align: middle;" /> outcomes and sums the probability of the remainder. I don't actually need to keep track of what has been discarded. In principle we could reconstruct this value by looking at how much the probabilities in our trimmed partial PDFs fall short of summing to 1. But confirming that our discarded probability and our partial PDF sums to 1 gives a nice safety check for our code and can give us some warning if numerical errors start creeping in. I'll call the total discarded probability the <i>tail</i> probability. </p><p><br />Here is the core function to keep the top <img src="https://chart.googleapis.com/chart?cht=tx&chl=N" style="vertical-align: middle;" /> values. In this case <img src="https://chart.googleapis.com/chart?cht=tx&chl=N" style="vertical-align: middle;" /> is given by a global constant called <tt>trimSize</tt>. (I'll talk about how to do this better later.) </p><p><br /></p><pre><br />> trimList :: (Ord p, Num p) => [(a, p)] -> (p, [(a, p)])<br />> trimList ps =<br />> let (keep, discard) = L.splitAt trimSize (L.sortOn (negate . snd) ps)<br />> in (sum (map snd discard), keep)<br /><p><br /><br />> trimAdd :: (Ord p, Num p) => p -> P p a -> P p a<br />> trimAdd e' (POrd e pdf) =<br />> let (f, trimmedPdf) = trimList (M.toList pdf)<br />> in POrd (e'+e+f) (M.fromList trimmedPdf)<br />> trimAdd e' (PAny e pdf) =<br />> let (f, trimmedPdf) = trimList pdf<br />> in PAny (e'+e+f) trimmedPdf<br /></p><p><br /><br />> runP :: (Num p, Ord a) => P p a -> (p, M.Map a p)<br />> runP (POrd e pdf) = (e, pdf)<br />> runP (PAny e pdf) = (e, fromList pdf)<br /></p><p><br /><br /></p></pre>And now some functions representing textbook probability distributions. First the uniform distribution on a finite set. Again this is very similar to Oleg's <tt>chooseOrd</tt> function apart from the fact that it assigns weights to each element: <p><br /></p><pre><br />> chooseP :: (Fractional p, Ord p, Ord a) =><br />> [a] -> P p a<br />> chooseP xs = let p = 1/fromIntegral (length xs)<br />> in POrd 0 $ fromList $ map (flip (,) p) xs<br /><p><br /><br /></p></pre>And the Bernoulli distribution, i.e. tossing a <tt>Bool</tt> coin that comes up <tt>True</tt> with probability <img src="https://chart.googleapis.com/chart?cht=tx&chl=p" style="vertical-align: middle;" />: <p><br /></p><pre><br />> bernoulliP :: (Fractional p, Ord p) =><br />> p -> P p Bool<br />> bernoulliP p = POrd 0 $ fromList $ [(False, 1-p), (True, p)]<br /><p><br /><br /></p></pre>Now we can try a random walk in one dimension. At each step we have a 50/50 chance of standing still or taking a step to the right: <p><br /></p><pre><br />> random_walk1 :: Int -> P Double Int<br />> random_walk1 0 = returnP 0<br />> random_walk1 n = do<br />> a <- random_walk1 (n-1)<br />> b <- chooseP [0, 1]<br />> returnP $ a+b<br /><p><br /><br /></p></pre>Below in <tt>main</tt> we take 2048 steps but only track 512 probabilities. The tail probability in this case is about <img src="https://chart.googleapis.com/chart?cht=tx&chl=10%5E%7B-29%7D" style="vertical-align: middle;" />. So only tracking 1/4 of the outcomes has had almost no impact on the numbers. This also illustrates why it is good to track the tail probabilities rather than inferring them from the missing probabilities in the bulk of the PDF - they can be so small they vanish compared to floating poimnt errors. We can afford to track a lot fewer than 512 (out of 2049 possible) outcomes and still have a good representative PDF. <blockquote><pre><br /><a href="https://2.bp.blogspot.com/-5g2q61iC3nw/Wd-XI9KZq9I/AAAAAAAA2Ao/6Sva9yVlIcsO4ZoETHA2MWvbc_nfxNaggCLcBGAs/s1600/figure_1.png"><img src="https://2.bp.blogspot.com/-5g2q61iC3nw/Wd-XI9KZq9I/AAAAAAAA2Ao/6Sva9yVlIcsO4ZoETHA2MWvbc_nfxNaggCLcBGAs/s400/figure_1.png" height="300" border="0" width="400" /></a><br /></pre></blockquote>Now here's a two-dimensional random walk for 32 steps. The tail probability is about <img src="https://chart.googleapis.com/chart?cht=tx&chl=2%5Ctimes%2010%5E%7B-5%7D" style="vertical-align: middle;" /> so we are getting a reasonably representative PDF. We have to run fewer steps than before, however, because the space of possible outcomes spans two dimensions, meaning that reduction doesn't help as much as it does in one dimension. <p><br /></p><pre><br />> random_walk2 :: Int -> (Int, Int) -> P Double (Int, Int)<br />> random_walk2 0 (x, y) = returnP (x, y)<br />> random_walk2 n (x, y) = do<br />> (x',y') <- random_walk2 (n-1) (x, y)<br />> dx <- chooseP [-1, 1]<br />> dy <- chooseP [-1, 1]<br />> returnP (x'+dx, y'+dy)<br /><p><br /><br /></p></pre>One last simulation. This is a queing scenario. Tasks come in once every tick of the clock. There are four queues a task can be assigned to. A task is assigned to the shortest queue. Meanwhile each queue as a 1/4 probability of clearing one item at each tick of the clock. We build the PDF for the maximum length any queue has at any time. <blockquote><pre><br /><a href="https://1.bp.blogspot.com/-FOEzOGZKNuA/Wd-kGpIp_II/AAAAAAAA2BE/O494hiM3KhUXdv0OUL45itF3TaU9eKNfwCLcBGAs/s1600/Untitled.png"><img src="https://1.bp.blogspot.com/-FOEzOGZKNuA/Wd-kGpIp_II/AAAAAAAA2BE/O494hiM3KhUXdv0OUL45itF3TaU9eKNfwCLcBGAs/s400/Untitled.png" height="276" border="0" width="400" /></a><br /></pre></blockquote>The first argument to <tt>queue</tt> is the number of ticks of the clock. The second argument is the list of lengths of the queues. It returns a PDF, not just on the current queue size, but also on the longest queue it has seen. <p><br /></p><pre><br />> queue :: Int -> [Int] -> P Double (Int, [Int])<br />> queue 0 ls = returnP (maximum ls, ls)<br />> queue n ls = do<br />> (longest, ls1) <- queue (n-1) ls<br />> ls2 <- forM ls1 $ \l -> do<br />> served <- bernoulliP (1/4)<br />> returnP $ if served && l > 0 then l-1 else l<br />> let ls3 = L.sort $ head ls2+1 : tail ls2<br />> returnP (longest `max` maximum ls3, ls3)<br /><p><br /><br /></p></pre>For the queing simulation the tail probability is around <img src="https://chart.googleapis.com/chart?cht=tx&chl=5%5Ctimes%2010%5E%7B-4%7D" style="vertical-align: middle;" /> despite the fact that we have discarded a vast possible set of possible outcomes. <blockquote><pre><br /><a href="https://1.bp.blogspot.com/-MomMNzoE4co/Wd-gtJI4kKI/AAAAAAAA2A4/RkRPJJ_RmHI6LLJVl-Wsh7DGiPQ1fwblwCLcBGAs/s1600/figure_2.png"><img src="https://1.bp.blogspot.com/-MomMNzoE4co/Wd-gtJI4kKI/AAAAAAAA2A4/RkRPJJ_RmHI6LLJVl-Wsh7DGiPQ1fwblwCLcBGAs/s400/figure_2.png" height="300" border="0" width="400" /></a><br /></pre></blockquote>It's a little ugly that <tt>trimSize</tt> is a global constant: <p><br /></p><pre><br />> trimSize = 512<br /><p><br /><br /></p></pre>The correct solution is probably to separate the probability "syntax" from its "semantics". In other words, we should implement a free monad supporting the language of probability with suitable constructors for <tt>bernoulliP</tt> and <tt>choiceP</tt>. We can then write a separate interpreter which takes a <tt>trimSize</tt> as argument. This has another advantage too: the <tt>Monad</tt> above isn't a true monad. It uses a greedy approach to discarding probabilities and different rearrangements of the code, that ought to give identical results, may end up diferent. By using a free monad we ensure that our interface is a true monad and we can put the part of the code that breaks the monad laws into the interpreter. The catch is that my first attempt at writing a free monad resulted in code with poor performance. So I'll leave an efficient version as an exercise :-) <p><br /></p><pre><br />> main = do<br />> print $ runP $ random_walk1 2048<br />> print $ runP $ random_walk2 32 (0, 0)<br />> print $ runP $ do<br />> (r, _) <- queue 128 [0, 0, 0, 0]<br />> returnP r<br /></pre>Sat, 14 Oct 2017 20:42:00 +0000noreply@blogger.com (Dan Piponi)Functional Jobs: Analytics Developer for a mobile app at EmployeeChannel, Inc. (Full-time)urn:uuid:09ab074c-83d1-1bc6-b401-d4ef948a3bb4
https://functionaljobs.com/jobs/9044-analytics-developer-for-a-mobile-app-at-employeechannel-inc
<p>Who we are</p>
<p>EmployeeChannel is a leading provider of award-winning mobile apps for employee engagement and communication, enabling HR and Internal Communications teams to boost the impact and effectiveness of employee communication, to create a positive employee experience, and to drive cultural and business outcomes. The EmployeeChannel app extends the knowledge and reach of organizational experts to employees anytime, anywhere and is dedicated to the interactions between an organization and its employees. </p>
<p>Employees can find and receive organizational information easier and faster, get personalized responses to requests, and react quickly to time-sensitive events. HR and Internal Communications teams can respond real-time to organizational imperatives and employee needs using behavioral insights from Voice of the Employee analytics. To learn more about the EmployeeChannel app and how it can be used to engage and communicate with employees, please visit www.employeechannelinc.com. </p>
<p>What we are looking for</p>
<p>We are an early stage start-up in downtown San Francisco with great product, a solid business model, and tremendous upside. Fully funded and about to scale – a great time to join! </p>
<p>We are searching for talented and motivated individuals who like a good challenge, enjoy teamwork, and are passionate about building something of real value. You’ll be joining an experienced team that knows how to execute to a market-defining strategy. We work hard, because we enjoy what we are doing and the people we work with. We are confident that EmployeeChannel is the team to beat in what is panning out as a huge opportunity. The person we're looking for is first and foremost a developer, who understands and does functional programming, with Haskell experience, and enough math background to understand statistics and analytics. Sound like what you’re looking for? Read on…</p>
<p>What you will do</p>
<p>As a member of our Product team, you will contribute to both leading edge research and rapid development in our delivery of high-quality, engaging, SaaS-based products. Machine learning, NLP, and analytics drive a significant portion of our product’s value proposition. You will engage with our highly experienced research team to explore and model state-of-the-art use of these frameworks to provide unique insights for our customers. </p>
<p>Your work with other developers can have a big impact, heavily influencing our approach to delivering high quality products. We are committed to delivering robust, scalable products that bring a consumer best-in-class feel to the Enterprise market. </p>
<p>Here¹s a sample of what that would look like:</p>
<p>• You will work with data scientists and software engineers to implement analytics and machine learning algorithms in production-quality Haskell code and deploy them to production servers. You will learn a ton and be challenged regularly in this role.
• You will devise tests of integrity and consistency for both data and analytics results.
• You¹ll work in an Agile environment with a team whose track record of meeting or exceeding their commitments is strong.
• You¹ll enjoy partnering with other developers, knowing that collaborative design sessions and the occasional pair programming are part of how we operate.
• We deliver quality code, because we are committed to test-driven development and we take pride in our work.
• 1 + 1 = 3 here. We look to leverage open source wherever it makes sense, knowing that we can move faster as a result. Speed matters.
• You won¹t feel pigeon-holed here. Working together with Product Managers and our Hosted Ops counterparts is a daily occurrence.
• You¹ll have a measurable impact on the success of our product and company. If you¹re looking for a place where you can get your fingerprints all over the product, this is a good place to call home.</p>
<p>Your qualifications</p>
<p>• The ideal candidate we're looking for is first and foremost a developer, who understands and does functional programming, with Haskell experience, and enough math background to understand statistics and analytics. And above all, our ideal candidate loves functional programming.
• 2-3 years of relevant work experience is required.
• You have experience with, and appreciation for, functional programming and where it can be advantageously applied.
• You have produced production-quality code for a high scale commercial application.
• You have used Haskell in projects. You know your way around GHCi, stack, and cabal.
• You have designed database schema and worked with databases like Postgres.
• You have built the back-end of a web service.
• While developing, you formally test your code with unit-test packages such as HSpec.
• You are no stranger to mathematics, statistics, and data analysis. If you encountered this with Python, we will not hold that against you.
• You have worked as part of an Agile development team.
• Git, Apache, and HSpec are all used in our shop.
• A BS in a Mathematics-intensive degree (Comp Sci, Math, Physics) or equivalent is required.</p>
<p>Get information on <a href="https://functionaljobs.com/jobs/9044-analytics-developer-for-a-mobile-app-at-employeechannel-inc">how to apply</a> for this position.</p>Thu, 12 Oct 2017 20:48:19 +0000Joachim Breitner: Isabelle functions: Always total, sometimes undefinedhttp://www.joachim-breitner.de/blog/732-Isabelle_functions__Always_total%2C_sometimes_undefined
http://www.joachim-breitner.de/blog/732-Isabelle_functions__Always_total%2C_sometimes_undefined
<p>Often, when I mention how things work in the interactive theorem prover [Isabelle/HOL] (in the following just “Isabelle”<a href="English_feed.rss.en#fn1" id="fnref1" class="footnoteRef"><sup>1</sup></a>) to people with a strong background in functional programming (whether that means Haskell or Coq or something else), I cause confusion, especially around the issue of <em>what is a function</em>, are function <em>total</em> and what is the business with <em>undefined</em>. In this blog post, I want to explain some these issues, aimed at functional programmers or type theoreticians.</p>
<p>Note that this is not meant to be a tutorial; I will not explain how to do these things, and will focus on what they mean.</p>
<h3 id="hol-is-a-logic-of-total-functions">HOL is a logic of total functions</h3>
<p>If I have a Isabelle function <code>f :: a ⇒ b</code> between two types <code>a</code> and <code>b</code> (the function arrow in Isabelle is <code>⇒</code>, not <code>→</code>), then – by definition of what it means to be a function in HOL – whenever I have a value <code>x :: a</code>, then the expression <code>f x</code> (i.e. <code>f</code> applied to <code>x</code>) <em>is</em> a value of type <code>b</code>. Therefore, and without exception, <em>every Isabelle function is total</em>.</p>
<p>In particular, it cannot be that <code>f x</code> does not exist for some <code>x :: a</code>. This is a first difference from Haskell, which does have partial functions like</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">spin ::</span> <span class="dt">Maybe</span> <span class="dt">Integer</span> <span class="ot">-></span> <span class="dt">Bool</span>
spin (<span class="dt">Just</span> n) <span class="fu">=</span> spin (<span class="dt">Just</span> (n<span class="fu">+</span><span class="dv">1</span>))</code></pre>
<p>Here, neither the expression <code>spin Nothing</code> nor the expression <code>spin (Just 42)</code> produce a value of type <code>Bool</code>: The former raises an exception (“incomplete pattern match”), the latter does not terminate. Confusingly, though, both expressions have type <code>Bool</code>.</p>
<p>Because every function is total, this confusion cannot arise in Isabelle: If an expression <code>e</code> has type <code>t</code>, then it <em>is</em> a value of type <code>t</code>. This trait is shared with other total systems, including Coq.</p>
<p>Did you notice the emphasis I put on the word “is” here, and how I deliberately did not write “evaluates to” or “returns”? This is because of another big source for confusion:</p>
<h3 id="isabelle-functions-do-not-compute">Isabelle functions do not compute</h3>
<p>We (i.e., functional programmers) stole the word “function” from mathematics and repurposed it<a href="English_feed.rss.en#fn2" id="fnref2" class="footnoteRef"><sup>2</sup></a>. But the word “function”, in the context of Isabelle, refers to the mathematical concept of a function, and it helps to keep that in mind.</p>
<p>What is the difference?</p>
<ul>
<li>A function <code>a → b</code> in functional programming is an algorithm that, given a value of type <code>a</code>, calculates (returns, evaluates to) a value of type <code>b</code>.</li>
<li>A function <code>a ⇒ b</code> in math (or Isabelle) associates with each value of type <code>a</code> a value of type <code>b</code>.</li>
</ul>
<p>For example, the following is a perfectly valid function definition in math (and HOL), but could not be a function in the programming sense:</p>
<pre class="isabelle"><code>definition foo :: "(nat ⇒ real) ⇒ real" where
"foo seq = (if convergent seq then lim seq else 0)"</code></pre>
<p>This assigns a real number to every sequence, but it does not <em>compute</em> it in any useful sense.</p>
<p>From this it follows that</p>
<h3 id="isabelle-functions-are-specified-not-defined">Isabelle functions are specified, not defined</h3>
<p>Consider this function definition:</p>
<pre class="isabelle"><code>fun plus :: "nat ⇒ nat ⇒ nat" where
"plus 0 m = m"
| "plus (Suc n) m = Suc (plus n m)"</code></pre>
<p>To a functional programmer, this reads</p>
<blockquote>
<p><code>plus</code> is a function that analyses its first argument. If that is <code>0</code>, then it returns the second argument. Otherwise, it calls itself with the predecessor of the first argument and increases the result by one.</p>
</blockquote>
<p>which is clearly a description of a computation.</p>
<p>But to Isabelle, the above reads</p>
<blockquote>
<p><code>plus</code> is a binary function on natural numbers, and it satisfies the following two equations: …</p>
</blockquote>
<p>And in fact, it is not so much Isabelle that reads it this way, but rather the <code>fun</code> command, which is external to the Isabelle logic. The <code>fun</code> command analyses the given equations, constructs a non-recursive definition of <code>plus</code> under the hood, passes <em>that</em> to Isabelle and then proves that the given equations hold for <code>plus</code>.</p>
<p>One interesting consequence of this is that different specifications can lead to the same functions. In fact, if we would define <code>plus'</code> by recursing on the second argument, we’d obtain the the same function (i.e. <code>plus = plus'</code> is a theorem, and there would be no way of telling the two apart).</p>
<h3 id="termination-is-a-property-of-specifications-not-functions">Termination is a property of specifications, not functions</h3>
<p>Because a function does not evaluate, it does not make sense to ask if it terminates. The question of termination arises <em>before</em> the function is defined: The <code>fun</code> command can only construct <code>plus</code> in a way that the equations hold if it passes a termination check – very much like <code>Fixpoint</code> in Coq.</p>
<p>But while the termination check of <code>Fixpoint</code> in Coq is a deep part of the basic logic, in Isabelle it is simply something that this particular command requires for its internal machinery to go through. At no point does a “termination proof of the function” exist as a theorem inside the logic. And other commands may have other means of defining a function that do not even require such a termination argument!</p>
<p>For example, a function specification that is tail-recursive can be turned in to a function, even without a termination proof: The following definition describes a higher-order function that iterates its first argument <code>f</code> on the second argument <code>x</code> until it finds a fixpoint. It is completely polymorphic (the single quote in <code>'a</code> indicates that this is a type variable):</p>
<pre class="isabelle"><code>partial_function (tailrec)
fixpoint :: "('a ⇒ 'a) ⇒ 'a ⇒ 'a"
where
"fixpoint f x = (if f x = x then x else fixpoint f (f x))"</code></pre>
<p>We can work with this definition just fine. For example, if we instantiate <code>f</code> with <code>(λx. x-1)</code>, we can prove that it will always return 0:</p>
<pre class="isabelle"><code>lemma "fixpoint (λ n . n - 1) (n::nat) = 0"
by (induction n) (auto simp add: fixpoint.simps)</code></pre>
<p>Similarly, if we have a function that works within the option monad (i.e. |Maybe| in Haskell), its specification can always be turned into a function without an explicit termination proof – here one that calculates the Collatz sequence:</p>
<pre class="isabelle"><code>partial_function (option) collatz :: "nat ⇒ nat list option"
where "collatz n =
(if n = 1 then Some [n]
else if even n
then do { ns <- collatz (n div 2); Some (n # ns) }
else do { ns <- collatz (3 * n + 1); Some (n # ns)})"</code></pre>
<p>Note that lists in Isabelle are finite (like in Coq, unlike in Haskell), so this function “returns” a list only if the collatz sequence eventually reaches 1.</p>
<p>I expect these definitions to make a Coq user very uneasy. How can <code>fixpoint</code> be a total function? What is <code>fixpoint (λn. n+1)</code>? What if we run <code>collatz n</code> for a <code>n</code> where the <a href="https://en.wikipedia.org/wiki/Collatz_conjecture">Collatz sequence</a> does <em>not</em> reach 1?<a href="English_feed.rss.en#fn3" id="fnref3" class="footnoteRef"><sup>3</sup></a> We will come back to that question after a little detour…</p>
<h3 id="hol-is-a-logic-of-non-empty-types">HOL is a logic of non-empty types</h3>
<p>Another big difference between Isabelle and Coq is that in Isabelle, <em>every type is inhabited</em>. Just like the totality of functions, this is a very fundamental fact about what HOL defines to be a type.</p>
<p>Isabelle gets away with that design because in Isabelle, we do not use types for propositions (like we do in Coq), so we do not need empty types to denote false propositions.</p>
<p>This design has an important consequence: It allows the existence of a polymorphic expression that inhabits any type, namely</p>
<pre class="isabelle"><code>undefined :: 'a</code></pre>
<p>The naming of this term alone has caused a great deal of confusion for Isabelle beginners, or in communication with users of different systems, so I implore you to not read too much into the name. In fact, you will have a better time if you think of it as <code>arbitrary</code> or, even better, <code>unknown</code>.</p>
<p>Since <code>undefined</code> can be instantiated at any type, we can instantiate it for example at <code>bool</code>, and we can observe an important fact: <code>undefined</code> is not an <em>extra</em> value besides the “usual ones”. It is simply <em>some</em> value of that type, which is demonstrated in the following lemma:</p>
<pre class="isabelle"><code>lemma "undefined = True ∨ undefined = False" by auto</code></pre>
<p>In fact, if the type has only one value (such as the unit type), then we know the value of <code>undefined</code> for sure:</p>
<pre class="isabelle"><code>lemma "undefined = ()" by auto</code></pre>
<p>It is very handy to be able to produce an expression of any type, as we will see as follows</p>
<h3 id="partial-functions-are-just-underspecified-functions">Partial functions are just underspecified functions</h3>
<p>For example, it allows us to translate incomplete function specifications. Consider this definition, Isabelle’s equivalent of Haskell’s partial <code>fromJust</code> function:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell">fun fromSome :: <span class="st">"'a option ⇒ 'a"</span> <span class="kw">where</span>
<span class="st">"fromSome (Some x) = x"</span></code></pre>
<p>This definition is accepted by <code>fun</code> (albeit with a warning), and the generated function <code>fromSome</code> behaves exactly as specified: when applied to <code>Some x</code>, it is <code>x</code>. The term <code>fromSome None</code> is also a value of type <code>'a</code>, we just do not know which one it is, as the specification does not address that.</p>
<p>So <code>fromSome None</code> behaves just like <code>undefined</code> above, i.e. we can prove</p>
<pre class="isabelle"><code>lemma "fromSome None = False ∨ fromSome None = True" by auto</code></pre>
<p>Here is a small exercise for you: Can you come up with an explanation for the following lemma:</p>
<pre class="isabelle"><code>fun constOrId :: "bool ⇒ bool" where
"constOrId True = True"
lemma "constOrId = (λ_.True) ∨ constOrId = (λx. x)"
by (metis (full_types) constOrId.simps)</code></pre>
<p>Overall, this behavior makes sense if we remember that function “definitions” in Isabelle are not really definitions, but rather specifications. And a partial function “definition” is simply a underspecification. The resulting function is simply any function hat fulfills the specification, and the two lemmas above underline that observation.</p>
<h3 id="nonterminating-functions-are-also-just-underspecified">Nonterminating functions are also just underspecified</h3>
<p>Let us return to the puzzle posed by <code>fixpoint</code> above. Clearly, the function – seen as a functional program – is not total: When passed the argument <code>(λn. n + 1)</code> or <code>(λb. ¬b)</code> it will loop forever trying to find a fixed point.</p>
<p>But Isabelle functions are not functional programs, and the definitions are just specifications. What does the specification say about the case when <code>f</code> has no fixed-point? It states that the equation <code>fixpoint f x = fixpoint f (f x)</code> holds. And this equation has a solution, for example <code>fixpoint f _ = undefined</code>.</p>
<p>Or more concretely: The specification of the <code>fixpoint</code> function states that <code>fixpoint (λb. ¬b) True = fixpoint (λb. ¬b) False</code> has to hold, but it does not specify which particular value (<code>True</code> or <code>False</code>) it should denote – any is fine.</p>
<h3 id="not-all-function-specifications-are-ok">Not all function specifications are ok</h3>
<p>At this point you might wonder: Can I just specify any equations for a function <code>f</code> and get a function out of that? But rest assured: That is not the case. For example, no Isabelle command allows you define a function <code>bogus :: () ⇒ nat</code> with the equation <code>bogus () = Suc (bogus ())</code>, because this equation does not have a solution.</p>
<p>We can actually prove that such a function cannot exist:</p>
<pre class="isabelle"><code>lemma no_bogus: "∄ bogus. bogus () = Suc (bogus ())" by simp</code></pre>
<p>(Of course, <code>not_bogus () = not_bogus ()</code> is just fine…)</p>
<h3 id="you-cannot-reason-about-partiality-in-isabelle">You cannot reason about partiality in Isabelle</h3>
<p>We have seen that there are many ways to define functions that one might consider “partial”. Given a function, can we prove that it is not “partial” in that sense?</p>
<p>Unfortunately, but unavoidably, no: Since <code>undefined</code> is not a separate, recognizable value, but rather simply an unknown one, there is no way of stating that “A function result is not specified”.</p>
<p>Here is an example that demonstrates this: Two “partial” functions (one with not all cases specified, the other one with a self-referential specification) are indistinguishable from the total variant:</p>
<pre class="isabelle"><code>fun partial1 :: "bool ⇒ unit" where
"partial1 True = ()"
partial_function (tailrec) partial2 :: "bool ⇒ unit" where
"partial2 b = partial2 b"
fun total :: "bool ⇒ unit" where
"total True = ()"
| "total False = ()"
lemma "partial1 = total ∧ partial2 = total" by auto</code></pre>
<p>If you really <em>do</em> want to reason about partiality of functional programs in Isabelle, you should consider implementing them not as plain HOL functions, but rather use <a href="http://isabelle.in.tum.de/dist/library/HOL/HOLCF/README.html">HOLCF</a>, where you can give equational specifications of functional programs and obtain <em>continuous functions</em> between <em>domains</em>. In that setting, <code>⊥ ≠ ()</code> and <code>partial2 = ⊥ ≠ total</code>. We have done that <a href="https://arxiv.org/abs/1306.1340">to verify some of HLint’s equations</a>.</p>
<h3 id="you-can-still-compute-with-isabelle-functions">You can still compute with Isabelle functions</h3>
<p>I hope by this point, I have not scared away anyone who wants to use Isabelle for functional programming, and in fact, you can use it for that. If the equations that you pass to `fun are a reasonable definition for a function (in the programming sense), then these equations, used as rewriting rules, will allow you to “compute” that function quite like you would in Coq or Haskell.</p>
<p>Moreover, Isabelle supports code extraction: You can take the equations of your Isabelle functions and have them expored into Ocaml, Haskell, Scala or Standard ML. See <a href="http://www21.in.tum.de/~popescua/rs3/CoCon.html">Concon</a> for a conference management system with confidentially verified in Isabelle.</p>
<p>While these usually are the equations you defined the function with, they don't have to: You can declare other proved equations to be used for code extraction, e.g. to refine your elegant definitions to performant ones.</p>
<p>Like with code extraction from Coq to, say, Haskell, the adequacy of the translations rests on a <a href="http://www.cse.chalmers.se/~nad/publications/danielsson-et-al-popl2006.html">“moral reasoning” foundation</a>. Unlike extraction from Coq, where you have an (unformalized) guarantee that the resulting Haskell code is terminating, you do not get that guarantee from Isabelle. Conversely, this allows you do reason about and extract non-terminating programs, like <code>fixpoint</code>, which is not possible in Coq.</p>
<p>There is <a href="https://www21.in.tum.de/~hupel/pub/isabelle-cakeml.pdf">currently ongoing work</a> about verified code generation, where the code equations are reflected into a deep embedding of HOL in Isabelle that would allow explicit termination proofs.</p>
<h3 id="conclusion">Conclusion</h3>
<p>We have seen how in Isabelle, <em>every function is total</em>. Function declarations have equations, but these do not <em>define</em> the function in an computational sense, but rather <em>specify</em> them. Because in HOL, there are no empty types, many specifications that appear partial (incomplete patterns, non-terminating recursion) have solutions in the space of total functions. Partiality in the specification is no longer visible in the final product.</p>
<h3 id="ps-axiom-undefined-in-coq">PS: Axiom <code>undefined</code> in Coq</h3>
<p><em>This section is speculative, and an invitation for discussion.</em></p>
<p>Coq already distinguishes between types used in programs (<code>Set</code>) and types used in proofs <code>Prop</code>.</p>
<p>Could Coq ensure that every <code>t : Set</code> is non-empty? I imagine this would require additional checks in the <code>Inductive</code> command, similar to the checks that the Isabelle command <code>datatype</code> has to perform<a href="English_feed.rss.en#fn4" id="fnref4" class="footnoteRef"><sup>4</sup></a>, and it would disallow <a href="https://coq.inria.fr/library/Coq.Init.Datatypes.html"><code>Empty_set</code></a>.</p>
<p>If so, then it would be sound to add the following axiom</p>
<pre class="coq"><code>Axiom undefined : forall (a : Set), a.</code></pre>
<p>wouldn't it? This axiom does not have any computational meaning, but that seems to be ok for optional Coq axioms, like classical reasoning or function extensionality.</p>
<p>With this in place, how much of what I describe above about function definitions in Isabelle could now be done soundly in Coq. Certainly pattern matches would not have to be complete and could sport an implicit case <code>_ ⇒ undefined</code>. Would it “help” with non-obviously terminating functions? Would it allow a Coq command <code>Tailrecursive</code> that accepts any tailrecursive function without a termination check?</p>
<div class="footnotes">
<hr />
<ol>
<li id="fn1"><p>Isabelle is a metalogical framework, and other logics, e.g. Isabelle/ZF, behave differently. For the purpose of this blog post, I always mean Isabelle/HOL.<a href="English_feed.rss.en#fnref1">↩</a></p></li>
<li id="fn2"><p>Isabelle is a metalogical framework, and other logics, e.g. Isabelle/ZF, behave differently. For the purpose of this blog post, I always mean Isabelle/HOL.<a href="English_feed.rss.en#fnref2">↩</a></p></li>
<li id="fn3"><p>Let me know if you find such an <span class="math"><em>n</em></span>. Besides <span class="math"><em>n</em> = 0</span>.<a href="English_feed.rss.en#fnref3">↩</a></p></li>
<li id="fn4"><p>Like <code>fun</code>, the constructions by <code>datatype</code> are not part of the logic, but create a type definition from more primitive notions that is isomorphic to the specified data type.<a href="English_feed.rss.en#fnref4">↩</a></p></li>
</ol>
</div>Thu, 12 Oct 2017 17:54:20 +0000mail@joachim-breitner.de (Joachim Breitner)Tweag I/O: Array fusion with vectorhttp://www.tweag.io/posts/2017-10-12-vector-package.html
http://www.tweag.io/posts/2017-10-12-vector-package.html
<div>Manuel M T Chakravarty</div><p><em>This is the fourth post in a series about array programming in Haskell — you might be interested in the <a href="http://www.tweag.io/posts/2017-08-09-array-programming-in-haskell.html">first</a>, <a href="http://www.tweag.io/posts/2017-08-31-hmatrix.html">second</a>, and <a href="http://www.tweag.io/posts/2017-09-27-array-package.html">third</a>, too.</em></p>
<p>In the <a href="http://www.tweag.io/posts/2017-09-27-array-package.html">previous post</a> of this series, we explored the basic, down-to-earth, index-based array interfaces that have their roots in the Haskell standard libraries. We also discussed the need for <em>strictness</em>, <em>unboxing</em>, and a <em>two-phase</em> initialisation strategy, freezing mutable structures after they have been populated. With those basics out of the way, we can now explore the design of higher-level, <em>collection-oriented</em> array interfaces whose efficient implementation critically relies on those basic techniques.</p>
<p>The <a href="https://hackage.haskell.org/package/vector">vector</a> package is probably the most popular array library in Haskell, and it is a prime example of this approach. The package, implemented by Roman Leshchinskiy, is a spin off from the <a href="https://wiki.haskell.org/GHC/Data_Parallel_Haskell">Data Parallel Haskell</a> project . Specifically, Data Parallel Haskell is organised as multiple layers of array libraries, and <a href="https://hackage.haskell.org/package/vector">vector</a> is a generalisation of what used to be the lowest layer: sequential, int-indexed arrays combined with a powerful array-fusion framework that makes the composition of successive collective operations efficient.</p>
<h2>Collective</h2>
<p>Collective operations that process all elements of an array in a single operation, such as <code>map</code>, <code>fold</code>, <code>zip</code>, <code>filter</code>, and so forth, fit more naturally into a functional style than the index-based loops that originated in FORTRAN. This is not just to favour combinator-based code, but also because index-based, one-element-at-a-time processing quickly leads to the use of mutable arrays. Indexed-based operations are sometimes inevitable, but they ought to be used sparingly; often we only need to extract individual or only a few elements of an array. In cases, where random data-driven access is required, collective index-based operations often suffice — such as, for example, <code>backpermute</code>:</p>
<pre><code>backpermute (fromList ['a', 'b', 'c', 'd']) (fromList [0, 3, 2, 3, 1, 0])
= fromList ['a', 'd', 'c', 'd', 'b', 'a']
</code></pre>
<p>The function <code>backpermute</code> returns a result of the same length as its second argument, where the integer values of the second argument are used to determine the index positions of the elements to pick from the first argument. (The function <code>fromList</code> turns a list into a <code>Vector</code>.)</p>
<h2>Array fusion</h2>
<p>Consider the following pipeline of collective vector operations, where we assume <code>import Data.Vector as V</code> to disambiguate <code>vector</code> operations from the standard list operations of the same name:</p>
<pre><code> V.map fst
$ V.filter (\t -> snd t > 0)
$ V.zip points cs
</code></pre>
<p>First, we combine a vector of <code>points :: Vector (Double, Double)</code> with a vector <code>cs :: Vector Double</code> by way of <code>V.zip</code> to obtain a value of type <code>Vector ((Double, Double), Double)</code>. Then, we <code>V.filter</code> to only retain those <code>((Double, Double), Double)</code> triples whose last component is strictly positive. Finally, we project the points out of the filtered vector. Overall, this pipeline drops all <code>points</code> whose corresponding (index-wise) <code>cs</code> value is zero or negative.</p>
<p>While this is a compact functional specification, its naive implementation would be rather inefficient. Each of the three collective operations creates a new immutable vector. All this, only to finally get a vector containing a subset of the elements of <code>points</code>. The main problem with this is not that it is wasteful to allocate those intermediate arrays, but that initialising and then consuming those intermediates generates a lot of memory traffic — this is rather unfortunate as memory bandwidth is the most severe limiting factor of current hardware.</p>
<p>Hence, it is absolutely crucial that a library, such as <code>vector</code>, performs <em>array fusion</em>, a compiler optimisation that eliminates intermediate arrays and combines multiple successive array traversals into one. In fact, the <code>vector</code> package, compiles the above code consisting of <code>V.map</code>, <code>V.filter</code>, and <code>V.zip</code> into a single traversal without any intermediates. It achieves this using a fusion framework known as <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.421.8551">stream fusion</a>.</p>
<h2>From generic to concrete</h2>
<p>The <code>vector</code> package is architected in a flexible manner with generic type classes <code>Vector</code> (for immutable vectors) and <code>MVector</code> (for mutable vectors), defined in <code>Data.Vector.Generic</code>. <code>Vector</code> is a subclass of <code>MVector</code> , as the latter provides mutating methods on top of the pure ones.</p>
<p>The modules <code>Data.Vector</code> and <code>Data.Vector.Unboxed</code> instantiate the generic framework with concrete boxed and unboxed vector types — both again providing an immutable and a mutable flavour. In addition, there is variant of unboxed vectors, called <em>storable</em>, in <code>Data.Vector.Storable</code>, which ties in with the <code>Storable</code> class of the foreign function interface and is useful for interoperability with C code.</p>
<p>Due to this generic set up, array algorithms can be implemented generically and, then, used with all of the concrete instances. It is also perfectly possible to extend <code>vector</code> with your own flavour of arrays; for example, for interoperability with another array library.</p>
<h2>Convex hull</h2>
<p>One of the algorithms included in the benchmarks that are part of <code>vector</code> is the <a href="https://en.wikipedia.org/wiki/Quickhull">quickhull algorithm</a> for computing the convex hull of a set of points. Quickhull is —just like quicksort— a split-based divide-and-conquer algorithm. It begins by cutting the set of points into two by drawing a line between the leftmost and rightmost point in the set. Then, it recursively operates on these two sets of points. In each recursive invocation, the point furthest from the last dividing line gives rise to two new lines (using the two end points of that last dividing line). This is illustrated by the following animation from <a href="https://commons.wikimedia.org/wiki/File%3AAnimation_depicting_the_quickhull_algorithm.gif">Wikimedia Commons</a>.</p>
<center>
<img src="https://upload.wikimedia.org/wikipedia/commons/4/42/Animation_depicting_the_quickhull_algorithm.gif" alt="Animation depicting the quickhull algorithm" style="" />
</center>
<p>Given a <code>Vector</code> of x/y-coordinates, we can implement the initial step of quickhull, namely determining the left and rightmost point, as follows:</p>
<pre><code>import Data.Vector.Unboxed as V
quickhull :: Vector (Double, Double) -> Vector (Double, Double)
quickhull points = hsplit points pmin pmax V.++ hsplit points pmax pmin
where
xs = V.map fst points
pmin = points V.! V.minIndex xs
pmax = points V.! V.maxIndex xs
</code></pre>
<p>The functions <code>V.minIndex</code> and <code>V.maxIndex</code> return the index position of the smallest and largest element of the argument vector, respectively (here, of the x-coordinates <code>xs</code> of all points). We use those to extract the leftmost (<code>pmin</code>) and rightmost (<code>pmax</code>) point. When invoking the recursive component of quickhull —here, implemented by <code>hsplit</code>— we pass the points and the dividing line in the form of its starting and end point. As the latter are swapped in the two recursive calls, one call processes the points above the dividing line and the other call processes the points below that line.</p>
<p>The function <code>hsplit</code>, first, computes the distance <code>cs</code> of all <code>points</code> to the line from <code>p1</code> to <code>p2</code>. Second, it uses the pipeline of collective operations that we discussed above to remove all <code>points</code> that are below that line. The remaining points, <code>packed</code>, are passed into the two recursive calls together with <code>pm</code>, the point that is furthest from the line.</p>
<pre><code>hsplit :: (Ord b, Num b, Unbox b)
=> Vector (b, b) -> (b, b) -> (b, b) -> Vector (b, b)
hsplit points p1 p2
| V.length packed < 2 = p1 `V.cons` packed
| otherwise = hsplit packed p1 pm V.++ hsplit packed pm p2
where
cs = V.map (\p -> cross p p1 p2) points
packed = V.map fst
$ V.filter (\t -> snd t > 0)
$ V.zip points cs
pm = points V.! V.maxIndex cs
-- cross p p1 p2 = distance of p from line between p1 and p2
cross :: Num a => (a, a) -> (a, a) -> (a, a) -> a
cross (x,y) (x1,y1) (x2,y2) = (x1-x)*(y2-y) - (y1-y)*(x2-x)
</code></pre>
<p>The function <code>cross</code> that we use to compute the distance of a point <code>p</code> from the line between points <code>p1</code> and <code>p2</code> returns a signed distance; i.e., a positive distance value is for a point above the line and a negative distance value for a point below the line. This is crucial for <code>V.filter (\t -> snd t > 0)</code> to remove all points located on or below the line.</p>
<h2>Fusion interrupted</h2>
<p>The <code>vector</code> package optimises the quickhull code rather well and gets within about a factor of two of the performance of hand-optimised C code. However, there are two kinds of obstacles that may prevent stream fusion and other fusion systems from optimising code as expected: (1) the interaction with other compiler optimisations and (2) the strength of the fusion system.</p>
<p>Concerning Point (1), the interaction of multiple compiler optimisations and finding the right order in which to apply them are a topic for PhD theses as well as a black art of skilled engineers. In the Glasgow Haskell Compiler (GHC), it is especially <em>inlining</em>, <em>let floating</em>, <em>strictness analysis</em>, and various forms of <em>rewriting</em> and <em>specialisation</em> that can have a huge impact on the effectiveness of fusion. The details are beyond the scope of this post, but some of it is covered in the rather extensive <a href="https://www.stackbuilders.com/tutorials/haskell/ghc-optimization-and-fusion/">GHC optimization and fusion</a>.</p>
<p>Concerning Point (2), practical fusion systems generally cannot fuse all fusable programs. In other words, given a particular fusion system, it will spot and optimise certain types of computations that can be fused. However, there are other types of computations that can also be fused, but this particular fusion system will not be able to recognise and exploit those opportunities.</p>
<p>In fact, the algorithmic structure of quickhull does expose a specific limit of <code>vector</code>’s array fusion framework (i.e., stream fusion for arrays). That limit is the reason why <code>vector</code> can only get within about a factor of two of the performance of hand-optimised C code for quickhull. The details of this limitation as well as a proposal to overcome it are explored in the paper <a href="https://dl.acm.org/citation.cfm?doid=2503778.2503782">Data flow fusion with series expressions in Haskell</a> (<a href="http://benl.ouroborus.net/papers/2013-series/flow-Haskell2013-rev1.pdf">PDF</a>). The paper also contains concrete benchmark results, including results for quickhull. Finally, the latest insights into overcoming this specific kind of limitation are described in <a href="https://dl.acm.org/citation.cfm?id=3131865">Machine fusion: merging merges, more or less</a> (<a href="http://benl.ouroborus.net/papers/2017-merges/merges-PPDP2017-final.pdf">PDF</a>).</p>
<h2>Conclusion</h2>
<p>The <code>vector</code> package is a tool that should be in every Haskell developers tool box. It is versatile, efficient, and widely used. Consequently, there is a range of add ons; in particular, <a href="https://hackage.haskell.org/package/vector-algorithms">vector-algorithms</a>, which implements search and sorting algorithms, is quite popular.</p>
<p>The <code>vector</code> package works hard to achieve excellent sequential performance. However, it, by design, ignores the option of improving performance by parallelisation. This will be the topic of the next post in this series.</p>Thu, 12 Oct 2017 00:00:00 +0000Russell O'Connor: Functor-Oriented Programminghttp://r6.ca/blog/20171010T001746Z.html
http://r6.ca/blog/20171010T001746Z.html
<p>My style of Haskell programming has been evolving over the 15 years that I have been working with it.
It is turning into something that I would like to call “functor oriented programming”.
The traditional use of typed functional programming focuses on data types.
One defines data types to model the data structures that your program operates on, and one writes functions to transform between these structures.
One of the primary goals in this traditional methodology is to create data structures that exclude erroneous states to the extent that is reasonably possible.
As long as one ensures that pattern matching is complete, then the type system catches many errors that would otherwise lead to these erroneous states, which have been crafted to be unrepresentable.
</p><p>Functor oriented programming is a refinement of this traditional focus on data types.
I was reminded of this concept recently when I was working with <a href="https://hackage.haskell.org/package/unification-fd" title="unification-fd: Simple generic unification algorithms.">wren’s fantastic unification-fd library</a>.
With functor oriented programming, one divides data structures into layers of functors that, when composed together, form the data structures that your program operates on.
Instead of writing transformations between data structures, one writes natural transformations between functors, where a natural transformation between functors <code>F</code> and <code>G</code> is a polymorphic function of type <code>forall a. F a -> G a</code>.
While traditional functions often operate on products of multiple inputs and/or outputs, with functor oriented programming one will often see functions operating on compositions of functors, including but not limited to distribution functions of type <code>forall a. F (G a) -> G (F a)</code> and half-way distribution functions <code>forall a. F (G a) -> G (H a)</code>, and many others.
</p><p>By dividing data structures up into layers of functors, one can create a separation of concerns that does not occur in traditional functional programming.
With functor oriented programming, polymorphism is not necessarily about using functions polymorphically.
Instead, polymorphism provides correctness guarantees by ensuring that a particular function can only touch the specific layers of functors in its type signature and is independent of the rest of the data structure.
<strong>One benefits from polymorphism even when a function is only ever invoked at a single type.</strong>
</p><p>The appearance of many natural transformations is one hallmark of functor oriented programming.
Higher-order natural transformations will invoke <code>Rank2Types</code> and <code>RankNTypes</code>, which is another hallmark of functor oriented programming.
Other hallmarks of functor oriented programming include open recursive types, which allows one to divide up recursive types into their layers of recursion and create natural transformations that operate on a single layer at a time.
Open recursive types plays an important role in wren’s unification library.
</p><p>As fine of a language that Haskell is, it is not actually that suitable for functor oriented programming.
The problem is that, under normal circumstances, there is no reduction or equivalence classes at the type level.
For example, the identity functor does not transparently disappear during composition, the <code>Compose</code> functor is not transparently associative, and the <code>Swap</code> functor composed with itself does not reduce to the identity functor.
To cope with this one must litter one’s code with <code>newtype</code> wrapper and unwrappers to make all these natural transformations explicit.
In principle, these transformations should have no run-time consequences, but when they are used in higher-order ways, unfortunately they sometimes do.
Despite the problems, I am not aware of any another practical language that better supports this style of programming.
I think Haskell’s higher-kinded type classes and the progression of <code>Monad</code>, <code>Applicative</code>, <code>Foldable</code>, <code>Traversable</code>, etc. classes have been instrumental in leading to the development of this style of programming as they further motivate the division of one’s data structures into these layers of functors.
</p><p>I have been thinking about writing this post for a few years now, and wanted to write something convincing; however, I do not think I am up to the task.
Instead of trying to persuade the reader, I have elected to try to simply name and describe this style of programming so that the reader might notice it themselves when reading and writing code.
Hopefully someone more capable than me can evangelize this approach, and perhaps even create a practical language suitable for this style of programming.
</p>Tue, 10 Oct 2017 00:17:46 +0000Russell O'Connor: Taking Propositions as Types Seriouslyhttp://r6.ca/blog/20171008T222703Z.html
http://r6.ca/blog/20171008T222703Z.html
<p>A while back I decided to try my hand at using <a href="http://wiki.portal.chalmers.se/agda/pmwiki.php" title="The Agda Wiki">Agda</a> and I wrote a proof of the <a href="https://hub.darcs.net/roconnor/STLC" title="roconnor's STLC :: hub.darcs.net">Church-Rosser theorem</a> in it.
It was a fun exercise.
I took all the knowledge I have picked up over the years about using dependent types to represent binders, to define well-typed terms, and what I have learned about the <a href="https://people.mpi-sws.org/~dreyer/courses/catlogic/jacobs.pdf" title="Categorical Logic and Type Theory">Category of Contexts</a> and applied it to <a href="https://hub.darcs.net/roconnor/STLC/browse/src/STLC/Terms.agda" title="Terms.agda :: hub.darcs.net">my definition of Terms for a simply typed lambda calculus</a>.
I am proud of the result.
</p><p>They say you do not understand a topic in mathematics until you can teach it.
And you do not really understand it until you can prove it in Coq.
And you do not really <em>really</em> understand it until you can prove it in Agda.
What really struck me was how my exercise in Adga affected my understanding of “Propositions as Types”.
</p><p>I view types as being divided into roughly two categories.
Some types are propositions.
They are the types that have at most one inhabitant, which is to say types for which you can prove that all their inhabitants are equal.
Other types are data types.
They are types with potentially more than one inhabitant.
As such you can distinguish between values by (possibly indirectly) doing case analysis on them.
Indeed, <a href="https://homotopytypetheory.org/" title="Homotopy Type Theory">HoTT</a> defines propositions in exactly this way.
</p><p>This classification of types is not fundamental in the theory of types.
The theory of types treats both propositions and data types uniformly.
I simply find it a useful way of characterizing types when programming and reasoning about programs with dependent type theory.
The void and unit types, ⊥ and ⊤ respectively, are both propositions.
We can define a function from the Boolean type to the universe of types which map the two Boolean values to these two types.
In this way we can covert any Boolean valued (computable) predicate into a logical (type-theoretical) predicate.
</p><p>To me the phrase “Proposition as Types” just meant the embedding of logical proposition as types with at most one inhabitant.
For example, given a decidable type <var>A</var>, we can compute if a given value of type <var>A</var> is a member of a given list of <var>A</var>s.
This computable predicate can be lifted to a logical predicate to define a logical membership relationship.
Expressions using this logical membership relationship are propositions according to the above definition of proposition.
</p><p>This is a fine way to do formal reasoning.
However, this is not the way that membership is typically defined in Agda.
Instead, Agda defines the membership relation as an inductive family whose constructors witness that an element is either the head of the list, or is a member of the tail of the list.
(Coq also defines the membership relation this way; however it is marked as non-extractable for computation by virtue of being a proposition.)
The type (<var>a</var> ∈ <var>l</var>) is, in general, a data type rather than a proposition.
When the value <var>a</var> occurs multiple times in <var>l</var>, then the type (<var>a</var> ∈ <var>l</var>) has multiple different “proofs” corresponding the the different occurrences of <var>a</var> within <var>l</var>.
Values of this type act as “the type of indexes where <var>a</var> occurs in <var>l</var>”, and one can write programs that operate over this type.
</p><p>Given two lists, <var>l</var><sub>1</sub> and <var>l</var><sub>2</sub>, the “proposition” that one list is a subset of the other states that any element of <var>l</var><sub>1</sub> is also an element of <var>l</var><sub>2</sub>:
</p><p><var>l</var><sub>1</sub> ⊆ <var>l</var><sub>2</sub> ≔ ∀<var>a</var>. <var>a</var>∈<var>l</var><sub>1</sub> → <var>a</var>∈<var>l</var><sub>2</sub>
</p><p>In dependent type theory this implication is represented as a function.
Because our membership relation is a data type, this subset relation is represented by a real program.
Specifically it is a program that, for any value, maps each index where it occurs in <var>l</var><sub>1</sub> to some index where it occurs in <var>l</var><sub>2</sub>; you can really evaluate this function.
This subset type is also, in general, a data type because there can be multiple such functions, which represent all the possible permutations that maps <var>l</var><sub>1</sub> onto <var>l</var><sub>2</sub>.
</p><p>The consequences of this are fantastic.
For example, what you normally think of as a theorem for weakening,
</p><pre>weaken : ∀ {Γ₁ Γ₂ A} → Γ₁ ⊆ Γ₂ → Term Γ₁ A → Term Γ₂ A</pre>
<p>also captures contraction and exchange due to this definition of subset.
All the <a href="https://en.wikipedia.org/w/index.php?title=Structural_rule&oldid=736195870" title="Structural rule">structural rules</a> of the lambda calculus are subsumed within a single theorem!
</p><p>It took me several attempts before I learned to take full advantage of this sort of logic.
For example, I originally defined a module as an inductive family.
However, I eventually settled on a functional representation for modules:
</p><pre>-- A Module is a list of terms of types Δ all in the same context Γ.
-- A Module is a block of terms for simultaneous substitution.
-- A Module is an arrow in the category of contexts.
-- A Module is a profuctor.
Module : List Type → List Type → Set
Module Γ Δ = ∀ {A} → A ∈ Δ → Term Γ A</pre>
<p>This definition means a module can evaluate.
In particular modules can be partially evaluated during type-checking, which greatly simplified many of my proofs as compared to using the inductive family way of defining modules.
</p><p>Notice how we make use of values of <code>A ∈ Δ</code> as data defining an index into the context <code>Δ</code>.
If <code>Δ</code> has multiple occurrences of <code>A</code>, then the term generated by the module can be different depending on which occurrence the index is referencing.
</p><p>In conclusion, although I do like thinking of types as divided between propositional types and data types, it can be fruitful to view expressions that mathematics traditionally treats as propositions instead as real data types, and really program with them.
This is what I mean by taking "Propositions as Types" seriously.
</p>Sun, 08 Oct 2017 22:27:03 +0000Gabriel Gonzalez: Why do our programs need to read input and write output?tag:blogger.com,1999:blog-1777990983847811806.post-3367018514902929143
http://www.haskellforall.com/2017/10/why-do-our-programs-need-to-read-input.html
<html xmlns="http://www.w3.org/1999/xhtml"><head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <meta content="text/css" http-equiv="Content-Style-Type"/> <meta content="pandoc" name="generator"/> <title></title> <style type="text/css">code{white-space: pre;}</style> <style type="text/css">div.sourceCode { overflow-x: auto; } table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode { margin: 0; padding: 0; vertical-align: baseline; border: none; } table.sourceCode { width: 100%; line-height: 100%; } td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; } td.sourceCode { padding-left: 5px; } code > span.kw { color: #007020; font-weight: bold; } /* Keyword */ code > span.dt { color: #902000; } /* DataType */ code > span.dv { color: #40a070; } /* DecVal */ code > span.bn { color: #40a070; } /* BaseN */ code > span.fl { color: #40a070; } /* Float */ code > span.ch { color: #4070a0; } /* Char */ code > span.st { color: #4070a0; } /* String */ code > span.co { color: #60a0b0; font-style: italic; } /* Comment */ code > span.ot { color: #007020; } /* Other */ code > span.al { color: #ff0000; font-weight: bold; } /* Alert */ code > span.fu { color: #06287e; } /* Function */ code > span.er { color: #ff0000; font-weight: bold; } /* Error */ code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ code > span.cn { color: #880000; } /* Constant */ code > span.sc { color: #4070a0; } /* SpecialChar */ code > span.vs { color: #4070a0; } /* VerbatimString */ code > span.ss { color: #bb6688; } /* SpecialString */ code > span.im { } /* Import */ code > span.va { color: #19177c; } /* Variable */ code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ code > span.op { color: #666666; } /* Operator */ code > span.bu { } /* BuiltIn */ code > span.ex { } /* Extension */ code > span.pp { color: #bc7a00; } /* Preprocessor */ code > span.at { color: #7d9029; } /* Attribute */ code > span.do { color: #ba2121; font-style: italic; } /* Documentation */ code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ </style></head><body><p>I wrote this post to challenge basic assumptions that people make about software architecture, which is why I chose a deliberately provocative title. You might not agree with all the points that I am about to make, but I do hope that this post changes the way that you think about programming</p><p>This post is an attempt to restate in my own words what Conal Elliot (and others before him) have been saying for a while: modern programming is a Rube-Goldberg machine that could be much simpler if we change the way we compose code.</p><p>Most programmers already intuitively understand this at some level. They will tell you that the programming ecosystem is deeply flawed, fragmented, and overly complex. I know that I felt that way myself for a long time, but in retrospect I believe that my objections at the time were superficial. There were deeper issues with programming that I was blind to because they are so ingrained in our collective psyche.</p><p>Disclaimer: This post uses my pet configuration language <a href="https://github.com/dhall-lang/dhall-lang">Dhall</a> to illustrate several points, mainly because Dhall is a constructive proof-of-concept of these ideas. The purpose of this post is not so much to convince you to use Dhall but rather to convince you to think about software in a new way</p><h2 id="input-and-output">Input and output</h2><p>Consider the title of this post for example:</p><blockquote><p><strong>"Why do our programs need to read input and write output?"</strong></p></blockquote><p>Most people will answer the title with something like:</p><ul><li><em>"Our programs need a way to communicate with the outside world"</em></li><li><em>"Programs need to do something other than heat up CPUs"</em></li></ul><p>Now suppose I phrased the question in a different way:</p><blockquote><p>"What if only the compiler could read input and write output?"</p></blockquote><p><em>"What's the difference?"</em>, you might ask. <em>"Surely you mean that the language provides some library function that I can call to read or write input to some handles, whether they be file handles or standard input/output."</em></p><p>No, I mean that only the compiler implementation is allowed to read input or write output, but programs written within the compiled language cannot read input or write output. You can only compute pure functions within the language proper.</p><p>Again, this probably seems ridiculous. How could you communicate at all with the program?</p><h2 id="imports">Imports</h2><p>Most languages have some way to import code, typically bundled in the form of packages. An imported function or value is something that our compiler reads as input statically at compile time as opposed to a value read by our program dynamically at run time.</p><p>Suppose I told you that our hypothetical programming language could only read input values by importing them</p><p><em>"Ridiculous!"</em> you exclaim as you spit out your coffee. <em>"Nobody would ever use such a language."</em> You probably wouldn't even know where to begin since so many things seem wrong with that proposal</p><p>Perhaps you would object to the heavyweight process for publishing and subscribing to new values. You would recite the package management process for your favorite programming language:</p><ul><li>Create a source module containing your value</li><li>Create a standard project layout</li><li>Create a package description file</li><li>Check your project into version control</li><li>Publish your package to a package repository</li></ul><p>Perhaps you would object to the heavyweight process for configuring programs via imports? Your favorite programming language would typically require you to:</p><ul><li>Retrieve the relevant program from version control</li><li>Modify the project description file to reference the newly published dependency</li><li>Modify project code to import your newly published value</li><li>Compile the program</li><li>Run the program</li></ul><p>Why would a non-technical end user do <em>any</em> of that just to read and write values?</p><p>This is exactly the Rube-Goldberg machine I'm referring to. We have come to expect a heavyweight process for source code to depend on other source code</p><h2 id="importing-paths">Importing paths</h2><p>Distributing code doesn't have to be heavyweight, though. Consider <a href="https://github.com/dhall-lang/dhall-lang">Dhall</a>'s import system which lets you reference expressions directly by their paths. For example, suppose we saved the value <code>True</code> to a file named <code>example.dhall</code>:</p><pre class="shell"><code>$ echo 'True' > example.dhall</code></pre><p>Another Dhall program can reference the above file anywhere the program expects a boolean value:</p><pre class="shell"><code>$ dhall <<< './example.dhall || False'<br />Bool<br /><br />True</code></pre><p>This is the exact same as if we had just replaced the path with the file's contents:</p><pre class="shell"><code>$ dhall <<< 'True || False'<br />Bool<br /><br />True</code></pre><p>Dhall doesn't need to support an explicit operation to read input because Dhall can read values by just importing them</p><p>Similarly, Dhall doesn't need to support an explicit write operation either. Just save a Dhall expression to a file using your favorite text editor.</p><p><em>"What if I need a way to automate the generation of files?"</em></p><p>You don't need to automate the process of saving a file because one file is always sufficiently rich to store as much information as you need. Dhall is a programmable configuration language which supports lists and records so any one file can store or programmatically generate any number of values. Files are human-generated artifacts which exist purely for our convenience but Dhall code does not behave any differently whether or not the program spans multiple files or a single file.</p><p>Most of the time people need to automate reads and writes because they are using non-programmable configuration file formats or data storage formats</p><h2 id="programmable-configuration">Programmable configuration</h2><p>You might object: <em>"Configuration files shouldn't be Turing-complete!"</em></p><p>However, Dhall is not Turing-complete. Dhall is a total programming language, meaning that evaluation eventually halts. In practice, we don't actually care that much if Dhall halts, since we cannot guarantee that the program halts on reasonable human timescales. However, we can statically analyze Dhall and most of the Halting Problem objections to static analysis don't apply.</p><p>For example, Dhall can statically guarantee that programs never fail or throw exceptions (because obviously a configuration file should never crash). Dhall also lets you simplify confusing files by eliminating all indirection because Dhall can reduce every program to a canonical normal form.</p><p>In fact, most objections to programmable configuration files are actually objections to Turing-completeness</p><h2 id="importing-urls">Importing URLs</h2><p>Dhall also lets you import code by URL instead of path. Dhall hosts the Prelude of standard utilities online using <a href="https://ipfs.io/">IPFS</a> (a distributed hashtable for the web),and you can browse the Prelude using this link, which redirects to the latest version of the Prelude:</p><ul><li><a href="http://prelude.dhall-lang.org/" class="uri">http://prelude.dhall-lang.org/</a></li></ul><p>For example, you can browse the above Prelude to find the <code>not</code> function hosted here:</p><ul><li><a href="https://ipfs.io/ipfs/QmQ8w5PLcsNz56dMvRtq54vbuPe9cNnCCUXAQp6xLc6Ccx/Prelude/Bool/not" class="uri">https://ipfs.io/ipfs/QmQ8w5PLcsNz56dMvRtq54vbuPe9cNnCCUXAQp6xLc6Ccx/Prelude/Bool/not</a></li></ul><p>... which has the following contents:</p><pre><code>$ curl https://ipfs.io/ipfs/QmQ8w5PLcsNz56dMvRtq54vbuPe9cNnCCUXAQp6xLc6Ccx/Prelude/Bool/not<br />{-<br />Flip the value of a `Bool`<br /><br />Examples:<br /><br />```<br />./not True = False<br /><br />./not False = True<br />```<br />-}<br />let not : Bool → Bool<br /> = λ(b : Bool) → b == False<br /><br />in not</code></pre><p>... and we can "apply" our URL to our file as if they were both just ordinary functions and values:</p><pre class="shell"><code>$ dhall <<< 'https://ipfs.io/ipfs/QmQ8w5PLcsNz56dMvRtq54vbuPe9cNnCCUXAQp6xLc6Ccx/Prelude/Bool/not ./example.dhall'<br />Bool<br /><br />False</code></pre><p>... or we could use <code>let</code>-expressions to improve readability without changing the result:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"> <span class="kw">let</span> not <span class="fu">=</span> https<span class="fu">://</span>ipfs<span class="fu">.</span>io<span class="fu">/</span>ipfs<span class="fu">/</span><span class="dt">QmQ8w5PLcsNz56dMvRtq54vbuPe9cNnCCUXAQp6xLc6Ccx</span><span class="fu">/</span><span class="dt">Prelude</span><span class="fu">/</span><span class="dt">Bool</span><span class="fu">/</span>not<br /><br /><span class="kw">in</span> <span class="kw">let</span> example <span class="fu">=</span> <span class="fu">./</span>example<span class="fu">.</span>dhall<br /><br /><span class="kw">in</span> not example</code></pre></div><p><em>"That's a security vulnerability!"</em>, you protest. <em>"You are ... literally ... injecting remote code into your program."</em></p><p>Playing the devil's advocate, I ask you what is wrong with remote code injection</p><p><em>"Well, for starters, if the URL is compromised the attacker can run arbitrary code like ..."</em></p><p>... reading and writing files? Dhall is totally pure and doesn't support any effects at all (besides heating up CPUs ☺).</p><p>This brings us full circle back to our original question:</p><blockquote><p><strong>"Why do our programs need to read input and write output?"</strong></p></blockquote><h2 id="conclusion">Conclusion</h2><p>Software doesn't have to be so complex if we challenge our assumptions that got us into this mess</p><p>These ideas were heavily inspired the following post by Conal Elliot:</p><ul><li><a href="http://conal.net/blog/posts/can-functional-programming-be-liberated-from-the-von-neumann-paradigm">Can functional programming be liberated from the von Neumann paradigm?</a></li></ul><p>... and this post is not about Dhall so much as Conal's vision of an effect-free purely functional future for programming. I believe explicitly reading input and writing output will eventually become low-level implementation details of higher-level languages, analogous to allocating stack registers or managing memory.</p></body></html>Sun, 08 Oct 2017 20:34:22 +0000noreply@blogger.com (Gabriel Gonzalez)Joachim Breitner: e.g. in TeXhttp://www.joachim-breitner.de/blog/731-e_g__in_TeX
http://www.joachim-breitner.de/blog/731-e_g__in_TeX
<p>When I learned TeX, I was told to not write <code>e.g. something</code>, because TeX would think the period after the “g” ends a sentence, and introduce a wider, inter-sentence space. Instead, I was to write <code>e.g.\␣</code>.</p>
<p>Years later, I learned from a convincing, but since forgotten source, that in fact <code>e.g.\@</code> is the proper thing to write. I vaguely remembering that <code>e.g.\␣</code> supposedly affected the inter-word space in some unwanted way. So I did that for many years.</p>
<p>Until I recently was called out for doing it wrong, and that infact <code>e.g.\␣</code> is the proper way. This was supported by <a href="https://tex.stackexchange.com/a/22563/15107">a StackExchange answer</a> written by a LaTeX authority and backed by a reference to documentation. The same question has, however, <a href="https://tex.stackexchange.com/a/22564/15107">another answer</a> by another TeX authority, backed by an analysis of the implementation, which concludes that <code>e.g.\@</code> is proper.</p>
<p>What now? I guess I just have to find it out myself.</p>
<div class="figure">
<img src="//www.joachim-breitner.de/various/tex-eg/tex-eg-at-1.gif" alt="The problem and two solutions" /><p class="caption">The problem and two solutions</p>
</div>
<p>The above image shows three variants: The obviously broken version with <code>e.g.</code>, and the two contesting variants to fix it. Looks like they yield equal results!</p>
<p>So maybe the difference lies in how <code>\@</code> and <code>\␣</code> react when the line length changes, and the word wrapping require differences in the inter-word spacing. Will there be differences? Let’s see;</p>
<div class="figure">
<img src="//www.joachim-breitner.de/various/tex-eg/tex-eg-at-2.gif" alt="Expanding whitespace, take 1" /><p class="caption">Expanding whitespace, take 1</p>
</div>
<div class="figure">
<img src="//www.joachim-breitner.de/various/tex-eg/tex-eg-at-3.gif" alt="Expanding whitespace, take 2" /><p class="caption">Expanding whitespace, take 2</p>
</div>
<p>I cannot see any difference. But the inter-sentence whitespace ate most of the expansion. Is there a difference visible if we have only inter-word spacing in the line?</p>
<div class="figure">
<img src="//www.joachim-breitner.de/various/tex-eg/tex-eg-at-4.gif" alt="Expanding whitespace, take 3" /><p class="caption">Expanding whitespace, take 3</p>
</div>
<div class="figure">
<img src="//www.joachim-breitner.de/various/tex-eg/tex-eg-at-5.gif" alt="Expanding whitespace, take 4" /><p class="caption">Expanding whitespace, take 4</p>
</div>
<p>Again, I see the same behaviour.</p>
<p><strong>Conclusion</strong>: It does not matter, but <code>e.g.\␣</code> is less hassle when using <a href="http://hackage.haskell.org/package/lhs2tex">lhs2tex</a> than <code>e.g.\@</code> (which has to be escaped as <code>e.g.\@@</code>), so the winner is <code>e.g.\␣</code>!</p>
<p>(Unless you put it in a macro, <a href="https://tex.stackexchange.com/a/116530/15107">then <code>\@</code> might be preferable</a>, and it is still <a href="https://tex.stackexchange.com/a/55112/15107">needed between a captial letter and a sentence period</a>.)</p>Sun, 08 Oct 2017 19:08:13 +0000mail@joachim-breitner.de (Joachim Breitner)Sandy Maguire: Review: Bananas, Lenses, Envelopes and Barbed Wirehttp://reasonablypolymorphic.com//blog/recursion-schemes
http://reasonablypolymorphic.com//blog/recursion-schemes
<div class="main">
<article>
<header>
<h1><a href="http://reasonablypolymorphic.com/blog/recursion-schemes">Review: Bananas, Lenses, Envelopes and Barbed Wire</a></h1>
</header>
<p class="meta">
<span class="prev">
<a href="http://reasonablypolymorphic.com/blog/modeling-music">←</a>
</span>
<time>October 6, 2017</time>
<span class="tags">
</span>
</p>
<div class="content">
<p>Today’s classic functional programming paper we will review is Meijer et al.’s <a href="https://maartenfokkinga.github.io/utwente/mmf91m.pdf">Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire</a>. The exciting gist of the paper is that all explicit recursion can be factored out into a few core combinators. As such, the reasoning is that we should instead learn these combinators (or “recursion schemes” as they’re called), rather than doing our own ad-hoc recursion whenever we need it.</p>
<p>Despite being a marvelous paper, it falls into the all-too-common flaw of functional programming papers, which is to have an absolutely horrible title. “Bananas”, “lenses”, “envelopes” and “barbed wire” correspond to obscure pieces of syntax invented to express these ideas. In our treatment of the literature, we will instead use standard Haskell syntax, and refer to the paper as Functional Programming with Recursion Schemes.</p>
<h2 id="specialized-examples-of-recursion-schemes">Specialized Examples of Recursion Schemes</h2>
<h3 id="catamorphisms-over-lists">Catamorphisms over Lists</h3>
<p><strong>Catamorphisms</strong> refer to a fold over a datastructure. A mnemonic to remember this is that a <em>cata</em>morphism tears down structures, and that if that structure were our civilization it’d be a <em>cata</em>strophe.</p>
<p>By way of example, Meijer et al. present the following specialization of a catamorphism over lists:</p>
<blockquote>
<p>Let <code>default :: b</code> and <code>step :: a -> b -> b</code>, then a <em>list-catamorphism</em> <code>h :: [a] -> b</code> if a function of the following form:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">h [] <span class="fu">=</span> default
h (a <span class="fu">:</span> as) <span class="fu">=</span> step a (h as)</code></pre></div>
</blockquote>
<p>This definition should look pretty familiar; if you specialize the function <code>foldr</code> to lists, you’ll see it has the type:</p>
<pre class="haskel"><code>foldr :: (a -> b -> b) -> b -> [a] -> b</code></pre>
<p>We can view <code>foldr</code> as taking our values <code>step :: a -> b -> b</code> and <code>default :: b</code>, and then giving back a function that takes an <code>[a]</code> and computes some <code>b</code>. For example, we can write a few of the common prelude functions over lists as catamorphisms of this form.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">length<span class="ot"> ::</span> [a] <span class="ot">-></span> <span class="dt">Int</span>
length <span class="fu">=</span> foldr (\_ n <span class="ot">-></span> n <span class="fu">+</span> <span class="dv">1</span>) <span class="dv">0</span></code></pre></div>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">filter<span class="ot"> ::</span> forall a<span class="fu">.</span> (a <span class="ot">-></span> <span class="dt">Bool</span>) <span class="ot">-></span> [a] <span class="ot">-></span> [a]
filter p <span class="fu">=</span> foldr step []
<span class="kw">where</span>
<span class="ot"> step ::</span> a <span class="ot">-></span> [a] <span class="ot">-></span> [a]
step a as <span class="fu">=</span> <span class="kw">if</span> p a
<span class="kw">then</span> a <span class="fu">:</span> as
<span class="kw">else</span> as</code></pre></div>
<p>When written this way – Meijer et al. are quick to point out – the so-called “fusion” law is easily seen:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">f <span class="fu">.</span> foldr step default <span class="fu">=</span> foldr step' default'
<span class="kw">where</span>
step' a b <span class="fu">=</span> step a (f b)
default' <span class="fu">=</span> f default</code></pre></div>
<p>which intuitively says that you can “fuse” a catamorphism with a subsequent composition into a single catamorphism.</p>
<h3 id="anamorphisms-over-lists">Anamorphisms over Lists</h3>
<p>If a catamorphism refers to a “fold”, an <strong>anamorphism</strong> corresponds to an <em>unfold</em> of a data structure. A good mnemonic for this is that an <em>ana</em>morphism builds things up, just like <em>ana</em>bolic steroids can be an easy way to build up muscle mass.</p>
<p>Meijer et al. present this concept over lists with the following (again, very specialized) definition:</p>
<p>unfoldr :: (b -> Maybe (a, b)) -> b -> [a]</p>
<blockquote>
<p>Given a function <code>produce :: b -> Maybe (a, b)</code>, a list-anamorphism <code>h :: b -> [a]</code> is defined as:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">h seed <span class="fu">=</span> <span class="kw">case</span> produce seed <span class="kw">of</span>
<span class="dt">Just</span> (a, b) <span class="fu">=</span> a <span class="fu">:</span> h b
<span class="dt">Nothing</span> <span class="fu">=</span> []</code></pre></div>
</blockquote>
<p>As expected, this corresponds to the <code>unfoldr :: (b -> Maybe (a, b)) -> b -> [a]</code> function from <code>Data.List</code>.</p>
<p>By way of example, they provide the following:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">zip<span class="ot"> ::</span> ([a], [b]) <span class="ot">-></span> [(a, b)]
zip <span class="fu">=</span> unfoldr produce
<span class="kw">where</span>
produce (as, bs) <span class="fu">=</span>
<span class="kw">if</span> null as <span class="fu">||</span> null bs
<span class="kw">then</span> <span class="dt">Nothing</span>
<span class="kw">else</span> <span class="dt">Just</span> ((head as, head bs), (tail as, tail bs))</code></pre></div>
<p>and</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">iterate<span class="ot"> ::</span> (a <span class="ot">-></span> a) <span class="ot">-></span> a <span class="ot">-></span> [a]
iterate f <span class="fu">=</span> unfoldr (\a <span class="ot">-></span> <span class="dt">Just</span> (a, f a))</code></pre></div>
<p>An interesting case is that of <code>map :: (a -> b) -> [a] -> [b]</code>. We note that both the input and output of this function are lists, and thus might suspect the function can be written as either a catamorphism <em>or</em> an anamorphism. And indeed, it can be:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">cataMap ::</span> (a <span class="ot">-></span> b) <span class="ot">-></span> [a] <span class="ot">-></span> [b]
cataMap f <span class="fu">=</span> foldr (\a bs <span class="ot">-></span> f a <span class="fu">:</span> bs) []
<span class="ot">anaMap ::</span> (a <span class="ot">-></span> b) <span class="ot">-></span> [a] <span class="ot">-></span> [b]
anaMap f <span class="fu">=</span> unfoldr produce
<span class="kw">where</span>
produce [] <span class="fu">=</span> <span class="dt">Nothing</span>
produce (a <span class="fu">:</span> as) <span class="fu">=</span> <span class="dt">Just</span> (f a, as)</code></pre></div>
<p>Neat!</p>
<h3 id="hylomorphisms-over-lists">Hylomorphisms over Lists</h3>
<p>A <strong>hylomorphism</strong> over lists is a recursive function of type <code>a -> b</code> whose call-tree is isomorphic to a list. A hylomorphism turns out to be nothing more than a catamorphism following an anamorphism; the anamorphism builds up the call-tree, and the catamorphism evaluates it.</p>
<p>An easy example of hylomorphisms is the factorial function, which can be naively (ie. without recursion schemes) implemented as follows:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">fact ::</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Int</span>
fact <span class="dv">0</span> <span class="fu">=</span> <span class="dv">1</span>
fact n <span class="fu">=</span> n <span class="fu">*</span> fact (n <span class="fu">-</span> <span class="dv">1</span>)</code></pre></div>
<p>When presented like this, it’s clear that <code>fact</code> will be called a linear number of times in a tail-recursive fashion. That sounds a lot like a list to me, and indeed we can implement <code>fact</code> as a hylomorphism:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">fact ::</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Int</span>
fact <span class="fu">=</span> foldr (<span class="fu">*</span>) <span class="dv">1</span> <span class="fu">.</span> unfoldr (\n <span class="ot">-></span> <span class="kw">if</span> n <span class="fu">==</span> <span class="dv">0</span>
<span class="kw">then</span> <span class="dt">Nothing</span>
<span class="kw">else</span> <span class="dt">Just</span> (n, n <span class="fu">-</span> <span class="dv">1</span>))</code></pre></div>
<p>The hylomorphic representation of <code>fact</code> works by unfolding its argument <code>n</code> into a list <code>[n, n-1 .. 1]</code>, and then folding that list by multiplying every element in it.</p>
<p>However, as Meijer et al. point out, this implementation of <code>fact</code> is a little unsatisfactory. Recall that the natural numbers are themselves an inductive type (<code>data Nat = Zero | Succ Nat</code>), however, according to the paper, there is no easy catamorphism (nor anamorphism) that implements <code>fact</code>.</p>
<h3 id="paramorphisms">Paramorphisms</h3>
<p>Enter <strong>paramorphisms</strong>: intuitively catamorphisms that have access to the current state of the structure-being-torn-down. Meijer et al.:</p>
<blockquote>
<p>[Let <code>init :: b</code> and <code>merge :: Nat -> b -> b</code>. ] For type <code>Nat</code>, a paramorphism is a function <code>h :: Nat -> b</code> of the form:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">h <span class="dt">Zero</span> <span class="fu">=</span> init
h (<span class="dt">Succ</span> n) <span class="fu">=</span> merge n (h n)</code></pre></div>
</blockquote>
<p>As far as I can tell, there is no function in the Haskell standard library that corresponds to this function-as-specialized, so we will write it ourselves:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">paraNat ::</span> (<span class="dt">Nat</span> <span class="ot">-></span> b <span class="ot">-></span> b) <span class="ot">-></span> b <span class="ot">-></span> <span class="dt">Nat</span> <span class="ot">-></span> b
paraNat _ init <span class="dt">Zero</span> <span class="fu">=</span> init
paraNat merge init (<span class="dt">Succ</span> n) <span class="fu">=</span> merge n (paraNat merge init n)</code></pre></div>
<p>We can thus write <code>fact :: Nat -> Nat</code> as a paramorphism:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">fact ::</span> <span class="dt">Nat</span> <span class="ot">-></span> <span class="dt">Nat</span>
fact <span class="fu">=</span> paraNat (\n acc <span class="ot">-></span> (<span class="dv">1</span> <span class="fu">+</span> n) <span class="fu">*</span> acc) <span class="dv">1</span></code></pre></div>
<p>Similarly, we can define paramorphisms over lists via the function:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">paraList ::</span> (a <span class="ot">-></span> [a] <span class="ot">-></span> b <span class="ot">-></span> b) <span class="ot">-></span> b <span class="ot">-></span> [a] <span class="ot">-></span> b
paraList _ init [] <span class="fu">=</span> init
paraList merge init (a <span class="fu">:</span> as) <span class="fu">=</span> merge a as (paraList merge init as)</code></pre></div>
<p>with which we can write the function <code>tails :: [a] -> [[a]]</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">tails ::</span> forall a<span class="fu">.</span> [a] <span class="ot">-></span> [[a]]
tails <span class="fu">=</span> paraList merge []
<span class="kw">where</span>
<span class="ot"> merge ::</span> a <span class="ot">-></span> [a] <span class="ot">-></span> [[a]] <span class="ot">-></span> [[a]]
merge a as ass <span class="fu">=</span> (a <span class="fu">:</span> as) <span class="fu">:</span> ass</code></pre></div>
<h2 id="general-recursion-schemes">General Recursion Schemes</h2>
<h3 id="intuition">Intuition</h3>
<p>As you’ve probably guessed, the reason we’ve been talking so much about these recursion schemes is that they generalize to all recursive data types. The trick, of course, is all in the representation.</p>
<p>Recall the standard definition of list:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">data</span> <span class="dt">List</span> a <span class="fu">=</span> <span class="dt">Nil</span>
<span class="fu">|</span> <span class="dt">Cons</span> a (<span class="dt">List</span> a)</code></pre></div>
<p>However, there’s no reason we need the explicit recursion in the <code>Cons</code> data structure. Consider instead, an alternative, “fixable” representation:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">data</span> <span class="dt">List'</span> a x <span class="fu">=</span> <span class="dt">Nil'</span>
<span class="fu">|</span> <span class="dt">Cons'</span> a x</code></pre></div>
<p>If we were somehow able to convince the typesystem to unify <code>x ~ List' a x</code>, we’d get the type <code>List' a (List' a (List' a ...))</code>, which is obviously isomorphic to <code>List a</code>. We’ll look at how to unify this in a second, but a more pressing question is “why would we want to express a list this way?”.</p>
<p>It’s a good question, and the answer is we’d want to do this because <code>List' a x</code> is obviously a functor in <code>x</code>. Furthermore, in general, any datastructure we perform this transformation on will be a functor in its previously-recursive <code>x</code> parameter.</p>
<p>We’re left only more curious, however. What good is it to us if <code>List' a x</code> is a functor in <code>x</code>? It means that we can replace <code>x</code> with some other type <code>b</code> which is <em>not</em> isomorphic to <code>List a</code>. If you squint and play a little loose with the type isomorphisms, this specializes <code>fmap :: (x -> b) -> List' a x -> List' a b</code> to <code>(List a -> b) -> List' a x -> List' a b</code>.</p>
<p>Notice the <code>List a -> b</code> part of this function – that’s a fold of a <code>List a</code> into a <code>b</code>! Unfortunately we’re still left with a <code>List' a b</code>, but this turns out to be a problem only in our handwaving of <code>x ~ List' a x</code>, and the actual technique will in fact give us just a <code>b</code> at the end of the day.</p>
<h3 id="algebras">Algebras</h3>
<p>An <code>f</code>-algebra is a function of type <code>forall z. f z -> z</code>, which intuitively removes the structure of an <code>f</code>. If you think about it, this is spiritually what a fold does; it removes some structure as it reduces to some value.</p>
<p>As it happens, the type of an <code>f</code>-algebra is identical to the parameters required by a catamorphism. Let’s look at <code>List' a</code>-algebras and see how the correspond with our previous examples of catamorphisms.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">length<span class="ot"> ::</span> [a] <span class="ot">-></span> <span class="dt">Int</span>
length <span class="fu">=</span> foldr (\_ n <span class="ot">-></span> n <span class="fu">+</span> <span class="dv">1</span>) <span class="dv">0</span>
<span class="ot">lengthAlgebra ::</span> <span class="dt">List'</span> a <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Int</span>
lengthAlgebra <span class="dt">Nil'</span> <span class="fu">=</span> <span class="dv">0</span>
lengthAlgebra (<span class="dt">Cons'</span> _ n) <span class="fu">=</span> n <span class="fu">+</span> <span class="dv">1</span></code></pre></div>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">filter<span class="ot"> ::</span> forall a<span class="fu">.</span> (a <span class="ot">-></span> <span class="dt">Bool</span>) <span class="ot">-></span> <span class="dt">List</span> a <span class="ot">-></span> <span class="dt">List</span> a
filter p <span class="fu">=</span> foldr step <span class="dt">Nil</span>
<span class="kw">where</span>
step a as <span class="fu">=</span>
<span class="kw">if</span> p a
<span class="kw">then</span> <span class="dt">Cons</span> a as
<span class="kw">else</span> as
<span class="ot">filterAlgebra ::</span> (a <span class="ot">-></span> <span class="dt">Bool</span>) <span class="ot">-></span> <span class="dt">List'</span> a (<span class="dt">List</span> a) <span class="ot">-></span> (<span class="dt">List</span> a)
filterAlgebra _ <span class="dt">Nil'</span> <span class="fu">=</span> <span class="dt">Nil</span>
filterAlgebra p (<span class="dt">Cons'</span> a as) <span class="fu">=</span>
<span class="kw">if</span> p a
<span class="kw">then</span> <span class="dt">Cons</span> a as
<span class="kw">else</span> as</code></pre></div>
<h3 id="coalgebras">Coalgebras</h3>
<p><code>f</code>-algebras correspond succinctly to the parameters of catamorphisms over <code>f</code>s. Since catamorphisms are dual to anamorphisms, we should expect that by turning around an algebra we might get a representation of the anamorphism parameters.</p>
<p>And we’d be right. Such a thing is called an <code>f</code>-coalgebra of type <code>forall z. z -> f z</code>, and corresponds exactly to these parameters. Let’s look at our previous examples of anamorphisms through this lens:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">zip<span class="ot"> ::</span> (<span class="dt">List</span> a, <span class="dt">List</span> b) <span class="ot">-></span> <span class="dt">List</span> (a, b)
zip <span class="fu">=</span> unfoldr produce
<span class="kw">where</span>
produce (as, bs) <span class="fu">=</span>
<span class="kw">if</span> null as <span class="fu">||</span> null bs
<span class="kw">then</span> <span class="dt">Nothing</span>
<span class="kw">else</span> <span class="dt">Just</span> ((head as, head bs), (tail as, tail bs))
<span class="ot">zipCoalgebra ::</span> ([a], [b]) <span class="ot">-></span> <span class="dt">List'</span> (a, b) ([a], [b])
zipCoalgebra (as, bs) <span class="fu">=</span>
<span class="kw">if</span> null as <span class="fu">||</span> null bs
<span class="kw">then</span> <span class="dt">Nil'</span>
<span class="kw">else</span> <span class="dt">Cons</span> (head as, head bs) (tail as, tail bs)</code></pre></div>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">iterate<span class="ot"> ::</span> (a <span class="ot">-></span> a) <span class="ot">-></span> a <span class="ot">-></span> [a]
iterate f <span class="fu">=</span> unfoldr (\a <span class="ot">-></span> <span class="dt">Just</span> (a, f a))
<span class="ot">iterateAlgebra ::</span> (a <span class="ot">-></span> a) <span class="ot">-></span> a <span class="ot">-></span> <span class="dt">List'</span> a a
iterateAlgebra f a <span class="fu">=</span> <span class="dt">Cons</span> a (f a)</code></pre></div>
<p>You might have noticed that these coalgebras don’t line up as nicely as the algebras did, due namely to the <code>produce</code> functions returning a type of <code>Maybe (a, b)</code>, while the coalgebras return a <code>List' a b</code>. Of course, these types are isomorphic (<code>Nothing <=> Nil'</code>, <code>Just (a, b) <=> Cons a b</code>), it’s just that the authors of <code>unfoldr</code> didn’t have our <code>List'</code> functor to play with.</p>
<h3 id="from-algebras-to-catamorphisms">From Algebras to Catamorphisms</h3>
<p>As we have seen, <code>f</code>-algebras correspond exactly to the parameters of a catamorphism over an <code>f</code>. But how can we actually implement the catamorphism? We’re almost there, but first we need some machinery.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">type</span> family <span class="dt">Fixable</span><span class="ot"> t ::</span> <span class="fu">*</span> <span class="ot">-></span> <span class="fu">*</span></code></pre></div>
<p>The type family <code>Fixable</code> takes a type to its fixable functor representation. For example, we can use it to connect our <code>List a</code> type to <code>List' a</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">type</span> <span class="kw">instance</span> <span class="dt">Fixable</span> (<span class="dt">List</span> a) <span class="fu">=</span> <span class="dt">List'</span> a</code></pre></div>
<p>Now, assuming we have a function <code>toFixable :: t -> Fixable t t</code>, which for lists looks like this:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">toFixable ::</span> <span class="dt">List</span> a <span class="ot">-></span> <span class="dt">Fixable</span> (<span class="dt">List</span> a) (<span class="dt">List</span> a)
<span class="co">-- equivalently: toFixable :: List a -> List' a (List a)</span>
toFixable <span class="dt">Nil</span> <span class="fu">=</span> <span class="dt">Nil'</span>
toFixable (<span class="dt">Cons</span> a as) <span class="fu">=</span> <span class="dt">Cons'</span> a as</code></pre></div>
<p>We can now write our catamorphism!</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">cata ::</span> (<span class="dt">Fixable</span> t z <span class="ot">-></span> z) <span class="ot">-></span> (t <span class="ot">-></span> z)
cata algebra <span class="fu">=</span> algebra
<span class="fu">.</span> fmap (cata algebra)
<span class="fu">.</span> toFixable</code></pre></div>
<p>Very cool. What we’ve built here is general machinery for tearing down any inductive data structure <code>t</code>. All we need to do it is its <code>Fixable t</code> representation, and a function <code>project :: t -> Fixable t t</code>. These definitions turn out to be completely mechanical, and thankfully, <a href="https://hackage.haskell.org/package/recursion-schemes-5.0.2/docs/Data-Functor-Foldable-TH.html#v:makeBaseFunctor">can be automatically derived</a> for you via the <a href="https://hackage.haskell.org/package/recursion-schemes-5.0.2">recursion-schemes</a> package.</p>
<h3 id="coalgebras-and-catamorphisms">Coalgebras and Catamorphisms</h3>
<p>We can turn all of our machinery around in order to implement anamorphisms. We’ll present this material quickly without much commentary, since there are no new insights here.</p>
<p>Given <code>fromFixable :: Fixable t t -> t</code>, we can implement <code>ana</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">ana ::</span> (z <span class="ot">-></span> <span class="dt">Fixable</span> t z) <span class="ot">-></span> z <span class="ot">-></span> t
ana coalgebra <span class="fu">=</span> fromFixable
<span class="fu">.</span> fmap (ana coalgebra)
<span class="fu">.</span> coalgebra</code></pre></div>
<h3 id="general-paramorphisms">General Paramorphisms</h3>
<p>Because there is nothing interesting about hylomorphisms when viewed via our <code>Fixable</code> machinery, we skip directly to paramorphisms.</p>
<p>Recall that a paramorphism is a fold that has access to both the accumulated value being constructed as well as the remainder of the structure at any given point in time. We can represent such a thing “algebraically:” <code>Fixable t (t, z) -> z</code>. With a minor tweak to <code>cata</code>, we can get <code>para</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">para ::</span> (<span class="dt">Fixable</span> t (t, z) <span class="ot">-></span> z) <span class="ot">-></span> t <span class="ot">-></span> z
para alg <span class="fu">=</span> teardown
<span class="kw">where</span>
teardown <span class="fu">=</span> alg
<span class="fu">.</span> fmap (\t <span class="ot">-></span> (t, teardown t))
<span class="fu">.</span> toFixable</code></pre></div>
<h2 id="miscellaneous-findings">Miscellaneous Findings</h2>
<h3 id="all-injective-functions-are-catamorphisms">All Injective Functions are Catamorphisms</h3>
<p>Meijer et al. make the somewhat-unsubstantiated claim that all injective functions are catamorphisms. We will reproduce their proof here, and then work through it to convince ourselves of its correctness.</p>
<blockquote>
<p>Let <code>f :: A -> B</code> be a strict function with left-inverse <code>g</code>. Then for any <code>φ :: F A -> A</code>, we have:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">g <span class="fu">.</span> f <span class="fu">=</span> id
f <span class="fu">.</span> cata φ <span class="fu">=</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)</code></pre></div>
<p>Taking `φ = fromFixable$ we immediately get that any strict injective function can be written as a catamorphism:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">f <span class="fu">=</span> cata (f <span class="fu">.</span> fromFixable <span class="fu">.</span> fmap g)</code></pre></div>
</blockquote>
<p>Sounds, good? I guess? Meijer et al. must think I’m very smart, because it took me about a week of bashing my head against this proof before I got it. There were two stumbling blocks for me which we’ll tackle together.</p>
<p>To jog our memories, we’ll look again at the definition of <code>cata</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">cata φ <span class="fu">=</span> φ <span class="fu">.</span> fmap (cata φ) <span class="fu">.</span> toFixable</code></pre></div>
<p>There are two claims we need to tackle here, the first of which is that given <code>φ = fromFixable</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">f <span class="fu">.</span> cata φ <span class="fu">=</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)</code></pre></div>
<p>We can show this by mathematical induction. We’ll first prove the base case, by analysis of the <code>[]</code> case over list-algebras.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g) []
<span class="co">-- definition of cata</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap g <span class="fu">.</span> fmap (cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">$</span> toFixable []
<span class="co">-- definition of toFixable</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap g <span class="fu">.</span> fmap (cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">$</span> <span class="dt">NilF</span>
<span class="co">-- fmap fusion</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap (g <span class="fu">.</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">$</span> <span class="dt">NilF</span>
<span class="co">-- NilF is a constant functor</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">$</span> <span class="dt">NilF</span>
<span class="co">-- substitute φ = fromFixable</span>
<span class="fu">=</span> f <span class="fu">$</span> fromFixable <span class="dt">NilF</span>
<span class="co">-- definition of fromFixable</span>
<span class="fu">=</span> f []</code></pre></div>
<p>Great! That’s the base case tackled. It’s easy to see why this generalizes away from lists to any data structure; the only way to terminate the recursion of a <code>cata</code> is for one of the type’s data constructors to not be recursive (such as <code>Nil</code>). If the data constructor isn’t recursive, its <code>Fixable</code> representation must be a constant functor, and thus the final <code>fmap</code> will always fizzle itself out.</p>
<p>Let’s tackle the inductive case now. We’ll look at cases of <code>cata</code> that don’t fizzle out immediately:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)
<span class="co">-- definition of cata</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap g <span class="fu">.</span> fmap (cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">.</span> toFixable
<span class="co">-- fmap fusion</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap (g <span class="fu">.</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">.</span> toFixable
<span class="co">-- definition of cata</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap (g <span class="fu">.</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap g <span class="fu">.</span> fmap (cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">.</span> toFixable) <span class="fu">.</span> toFixable
<span class="co">-- fmap fusion</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap (g <span class="fu">.</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap (g <span class="fu">.</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">.</span> toFixable) <span class="fu">.</span> toFixable
<span class="co">-- g . f = id</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap (id <span class="fu">.</span> φ <span class="fu">.</span> fmap (g <span class="fu">.</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">.</span> toFixable) <span class="fu">.</span> toFixable
<span class="co">-- definition of id</span>
<span class="fu">=</span> f <span class="fu">.</span> φ <span class="fu">.</span> fmap (φ <span class="fu">.</span> fmap (g <span class="fu">.</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)) <span class="fu">.</span> toFixable) <span class="fu">.</span> toFixable</code></pre></div>
<p>As you can see here, subsequent expansions of <code>cata</code> line their <code>g</code>s and <code>f</code>s up in such a way that they cancel out. Also, we know from our experience looking at the base case that the final <code>g</code> will always sizzle out, and so we don’t need to worry about it only being a left-inverse.</p>
<p>The other stumbling block for me was that <code>cata fromFixable = id</code>, but this turns out to be trivial:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"> cata fromFixable
<span class="fu">=</span> fromFixable <span class="fu">.</span> fmap (cata fromFixable) <span class="fu">.</span> toFixable</code></pre></div>
<p>Eventually this will all bottom out when it hits the constant functor, which will give us a giant chain of <code>fromFixable . toFixable</code>s, which is obviously <code>id</code>.</p>
<p>To circle back to our original claim that all injective functions are catamorphisms, we’re now ready to tackle it for real.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">f <span class="fu">.</span> cata φ <span class="fu">=</span> cata (f <span class="fu">.</span> φ <span class="fu">.</span> fmap g)
f <span class="fu">.</span> cata fromFixable <span class="fu">=</span> cata (f <span class="fu">.</span> fromFixable <span class="fu">.</span> fmap g)
f <span class="fu">.</span> id <span class="fu">=</span> cata (f <span class="fu">.</span> fromFixable <span class="fu">.</span> fmap g)
f <span class="fu">=</span> cata (f <span class="fu">.</span> fromFixable <span class="fu">.</span> fmap g)</code></pre></div>
<p><span class="math inline">\(\blacksquare\)</span></p>
<h3 id="all-surjective-functions-are-anamorphisms">All Surjective Functions are Anamorphisms</h3>
<p>Anamorphisms are dual to catamorphisms, and surjective functions are dual (in <span class="math inline">\(\mathbb{Set}\)</span>) to injective functions. Therefore, we can get this proof via duality from the proof that injective functions are catamorphisms. <span class="math inline">\(\blacksquare\)</span></p>
<h2 id="closing-remarks">Closing Remarks</h2>
<p><em>Functional Programming with Recursion Schemes</em> has some other (in my opinion) minor contributions about this stuff, such as how catamorphisms preserve strictness, but I feel like we’ve tackled the most interesting pieces of it. It is my hope that this review will serve as a useful complement in understanding the original paper.</p>
</div>
<p class="meta">
<span class="prev">
<a href="http://reasonablypolymorphic.com/blog/modeling-music">←</a>
</span>
</p>
</article>
</div>Fri, 06 Oct 2017 00:00:00 +0000Christopher Allen: Comparing Persistent with Ecto and ActiveRecordhttp://bitemyapp.com//posts/2017-10-06-persistent-ecto-activerecord.html
http://bitemyapp.com//posts/2017-10-06-persistent-ecto-activerecord.html
<div class="info">
</div>
<div class="post">
<p>Rejected title: You’re not special</p>
<p>I saw this article comparing Ecto and ActiveRecord: <a href="https://www.dailydrip.com/blog/ecto-vs-activerecord.html" class="uri">https://www.dailydrip.com/blog/ecto-vs-activerecord.html</a></p>
<p>I thought I would track alongside that post and show what the equivalent code looks like if you’re using the <a href="https://github.com/yesodweb/persistent">Persistent</a> Haskell library.</p>
<h1 id="some-examples-from-their-side-by-side-comparison">Some examples from their side-by-side comparison</h1>
<p>I’ll start by simply translating some small, simple examples linked to at the beginning of their article.</p>
<h2 id="get-all-records">Get all records</h2>
<h3 id="activerecord">ActiveRecord</h3>
<div class="sourceCode"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span class="dt">Model</span>.all</code></pre></div>
<h3 id="ecto">Ecto</h3>
<div class="sourceCode"><pre class="sourceCode elixir"><code class="sourceCode elixir"><span class="cn">Repo</span><span class="op">.</span>all(<span class="cn">App</span><span class="op">.</span><span class="cn">Model</span>)</code></pre></div>
<h3 id="persistent">Persistent</h3>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">getAllUsers ::</span> <span class="dt">DB</span> [<span class="dt">Entity</span> <span class="dt">User</span>]
getAllUsers <span class="fu">=</span> selectList [] []</code></pre></div>
<h2 id="search-by-name">Search by name</h2>
<h3 id="activerecord-1">ActiveRecord</h3>
<div class="sourceCode"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span class="dt">Model</span>.find_by(<span class="st">name: </span>name)</code></pre></div>
<h3 id="ecto-1">Ecto</h3>
<div class="sourceCode"><pre class="sourceCode elixir"><code class="sourceCode elixir"><span class="cn">Repo</span><span class="op">.</span>one(from t <span class="kw">in</span> <span class="cn">App</span><span class="op">.</span><span class="cn">Model</span>, <span class="va">where:</span> t<span class="op">.</span>name <span class="op">==</span> <span class="op">^</span>name, <span class="va">limit:</span> <span class="dv">1</span>)</code></pre></div>
<h3 id="persistent-1">Persistent</h3>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">getFirstUserByEmail ::</span> <span class="dt">Text</span> <span class="ot">-></span> <span class="dt">DB</span> (<span class="dt">Maybe</span> (<span class="dt">Entity</span> <span class="dt">User</span>))
getFirstUserByEmail email <span class="fu">=</span>
selectFirst [<span class="dt">UserEmail</span> <span class="fu">==.</span> email] []</code></pre></div>
<h2 id="fetch-a-single-record-based-on-id1">Fetch a single record based on id=1</h2>
<h3 id="activerecord-2">ActiveRecord</h3>
<div class="sourceCode"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span class="dt">Model</span>.find(<span class="dv">1</span>)</code></pre></div>
<h3 id="ecto-2">Ecto</h3>
<div class="sourceCode"><pre class="sourceCode elixir"><code class="sourceCode elixir"><span class="cn">Model</span> <span class="op">|></span> <span class="cn">Repo</span><span class="op">.</span>get!(<span class="dv">1</span>)</code></pre></div>
<h3 id="persistent-2">Persistent</h3>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">getIdOneUser ::</span> <span class="dt">DB</span> (<span class="dt">Maybe</span> (<span class="dt">Entity</span> <span class="dt">User</span>))
getIdOneUser <span class="fu">=</span> getEntity (toSqlKey <span class="dv">1</span>)</code></pre></div>
<h1 id="comparing-against-the-rest-of-the-article">Comparing against the rest of the article</h1>
<p>I’ll do my usual thing and cite what the original article said, then reply with either code or prose.</p>
<blockquote>
<p>Let’s talk about the main ideas behind Ecto, and try to compare it with ActiveRecord.</p>
</blockquote>
<blockquote>
<p>Main difference ActiveRecord: We can represent data using: behaviors + state.</p>
</blockquote>
<blockquote>
<p>Ecto: We need to represent data using: functions.</p>
</blockquote>
<p>It’s just functions and data in Haskell too.</p>
<blockquote>
<p>Active Record pattern ActiveRecord has a general pattern of accessing data used in Object-oriented languages. So, this is not specfically the Active Record pattern.</p>
</blockquote>
<blockquote>
<p>Using ActiveRecord, we can do:</p>
</blockquote>
<div class="sourceCode"><pre class="sourceCode ruby"><code class="sourceCode ruby">artist = <span class="dt">Artist</span>.get(<span class="dv">1</span>)
artist.name = <span class="st">"Name"</span>
artist.save</code></pre></div>
<blockquote>
<p>This makes a lot of sense for Object-Oriented languages. Data has behavior and state. This is pretty straightforward. How does Ecto handle that?</p>
</blockquote>
<blockquote>
<p>Repository Pattern</p>
</blockquote>
<blockquote>
<p>As a functional language we don’t have data with state, nor do we have behavior. We only have functions.</p>
</blockquote>
<blockquote>
<p>In general, if you want to talk with the database, you need to talk with the Repository first.</p>
</blockquote>
<div class="sourceCode"><pre class="sourceCode elixir"><code class="sourceCode elixir">artist <span class="op">=</span> <span class="cn">Repo</span><span class="op">.</span>get(<span class="cn">Artist</span>, <span class="dv">1</span>)
changeset <span class="op">=</span> <span class="cn">Artist</span><span class="op">.</span>changeset(artist, <span class="va">name:</span> <span class="st">"Changed name"</span>)
<span class="cn">Repo</span><span class="op">.</span>update(changeset)</code></pre></div>
<blockquote>
<p>If we check side-by-side what Active Record and repository does, we cannot see when Active Record touches the Database. We just do a save and it hits the database implicitly. In Ecto, you always interact with the database explicitly.</p>
</blockquote>
<blockquote>
<p>Ecto will not talk to the database without you asking it to. Everything is totally explicit. Any interaction with the database should pass through the Repository.</p>
</blockquote>
<p>This is true of Haskell too, it only talks to the database when you ask it to. Most people have a function named <code>runDB</code> or similar which makes it easy to audit what code actually talks to the database. It also serves to tell you where your transaction boundaries are which is tremendously helpful for atomicity and correctly using your SQL database. Here’s the name-changing example above in Haskell:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">updateFirstUserName ::</span> <span class="dt">Text</span> <span class="ot">-></span> <span class="dt">DB</span> ()
updateFirstUserName newName <span class="fu">=</span> <span class="kw">do</span>
update (toSqlKey <span class="dv">1</span>) [<span class="dt">UserName</span> <span class="fu">=.</span> newName]</code></pre></div>
<p>If you wanted something that could do the same for any primary key:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">updateFirstUserName' ::</span> <span class="dt">Key</span> <span class="dt">User</span> <span class="ot">-></span> <span class="dt">Text</span> <span class="ot">-></span> <span class="dt">DB</span> ()
updateFirstUserName' userKey newName <span class="fu">=</span> <span class="kw">do</span>
update userKey [<span class="dt">UserName</span> <span class="fu">=.</span> newName]
<span class="ot">updateFirstUserName ::</span> <span class="dt">Text</span> <span class="ot">-></span> <span class="dt">DB</span> ()
updateFirstUserName <span class="fu">=</span> updateFirstUserName' (toSqlKey <span class="dv">1</span>)</code></pre></div>
<h2 id="schema">Schema</h2>
<blockquote>
<p>Schema is normally a map between your types and your database. But not necessarily.</p>
</blockquote>
<blockquote>
<p>If we check the documentation:</p>
</blockquote>
<blockquote>
<p>An Ecto schema is used to map any data source into an Elixir struct. One of such use cases is to map data coming from a repository, usually a table, into Elixir structs. An interesting thing to mention is that we don’t need a schema for using Ecto. We can bypass the use of Schemas by using the table name as a string. Schemas are very flexible.</p>
</blockquote>
<blockquote>
<p>Here is an example of Schema definition in Ecto.</p>
</blockquote>
<div class="sourceCode"><pre class="sourceCode elixir"><code class="sourceCode elixir"><span class="kw">defmodule</span> <span class="cn">SlackPosting</span><span class="op">.</span><span class="cn">Journals</span><span class="op">.</span><span class="cn">Post</span> <span class="kw">do</span>
<span class="im">use</span> <span class="cn">Ecto</span><span class="op">.</span><span class="cn">Schema</span>
<span class="im">import</span> <span class="cn">Ecto</span><span class="op">.</span><span class="cn">Changeset</span>
<span class="im">alias</span> <span class="cn">SlackPosting</span><span class="op">.</span><span class="cn">Journals</span><span class="op">.</span><span class="cn">Post</span>
schema <span class="st">"posts"</span> <span class="kw">do</span>
field <span class="va">:user_slack_id</span>, <span class="va">:string</span>
field <span class="va">:user_name</span>, <span class="va">:string</span>
field <span class="va">:text</span>, <span class="va">:string</span>
many_to_many <span class="va">:tags</span>, <span class="cn">SlackPosting</span><span class="op">.</span><span class="cn">Journals</span><span class="op">.</span><span class="cn">Tag</span>, <span class="va">join_through:</span> <span class="cn">SlackPosting</span><span class="op">.</span><span class="cn">Journals</span><span class="op">.</span><span class="cn">PostTag</span>
has_many <span class="va">:comments</span>, <span class="cn">SlackPosting</span><span class="op">.</span><span class="cn">Journals</span><span class="op">.</span><span class="cn">Comment</span>
timestamps()
<span class="kw">end</span>
<span class="ot">@doc</span> <span class="cn">false</span>
<span class="kw">def</span> changeset(%<span class="cn">Post</span>{} <span class="op">=</span> post, attrs) <span class="kw">do</span>
post
<span class="op">|></span> cast(attrs, [<span class="va">:text</span>, <span class="va">:user_slack_id</span>, <span class="va">:user_name</span>])
<span class="op">|></span> validate_required([<span class="va">:text</span>, <span class="va">:user_slack_id</span>])
<span class="kw">end</span>
<span class="kw">end</span></code></pre></div>
<p>Then in Persistent, with a little more of the mentioned tables fleshed out:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">share [mkPersist sqlSettings, mkMigrate <span class="st">"migrateAll"</span>] [persistLowerCase|
Post
userSlackId Text
userName Text
someText Text
someOtherText Text
deriving Show
Comment
comment Text
post PostId
deriving Show
Tag
tagName Text
PostTags
tag TagId
post PostId
|]</code></pre></div>
<h2 id="migrations">Migrations</h2>
<blockquote>
<p>Migrations Ecto also has migrations. This is not really different from what ActiveRecord offers to us.</p>
</blockquote>
<div class="sourceCode"><pre class="sourceCode elixir"><code class="sourceCode elixir"><span class="kw">defmodule</span> <span class="cn">SlackPosting</span><span class="op">.</span><span class="cn">Repo</span><span class="op">.</span><span class="cn">Migrations</span><span class="op">.</span><span class="cn">CreatePosts</span> <span class="kw">do</span>
<span class="im">use</span> <span class="cn">Ecto</span><span class="op">.</span><span class="cn">Migration</span>
<span class="kw">def</span> change <span class="kw">do</span>
create table(<span class="va">:posts</span>) <span class="kw">do</span>
add <span class="va">:text</span>, <span class="va">:text</span>
add <span class="va">:user_slack_id</span>, <span class="va">:string</span>
add <span class="va">:user_name</span>, <span class="va">:string</span>
timestamps()
<span class="kw">end</span>
<span class="kw">end</span>
<span class="kw">end</span></code></pre></div>
<p>Persistent does too, but approaches it differently by focusing on generating fresh and differential migrations against the database rather than creating a migration DSL. The macros generate the code necessary to see what migrations it recommends or to run the migrations directly automatically. The usual <code>runDB</code> function for running a database action against the database works for this.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">dumpMigration ::</span> <span class="dt">DB</span> ()
dumpMigration <span class="fu">=</span> printMigration migrateAll
<span class="ot">runMigrations ::</span> <span class="dt">DB</span> ()
runMigrations <span class="fu">=</span> runMigration migrateAll</code></pre></div>
<p>If we were to dump the post/comment/tag schema from earlier for SQLite, the migration would look like:</p>
<div class="sourceCode"><pre class="sourceCode sql"><code class="sourceCode sql"><span class="kw">CREATE</span> <span class="kw">TABLE</span> <span class="ot">"post"</span>(<span class="ot">"id"</span> <span class="dt">INTEGER</span> <span class="kw">PRIMARY</span> <span class="kw">KEY</span>,<span class="ot">"user_slack_id"</span> <span class="dt">VARCHAR</span> <span class="kw">NOT</span> <span class="kw">NULL</span>,<span class="ot">"user_name"</span> <span class="dt">VARCHAR</span> <span class="kw">NOT</span> <span class="kw">NULL</span>,<span class="ot">"some_text"</span> <span class="dt">VARCHAR</span> <span class="kw">NOT</span> <span class="kw">NULL</span>,<span class="ot">"some_other_text"</span> <span class="dt">VARCHAR</span> <span class="kw">NOT</span> <span class="kw">NULL</span>);
<span class="kw">CREATE</span> <span class="kw">TABLE</span> <span class="ot">"comment"</span>(<span class="ot">"id"</span> <span class="dt">INTEGER</span> <span class="kw">PRIMARY</span> <span class="kw">KEY</span>,<span class="ot">"comment"</span> <span class="dt">VARCHAR</span> <span class="kw">NOT</span> <span class="kw">NULL</span>,<span class="ot">"post"</span> <span class="dt">INTEGER</span> <span class="kw">NOT</span> <span class="kw">NULL</span> <span class="kw">REFERENCES</span> <span class="ot">"post"</span>);
<span class="kw">CREATE</span> <span class="kw">TABLE</span> <span class="ot">"tag"</span>(<span class="ot">"id"</span> <span class="dt">INTEGER</span> <span class="kw">PRIMARY</span> <span class="kw">KEY</span>,<span class="ot">"tag_name"</span> <span class="dt">VARCHAR</span> <span class="kw">NOT</span> <span class="kw">NULL</span>);
<span class="kw">CREATE</span> <span class="kw">TABLE</span> <span class="ot">"post_tags"</span>(<span class="ot">"id"</span> <span class="dt">INTEGER</span> <span class="kw">PRIMARY</span> <span class="kw">KEY</span>,<span class="ot">"tag"</span> <span class="dt">INTEGER</span> <span class="kw">NOT</span> <span class="kw">NULL</span> <span class="kw">REFERENCES</span> <span class="ot">"tag"</span>,<span class="ot">"post"</span> <span class="dt">INTEGER</span> <span class="kw">NOT</span> <span class="kw">NULL</span> <span class="kw">REFERENCES</span> <span class="ot">"post"</span>);</code></pre></div>
<h2 id="changeset">Changeset</h2>
<p>I have no idea what this is about and their post doesn’t make it clearer. Validation is orthogonal to Persistent, you usually validate stuff at the edges so that any value of type <code>MyModel</code> is only ever a valid value for that database table.</p>
<h2 id="associations">Associations</h2>
<p>We covered this a little earlier with the Post/Comment/Tag example but I’ll explain a little:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="dt">Comment</span>
comment <span class="dt">Text</span>
post <span class="dt">PostId</span>
<span class="kw">deriving</span> <span class="dt">Show</span>
<span class="dt">Tag</span>
tagName <span class="dt">Text</span>
<span class="dt">PostTags</span>
tag <span class="dt">TagId</span>
post <span class="dt">PostId</span></code></pre></div>
<p>You can reference the primary key column of a model defined elsewhere in the quasiquoter, so <code>TagId</code> is something the code understands is primary key of the <code>Tag</code> table. From there, it’s able to generate the foreign key relationships in the migrations automatically. It also gives you better type-safety with managing keys:</p>
<pre><code>Prelude> :t Comment
Comment :: Text -> Key Post -> Comment
Prelude> let tagKey :: Key Tag; tagKey = toSqlKey 1
Prelude> Comment "my comment" tagKey
<interactive>:12:22: error:
• Couldn't match type ‘Tag’ with ‘Post’
Expected type: Key Post
Actual type: Key Tag
• In the second argument of ‘Comment’, namely ‘tagKey’
In the expression: Comment "my comment" tagKey
In an equation for ‘it’: it = Comment "my comment" tagKey</code></pre>
<p>When our keys aren’t just strings or numbers, we can avoid a lot of unnecessary mistakes!</p>
<h2 id="lazy-loading">Lazy loading</h2>
<blockquote>
<p>Ecto does not support Lazy Loading.</p>
</blockquote>
<p>Persistent doesn’t either. If you want to pull related data together you can do so via separate database actions in a transaction or you can use <a href="https://github.com/bitemyapp/esqueleto">Esqueleto</a> to do in so:</p>
<div class="sourceCode"><pre class="sourceCode elixir"><code class="sourceCode elixir"><span class="kw">def</span> list_posts <span class="kw">do</span>
<span class="cn">Repo</span><span class="op">.</span>all(<span class="cn">Post</span>)
<span class="op">|></span> <span class="cn">Repo</span><span class="op">.</span>preload([<span class="va">:comments</span>, <span class="va">:tags</span>])
<span class="kw">end</span></code></pre></div>
<p>Here’s a somewhat serious (this is modeled after some production code I wrote) example of how to do this with Esqueleto on top of Persistent, by returning a mapping of posts, the comments on the posts, and all tags associated with the posts.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">tagsForPosts ::</span> [<span class="dt">Key</span> <span class="dt">Post</span>] <span class="ot">-></span> <span class="dt">DB</span> [(<span class="dt">Key</span> <span class="dt">Post</span>, <span class="dt">Entity</span> <span class="dt">Tag</span>)]
tagsForPosts postKeys <span class="fu">=</span>
unValueThePostKeys <span class="fu">$</span>
select <span class="fu">$</span>
from <span class="fu">$</span> \ ( postTag <span class="ot">`InnerJoin`</span> tag ) <span class="ot">-></span> <span class="kw">do</span>
on (tag <span class="fu">^.</span> <span class="dt">TagId</span>
<span class="fu">E.==.</span> postTag <span class="fu">^.</span> <span class="dt">PostTagTag</span>)
where_ (postTag <span class="fu">^.</span> <span class="dt">PostTagPost</span>
<span class="ot">`in_`</span> valList postKeys)
return (postTag <span class="fu">^.</span> <span class="dt">PostTagPost</span>, tag)
<span class="kw">where</span><span class="ot"> unValueThePostKeys ::</span> <span class="dt">DB</span> [(<span class="dt">E.Value</span> (<span class="dt">Key</span> <span class="dt">Post</span>), <span class="dt">Entity</span> <span class="dt">Tag</span>)]
<span class="ot">-></span> <span class="dt">DB</span> [(<span class="dt">Key</span> <span class="dt">Post</span>, <span class="dt">Entity</span> <span class="dt">Tag</span>)]
unValueThePostKeys <span class="fu">=</span> (fmap <span class="fu">.</span> fmap) (first E.unValue)
<span class="ot">postsWithCommentsAndTags ::</span> <span class="dt">DB</span> (<span class="dt">Map</span>
(<span class="dt">Key</span> <span class="dt">Post</span>)
(<span class="dt">Entity</span> <span class="dt">Post</span>, [<span class="dt">Entity</span> <span class="dt">Comment</span>], [<span class="dt">Entity</span> <span class="dt">Tag</span>]))
postsWithCommentsAndTags <span class="fu">=</span> <span class="kw">do</span>
postsAndComments <span class="ot"><-</span> posts
<span class="kw">let</span> postKeys <span class="fu">=</span> fmap (entityKey <span class="fu">.</span> fst) postsAndComments
postKeysWithTags <span class="ot"><-</span> tagsForPosts postKeys
<span class="kw">let</span> initialMap <span class="fu">=</span> postsInitialMap postsAndComments
postsWithTags <span class="fu">=</span> addTagsToMap postKeysWithTags initialMap
return postsWithTags
<span class="kw">where</span>
posts <span class="fu">=</span>
select <span class="fu">$</span>
from <span class="fu">$</span> \ ( post
<span class="ot">`InnerJoin`</span>
comment ) <span class="ot">-></span> <span class="kw">do</span>
on (post <span class="fu">^.</span> <span class="dt">PostId</span>
<span class="fu">E.==.</span> (comment <span class="fu">^.</span> <span class="dt">CommentPost</span>))
return (post, comment)
<span class="ot"> postsInitialMap ::</span> [(<span class="dt">Entity</span> <span class="dt">Post</span>, <span class="dt">Entity</span> <span class="dt">Comment</span>)]
<span class="ot">-></span> <span class="dt">Map</span> (<span class="dt">Key</span> <span class="dt">Post</span>) (<span class="dt">Entity</span> <span class="dt">Post</span>, [<span class="dt">Entity</span> <span class="dt">Comment</span>], [<span class="dt">Entity</span> <span class="dt">Tag</span>])
postsInitialMap postsAndComments <span class="fu">=</span>
foldl' insertPostCom M.empty postsAndComments
<span class="kw">where</span> insertPostCom m (post, comment) <span class="fu">=</span>
M.insertWith
(\ _ (post, comments, tags) <span class="ot">-></span> (post, comment <span class="fu">:</span> comments, tags))
(entityKey post) (post, [comment], []) m
<span class="ot"> addTagsToMap ::</span> [(<span class="dt">Key</span> <span class="dt">Post</span>, <span class="dt">Entity</span> <span class="dt">Tag</span>)]
<span class="ot">-></span> <span class="dt">Map</span> (<span class="dt">Key</span> <span class="dt">Post</span>) (<span class="dt">Entity</span> <span class="dt">Post</span>, [<span class="dt">Entity</span> <span class="dt">Comment</span>], [<span class="dt">Entity</span> <span class="dt">Tag</span>])
<span class="ot">-></span> <span class="dt">Map</span> (<span class="dt">Key</span> <span class="dt">Post</span>) (<span class="dt">Entity</span> <span class="dt">Post</span>, [<span class="dt">Entity</span> <span class="dt">Comment</span>], [<span class="dt">Entity</span> <span class="dt">Tag</span>])
addTagsToMap postKeysTags initialMap <span class="fu">=</span>
foldl' insertPostKeyTag initialMap postKeysTags
<span class="kw">where</span><span class="ot"> insertPostKeyTag ::</span> <span class="dt">Map</span> (<span class="dt">Key</span> <span class="dt">Post</span>)
(<span class="dt">Entity</span> <span class="dt">Post</span>, [<span class="dt">Entity</span> <span class="dt">Comment</span>], [<span class="dt">Entity</span> <span class="dt">Tag</span>])
<span class="ot">-></span> (<span class="dt">Key</span> <span class="dt">Post</span>, <span class="dt">Entity</span> <span class="dt">Tag</span>)
<span class="ot">-></span> <span class="dt">Map</span> (<span class="dt">Key</span> <span class="dt">Post</span>)
(<span class="dt">Entity</span> <span class="dt">Post</span>, [<span class="dt">Entity</span> <span class="dt">Comment</span>], [<span class="dt">Entity</span> <span class="dt">Tag</span>])
insertPostKeyTag m (postKey, tagEntity) <span class="fu">=</span>
M.adjust
(\(post, comment, tags) <span class="ot">-></span>
(post, comment, tagEntity <span class="fu">:</span> tags))
postKey
m</code></pre></div>
<p>Then running this code with some test data:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">migrateFixturesTest ::</span> <span class="dt">IO</span> ()
migrateFixturesTest <span class="fu">=</span> <span class="kw">do</span>
runDB <span class="fu">$</span> <span class="kw">do</span>
runMigrations
pk1 <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">Post</span> <span class="st">"slack1"</span> <span class="st">"name1"</span> <span class="st">""</span> <span class="st">""</span>
pk2 <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">Post</span> <span class="st">"slack2"</span> <span class="st">"name2"</span> <span class="st">""</span> <span class="st">""</span>
pk3 <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">Post</span> <span class="st">"slack2"</span> <span class="st">"name3"</span> <span class="st">""</span> <span class="st">""</span>
_ <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">Comment</span> <span class="st">"pk1 c1"</span> pk1
_ <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">Comment</span> <span class="st">"pk1 c2"</span> pk1
_ <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">Comment</span> <span class="st">"pk2 c3"</span> pk2
tg1 <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">Tag</span> <span class="st">"tag1"</span>
ptg1 <span class="ot"><-</span> insert <span class="fu">$</span> <span class="dt">PostTag</span> tg1 pk1
pwcat <span class="ot"><-</span> postsWithCommentsAndTags
liftIO <span class="fu">$</span> pPrint pwcat</code></pre></div>
<p>We get the following output, which looks right!</p>
<pre><code>Prelude> migrateFixturesTest
Migrating: CREATE TABLE "post"("id" INTEGER PRIMARY KEY,"user_slack_id" VARCHAR NOT NULL,"user_name" VARCHAR NOT NULL,"some_text" VARCHAR NOT NULL,"some_other_text" VARCHAR NOT NULL)
Migrating: CREATE TABLE "comment"("id" INTEGER PRIMARY KEY,"comment" VARCHAR NOT NULL,"post" INTEGER NOT NULL REFERENCES "post")
Migrating: CREATE TABLE "tag"("id" INTEGER PRIMARY KEY,"tag_name" VARCHAR NOT NULL)
Migrating: CREATE TABLE "post_tag"("id" INTEGER PRIMARY KEY,"tag" INTEGER NOT NULL REFERENCES "tag","post" INTEGER NOT NULL REFERENCES "post")
fromList
[ ( PostKey { unPostKey = SqlBackendKey { unSqlBackendKey = 1 } }
, ( Entity
{ entityKey =
PostKey { unPostKey = SqlBackendKey { unSqlBackendKey = 1 } }
, entityVal =
Post
{ postUserSlackId = "slack1"
, postUserName = "name1"
, postSomeText = ""
, postSomeOtherText = ""
}
}
, [ Entity
{ entityKey =
CommentKey { unCommentKey = SqlBackendKey { unSqlBackendKey = 2 } }
, entityVal =
Comment
{ commentComment = "pk1 c2"
, commentPost =
PostKey { unPostKey = SqlBackendKey { unSqlBackendKey = 1 } }
}
}
, Entity
{ entityKey =
CommentKey { unCommentKey = SqlBackendKey { unSqlBackendKey = 1 } }
, entityVal =
Comment
{ commentComment = "pk1 c1"
, commentPost =
PostKey { unPostKey = SqlBackendKey { unSqlBackendKey = 1 } }
}
}
]
, [ Entity
{ entityKey =
TagKey { unTagKey = SqlBackendKey { unSqlBackendKey = 1 } }
, entityVal = Tag { tagTagName = "tag1" }
}
]
)
)
, ( PostKey { unPostKey = SqlBackendKey { unSqlBackendKey = 2 } }
, ( Entity
{ entityKey =
PostKey { unPostKey = SqlBackendKey { unSqlBackendKey = 2 } }
, entityVal =
Post
{ postUserSlackId = "slack2"
, postUserName = "name2"
, postSomeText = ""
, postSomeOtherText = ""
}
}
, [ Entity
{ entityKey =
CommentKey { unCommentKey = SqlBackendKey { unSqlBackendKey = 3 } }
, entityVal =
Comment
{ commentComment = "pk2 c3"
, commentPost =
PostKey { unPostKey = SqlBackendKey { unSqlBackendKey = 2 } }
}
}
]
, []
)
)
]</code></pre>
<p>This stuff could get abstracted away or code-gen’d but I haven’t had cause to bother yet. Incidentally, this happens to be a decent example of how to pull together data associated by one-to-many and many-to-many relationships using Esqueleto and Persistent.</p>
<p>If you’d like a working, running git repository of what I did in this article, <a href="https://github.com/bitemyapp/persistent-activerecord-ecto">take a look here</a>.</p>
<p>Some relevant libraries:</p>
<ul>
<li><a href="https://github.com/yesodweb/persistent">Persistent</a></li>
<li><a href="https://github.com/yesodweb/persistent-sqlite">Persistent SQLite</a></li>
<li><a href="https://github.com/yesodweb/persistent-template">Persistent code-gen</a></li>
<li><a href="https://github.com/bitemyapp/esqueleto">Esqueleto</a>, has the ability to perform joins and otherwise compose relational queries, whereas Persistent is limited to basic CRUD.</li>
<li><a href="https://github.com/yav/pretty-show">pretty-show</a> has a nice pretty printer I used in my code.</li>
</ul>
</div>
<div class="blurb">
<p>
I know this site is a bit of a disaster zone, but if you like my writing or think you could learn something useful from me, please <a href="http://haskellbook.com/">take a look at the Haskell book</a> I've been writing. There's a free sample available too!
</p>
</div>
<div class="footer">
<p>
Posted on October 6, 2017
</p>
</div>Fri, 06 Oct 2017 00:00:00 +0000Noam Lewis: Type-safe enums in C, using a clang pluginhttp://noamlewis.wordpress.com/?p=638
https://noamlewis.wordpress.com/2017/10/05/type-safe-enums-in-c-using-a-clang-plugin/
<p>The C programming language generally treats enums as integers (see “Appendix: For Language Lawyers” for reference).</p>
<p>Wouldn’t it be nice if we could do more with enums, and do it safely?</p>
<p>Some other languages have anything from integer-incompatible enums to full-blown sum types. It would be nice to have something like that in C.</p>
<p>I wrote the <a href="https://github.com/sinelaw/elfs-clang-plugins">enums_conversion clang plugin</a> aiming to do just that, by treating enums as incompatible with integers (except via explicit casting).</p>
<h2><a href="https://github.com/sinelaw/elfs-clang-plugins/blob/docs/enums_conversion/README.md#a-motivating-example" id="user-content-a-motivating-example" class="anchor"></a>A motivating example</h2>
<p>Some people are surprised at the goals of this plugin. Here is a simple example to explain the motivation.</p>
<p>Consider the following (totally fabricated) API:</p>
<pre><code>enum OpGetResult {
OP_GET_ERROR,
OP_GET_OK,
};
enum OpGetResult get_items(void);
/* Implementation: */
enum OpGetResult get_items(void)
{
/* ... do something with side effects ... */
return OP_GET_OK;
}
</code></pre>
<p>So far so good. Safe it as <code>test.c</code> and compile this program with gcc:</p>
<pre><code>gcc -std=c11 -Wall -Wextra -Werror -c test.c
</code></pre>
<p>No errors, yay!</p>
<h3><a href="https://github.com/sinelaw/elfs-clang-plugins/blob/docs/enums_conversion/README.md#a-simple-bug" id="user-content-a-simple-bug" class="anchor"></a>A simple bug</h3>
<p>Now, let’s introduce a bug. Someone decided the API is no good, and <code>get_items</code> should just return the number of items it “got”. So the new API is:</p>
<pre><code>/* This enum is in use elsewhere... */
enum OpGetResult {
OP_GET_ERROR,
OP_GET_OK,
};
int get_items(void); /* return value changed to 'int' */
/* Implementation: */
int get_items(void)
{
/* ... do something with side effects ... */
return OP_GET_OK; /* oops! forgot to change this */
}
</code></pre>
<p>The bug is that <code>get_items</code> still returns the enum value <code>OP_GET_OK</code> instead of a number.</p>
<p>Save as test2.c and compile (tested on gcc 6.3.0):</p>
<pre><code>gcc -std=c11 -Wall -Wextra -Werror -c test2.c
</code></pre>
<p>Oh no! No error! Let’s try with clang 5.0 and the wonderful <code>-Weverything</code> which enables all warnings:</p>
<pre><code>clang -std=c11 -Weverything -Werror -c test2.c
</code></pre>
<p>Nope! Still no error.</p>
<p>The compilers are ok with this code <em>because it’s allowed</em>. However, it’s clearly not what we intended.</p>
<h3><a href="https://github.com/sinelaw/elfs-clang-plugins/blob/docs/enums_conversion/README.md#a-bunch-of-other-possible-bugs" id="user-content-a-bunch-of-other-possible-bugs" class="anchor"></a>A bunch of other possible bugs</h3>
<p>Here is a snippet with different ‘bad code’ examples: (for testing it can be appended to one of the previous files)</p>
<pre><code>int func(enum OpGetResult e, unsigned int x, unsigned int y);
int func(enum OpGetResult e, unsigned int x, unsigned int y)
{
handle_result(x); /* passing arbitrary integer where one of several enum values was expected */
enum OpGetResult e2 = x; /* assigning from arbitrary integer (which may not be a valid enum value) */
if (e2 == y) { /* comparing enum to arbitrary integer */
}
return e; /* returning enum where arbitrary integer is expected by caller */
}
</code></pre>
<p>Neither gcc 6.3.0 nor clang 5.0 emit any kind of warning about the above code.</p>
<pre><code># Let's try gcc with some extra warnings:
gcc -std=c11 -Wall -Wextra -Werror -Wconversion -Wenum-compare -Wswitch-enum -Wsign-conversion -c test2.c
# clang with -Weverything:
clang -std=c11 -Weverything -Werror -c test2.c
</code></pre>
<h3><a href="https://github.com/sinelaw/elfs-clang-plugins/blob/docs/enums_conversion/README.md#clang-plugin-to-the-rescue" id="user-content-clang-plugin-to-the-rescue" class="anchor"></a>clang plugin to the rescue</h3>
<p>The <code>enums_converesion</code> clang plugin detects and warns about all of the above.</p>
<pre><code># clang -std=c11 -Weverything -c test2.c -Xclang -load -Xclang ./clang_plugins.so -Xclang -add-plugin -Xclang enums_conversion
test2.c:22:23: error: enum conversion to or from enum OpGetResult
handle_result(x); /* passing arbitrary integer where one of several enum values was expected */
^
test2.c:24:31: error: enum conversion to or from enum OpGetResult
enum OpGetResult e2 = x; /* assigning from arbitrary integer (which may not be a valid enum value) */
^
test2.c:26:13: error: enum conversion to or from enum OpGetResult
if (e2 == y) { /* comparing enum to arbitrary integer */
^
test2.c:29:16: error: enum conversion to or from enum OpGetResult
return e; /* returning enum where arbitrary integer is expected by caller */
^
4 errors generated.
</code></pre>
<h2><a href="https://github.com/sinelaw/elfs-clang-plugins/blob/docs/enums_conversion/README.md#frequently-asked-questions" id="user-content-frequently-asked-questions" class="anchor"></a>Frequently Asked Questions</h2>
<ol>
<li>But this isn’t standard C!</li>
</ol>
<p>Correct, it is a <em>restrictive subset</em> of C. Some “valid” C programs will be flagged by this plugin. I believe writing code in the spirit of this plugin will improve your code’s readability while preventing a class of bugs from ever occurring.</p>
<ol start="2">
<li>How is this different from gcc’s <code>-Wenum-compare</code>?</li>
</ol>
<p>The warning flag <code>-Wenum-compare</code> find comparisons between different enums, but does not look at comparing enums to integers, implicit casting to/from integers, etc. In the following program only the second <code>if</code> is flagged by <code>-Wenum-compare</code>:</p>
<pre><code>enum A { A_FIRST, A_SECOND };
enum B { B_FIRST, B_SECOND };
int foo(enum A a, unsigned int x);
int foo(enum A a, unsigned int x) {
if (x == a) { // no warning emitted
return 1;
}
if (B_FIRST == a) { // will cause warning: comparison between ‘enum B’ and ‘enum A’
return 2;
}
return 0;
}
</code></pre>
<ol start="3">
<li>How is this different from clang’s <code>-Wenum-conversion</code>?</li>
</ol>
<p><code>-Wenum-conversion</code> doesn’t catch implicit casts to/from integral types (the plugin does).</p>
<p><code>-Wenum-conversion</code> does catch conversion from one enum type to another, like so:</p>
<pre><code>enum EnumA { E_A };
enum EnumB { E_B };
enum EnumA do_something(void) {
return E_B;
}
</code></pre>
<ol start="4">
<li>What about enums being used as combinable bits? Won’t the plugin disallow them?</li>
</ol>
<p>A common pattern is using an enum to describe the allowed bits for an “options” value that can be ORed together. For example:</p>
<pre><code>enum Flags {
FLAG_NONE = 0,
FLAG_READ = 1,
FLAG_WRITE = 2,
};
enum Flags do_something(void);
enum Flags do_something(void) {
return FLAG_WRITE | FLAG_READ;
}
</code></pre>
<p>The plugin is OK with this. clang -Weverything doesn’t like this (-Wassign-enum):</p>
<pre><code>clang -std=c11 -c /tmp/test.c -Weverything
/tmp/test.c:8:12: warning: integer constant not in range of enumerated type 'enum Flags' [-Wassign-enum]
return FLAG_WRITE | FLAG_READ;
^
1 warning generated.
</code></pre>
<p>That’s a false error (if you use | with a runtime variable, <code>-Wassign-enum</code> seems to not flag this). However, the plugin does catch errors of putting an invalid value in the OR expression:</p>
<pre><code>...
return FLAG_WRITE | 5;
</code></pre>
<p>Now clang -Weverything doesn’t complain (despite the possible bug).</p>
<p>Running with the plugin gives:</p>
<pre><code>/tmp/test.c:10:16: error: enum conversion to or from enum Flags
return FLAG_WRITE | 5;
</code></pre>
<ol start="5">
<li>I’m afraid to use this in production.</li>
</ol>
<p>The plugin only analyzes the AST produced by clang, and does not affect the emitted code in any way.</p>
<ol start="6">
<li>I don’t use clang! Can I benefit from this plugin?</li>
</ol>
<p>At <a href="http://elastifile.com/">elastifile</a>, the plugin is being used as part of the CI process. Code that is being merged into master must pass the plugin’s checks (as well as other plugins from this suite). The actual production executable is built by gcc (for various unrelated reasons).</p>
<p>The plugin is available as part of the <a href="https://github.com/sinelaw/elfs-clang-plugins">elfs-clang-plugins suite github.</a></p>
<h2><a href="https://github.com/sinelaw/elfs-clang-plugins/blob/docs/enums_conversion/README.md#appendix-for-language-lawyers" id="user-content-appendix-for-language-lawyers" class="anchor"></a>Appendix: For Language Lawyers</h2>
<p>The <a href="http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf">C11 standard (draft)</a> says:</p>
<blockquote><p>An enumeration comprises a set of named integer constant values. Each distinct enumeration constitutes a different enumerated type. The type char, the signed and unsigned integer types, and the enumerated types are collectively called integer types…</p></blockquote><br /> <a href="http://feeds.wordpress.com/1.0/gocomments/noamlewis.wordpress.com/638/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/noamlewis.wordpress.com/638/" alt="" border="0" /></a> <a href="http://feeds.wordpress.com/1.0/godelicious/noamlewis.wordpress.com/638/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/delicious/noamlewis.wordpress.com/638/" alt="" border="0" /></a> <a href="http://feeds.wordpress.com/1.0/gofacebook/noamlewis.wordpress.com/638/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/facebook/noamlewis.wordpress.com/638/" alt="" border="0" /></a> <a href="http://feeds.wordpress.com/1.0/gotwitter/noamlewis.wordpress.com/638/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/twitter/noamlewis.wordpress.com/638/" alt="" border="0" /></a> <a href="http://feeds.wordpress.com/1.0/gostumble/noamlewis.wordpress.com/638/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/stumble/noamlewis.wordpress.com/638/" alt="" border="0" /></a> <a href="http://feeds.wordpress.com/1.0/godigg/noamlewis.wordpress.com/638/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/digg/noamlewis.wordpress.com/638/" alt="" border="0" /></a> <a href="http://feeds.wordpress.com/1.0/goreddit/noamlewis.wordpress.com/638/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/reddit/noamlewis.wordpress.com/638/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=noamlewis.wordpress.com&blog=12721713&post=638&subd=noamlewis&ref=&feed=1" alt="" height="1" border="0" width="1" />Thu, 05 Oct 2017 19:10:28 +0000Tweag I/O: How to compose streaming programshttp://www.tweag.io/posts/2017-10-05-streaming2.html
http://www.tweag.io/posts/2017-10-05-streaming2.html
<div>Facundo Domínguez</div><p>In our first <a href="http://www.tweag.io/posts/2017-07-27-streaming-programs.html">blog post about streaming</a>,
we discussed how streaming libraries strengthen types to catch more
errors. However, when we strengthen types, we need to be careful to
not hinder program composition. After all, precise type information
can make it more onerous to convince the compiler that a particular
program is well formed. In this post, we show that
streaming libraries handle this issue well,
as they allow programs to be composed conveniently. To illustrate this point,
we will discuss the most important composition forms for streaming programs and illustrate the explained concepts
at the example of four concrete streaming libraries, namely
<a href="http://hackage.haskell.org/package/pipes">pipes</a>,
<a href="http://hackage.haskell.org/package/conduit">conduit</a>,
<a href="http://hackage.haskell.org/package/streaming">streaming</a>,
and
<a href="http://hackage.haskell.org/package/io-streams">io-stream</a>. This ensures that the discussed concepts are of a general
nature and not specific to just one particular library.</p>
<h1>Composition forms</h1>
<p>Generally speaking, streaming libraries offer two flavors of composition:
<em>appending</em> and <em>pipelining</em>. When appending, the values produced by one
program are being followed by the values produced by another.
When pipelining, some values produced by one streaming program are
processed by another while the first program still has more values to
produce.</p>
<p>Different libraries implement these forms of composition in different
ways. Some streaming libraries have a first-class notion of <em>stream
processors</em>, not streams. Generally, a stream processor
<code>SP i o m r</code> gets a stream of values of type
<code>i</code>, yields a stream of values of type <code>o</code>, and finally yields a value of
type <code>r</code> when it terminates. The parameter <code>m</code> represents the monad
in which the effects of the processors are sequenced. For example, we have,</p>
<pre><code class="language-Haskell">conduit: ConduitM i o m r
pipes: Pipe i o m r
</code></pre>
<p><code>pipes</code> and <code>conduit</code>
define the monad bind operation (<code>>>=</code>) as an appending
composition. In <code>p >>= \r -> q</code>, the input stream is first passed to
<code>p</code>, and when <code>p</code> completes the unconsumed elements from the input
stream are passed to <code>q</code>.
As a result, the output stream of <code>p</code> is followed by the output stream
of <code>q</code>.</p>
<p>In contrast to <code>pipes</code> and <code>conduit</code>, the
<code>streaming</code> and <code>io-stream</code> packages manipulate streams directly.
Consequently, there is no notion of an <em>input stream</em>. Still, monadic
composition works in a similar manner in the case of the <code>streaming</code> package. The
stream <code>p >>= \r -> q</code> starts with the <em>output stream</em> of values produced
by <code>p</code>, followed by the <em>output stream</em> of values produced by <code>q</code>. The involved types are</p>
<pre><code class="language-Haskell">streaming: Stream (Of a) m r
io-streams: InputStream a
</code></pre>
<p>Values of type <code>Stream (Of a) m r</code> are streams of values of type <code>a</code>,
which might be produced by performing effectful computations in the
monad <code>m</code>. When there are no more elements of type <code>a</code>, a value of type
<code>r</code> is produced. Values of type <code>InputStream a</code> are also streams of
values of type <code>a</code>, and values might be produced by performing
computations in the monad <code>IO</code>.</p>
<p>Although the package <code>io-stream</code> does not define a monad instance (and hence, does not overload <code>(>>=)</code>) for
<code>InputStream</code>, it provides a separate function for the purpose of concatenating streams:</p>
<pre><code class="language-Haskell">appendInputStream :: InputStream a -> InputStream a -> IO (InputStream a)
</code></pre>
<p>The second form of composition, namely pipelining, is realised by the following two combinators in the libraries based on
stream processors:</p>
<pre><code class="language-Haskell">conduit: (.|) :: Monad m => ConduitM a b m () -> ConduitM b c m r -> ConduitM a c m r
pipes: (>->) :: Monad m => Pipe a b m r -> Pipe b c m r -> Pipe a c m r
</code></pre>
<p>In <code>p .| q</code>, the output stream of <code>p</code> is fed into <code>q</code> as its input
stream and so it happens with <code>p >-> q</code> as well. But there is a
difference in the types of the left operand.
In <code>conduit</code>, the left operand yields <code>()</code> when it terminates. The right
operand is free to continue producing values in the output stream for as
long as it wants. Not so in <code>pipes</code>, where the termination of the left
operand produces a value of type <code>r</code> and causes the composition to
terminate and yield the same value.</p>
<p>In <code>streaming</code> and <code>io-streams</code>, pipelines are constructed
by composing functions which transform streams. For instance</p>
<pre><code class="language-Haskell">streaming: map :: (a -> b) -> Stream (Of a) m r -> Stream (Of b) m r
io-streams: decompress :: InputStream ByteString -> IO (InputStream ByteString)
</code></pre>
<p>The function
<a href="http://hackage.haskell.org/package/io-streams-1.5.0.1/docs/System-IO-Streams-Zlib.html#v:decompress">decompress</a>
takes an input stream of bytestrings carrying compressed data in the
zlib format and outputs a stream of bytestrings carrying the result of
decompression.</p>
<h2>Leftovers and parsing</h2>
<p>Most streaming libraries offer the ability to push back a value from
the input. The returned inputs are sometimes called <em>leftovers</em>. This facilitates
looking ahead in the input stream to peek at some elements needed to decide on alternative behaviors of
the program.</p>
<p>To illustrate the need for looking ahead, consider the following <code>conduit</code> program, which acts as a parser. It yields the number
of <code>'a'</code> characters in the input followed by the number of <code>'b'</code> characters.</p>
<pre><code class="language-Haskell">abConduit :: Monad m => Consumer Char m (Int, Int)
abConduit =
(,) <$> (takeWhileC (== 'a') .| lengthC)
<*> (takeWhileC (== 'b') .| lengthC)
</code></pre>
<p>As <code>conduit</code> implements leftovers, the first <code>takeWhileC</code> can return an
element to the input stream upon discovering that it isn't an <code>'a'</code>
character. Then the following <code>takeWhileC</code> has the opportunity to
examine that same element again.</p>
<p>However, it is important to note that pipelining composition only allows the left operand to put values
back. That is, in <code>p .| q</code>, only <code>p</code> can return values to
the input stream; <code>q</code>'s leftovers are ignored. This is a constraint
stemming from the type of the composition:</p>
<pre><code class="language-Haskell">(.|) :: Monad m => ConduitM a b m () -> ConduitM b c m r -> ConduitM a c m r
</code></pre>
<p>The input stream provides values of type <code>a</code>, but the leftovers of the
right operand are of type <code>b</code>. To address this, <code>conduit</code> offers the
following composition operator that resolves the type mismatch with
a conversion function.</p>
<pre><code class="language-Haskell">fuseLeftovers :: Monad m => ([b] -> [a]) -> ConduitM a b m () -> ConduitM b c m r -> ConduitM a c m r
</code></pre>
<p>In contrast to <code>conduit</code>, the <code>pipes</code> package does not implement leftovers, but there is an additional
package <a href="http://hackage.haskell.org/package/pipes-parse">pipes-parse</a>,
which defines <code>Parser</code>s in terms of the core
abstraction of <code>Pipe</code>s. <code>Parser</code>s in turn use lenses of the input to
return values back. This is what an implementation with <code>pipes-parse</code>
looks like for our example.</p>
<pre><code class="language-Haskell">abPipes :: Monad m => Pipes.Parse.Parser Text m (Int, Int)
abPipes =
(,) <$> zoom (Pipes.Text.span (== 'a')) lengthP
<*> zoom (Pipes.Text.span (== 'b')) lengthP
lengthP :: (Monad m, Num n) => Pipes.Parse.Parser Text m n
lengthP =
Pipes.Parse.foldAll (\n txt -> n + fromIntegral (Text.length txt)) 0 id
</code></pre>
<p>The function <code>zoom</code> is a primitive offered by packages implementing
lenses. It changes the state of a stateful computation in a given scope,
and a lens explains how the state is changed.</p>
<pre><code class="language-Haskell">zoom :: Monad m => Lens' a b -> StateT b m c -> StateT a m c
</code></pre>
<p>In the case of <code>zoom (Pipes.Text.span (== 'a')) p</code>, it changes the input
fed to the parser <code>p</code> by only giving the greatest prefix containing
<code>'a'</code>s. The rest of the input, even the first character failing the
predicate, is left for the next parser.</p>
<p>The parser <code>lengthP</code> is defined to count the amount of characters in the
input. Thus</p>
<pre><code class="language-Haskell">zoom (Pipes.Text.span (== 'a')) lengthP :: Parser Text m Int
</code></pre>
<p>stands for the same computation as</p>
<pre><code class="language-Haskell">takeWhileC (== 'a') .| lengthC :: Consumer Char m Int
</code></pre>
<p>Turning our attention to the packages directly manipulating streams,
<code>Stream</code>s from <code>streaming</code> provide a function <code>cons</code> to push values
back into a stream. The package <code>io-streams</code> provides <code>unRead</code>, for
the same purpose. While the <code>cons</code> function just constructs a new
stream, the function <code>unRead</code> modifies the input stream. We have,</p>
<pre><code class="language-Haskell">cons :: Monad m => a -> Stream (Of a) m r -> Stream (Of a) m r
unRead :: a -> InputStream a -> IO ()
</code></pre>
<p>More generally, you may wonder, whether streaming libraries should try to do the job of parsers, for which the
Haskell ecosystem already provides plenty of alternatives? To answer this question, consider that parsing
libraries like
<a href="http://hackage.haskell.org/package/parsec">parsec</a> or
<a href="http://hackage.haskell.org/package/attoparsec">attoparsec</a>
do not place any bounds on their memory consumption. In particular, the parsers of neither library
return a result value before the entire input has been parsed. For instance,
if a parser is parsing a list, it can't yield the first elements of the
list as they are discovered. Instead, the list is handed back to the caller only
when all of it resides in memory. The <code>attoparsec</code> package is even more
problematic, as it
<a href="https://hackage.haskell.org/package/attoparsec-0.13.2.0/docs/Data-Attoparsec-ByteString.html#g:2">retains all of the input</a>,
no matter what the parser does.</p>
<p>More sophisticated parsing libraries can do better; e.g.,
<a href="http://foswiki.cs.uu.nl/foswiki/HUT/ParserCombinators">uu-parsinglib</a>.
This package, however, returns the parser results lazily
and so it is a challenge to test whether specific parts of the
result are available without blocking the program. Overall, this means that to parse unbounded streams of input in
a bounded amount of memory, we do actually require the streaming libraries to provides the discussed parser functionality.
After all, they provide a crucial resource usage guarantee that we cannot obtain from conventional parser libraries.</p>
<h1>Folds and non-linear uses of streams</h1>
<p>Finally, we need to discuss reductions, where we consume a stream to compute a non-stream result. For example, consider
the following naive and incorrect attempt at computing the average of a stream of values:</p>
<pre><code class="language-Haskell">average0 :: Stream (Of Double) IO () -> IO Double
average0 xs = (/) <$> Streaming.Prelude.sum_ xs
<*> (fromIntegral <$> Streaming.Prelude.length_ xs)
</code></pre>
<p>The problem with this program is that the same stream is traversed
twice. Even if the stream is not effectful, this program does not run in
bounded memory. If the stream is, however, effectful, the situation is even worse, as each traversal
produces different results leading to an inconsistent computation. To fix these problems, we need to compute the length
and the average in a single pass.</p>
<pre><code class="language-Haskell">average1 :: Stream (Of Double) IO () -> IO Double
average1 xs = uncurry (/) <$>
Streaming.Prelude.fold_ (\(!s, !c) d -> (s + 1, c + 1)) (0, 0) id xs
</code></pre>
<p>With <code>conduit</code> and <code>pipes</code> both the mistake as well as the required fix are
similar. Thus, streaming libraries seem to require that the programmer takes care to only
consume streaming sources in linear manner, that is, at most once in
a program.</p>
<p>Interestingly, <code>conduit</code> offers an alternative solution, where existing
stream processors can be reused and combined using <code>ZipSink</code>s instead of
explicitly using a <code>fold</code>. A <code>ZipSink</code> is a combination of stream
processors, all of which are fed the same input stream, and a final
result is produced from the outputs of all the output streams. This results in the following solution to our example,
which is quite close to the original naive attempt to solving the problem:</p>
<pre><code class="language-Haskell">average2 :: Monad m => Consumer Double m Double
average2 = toConsumer $
getZipSink ((/) <$> ZipSink sumC <*> fmap fromIntegral (ZipSink lengthC))
</code></pre>
<p>Facilities, such as this, are one of advantages of libraries based on stream processors as opposed to those manipulating
streams directly.
<code>ZipSink</code>s are an incarnation of the ideas in the
<a href="https://hackage.haskell.org/package/foldl">foldl</a> package for stream
processors, while the ecosystem of <code>pipes</code> offers the
<a href="https://hackage.haskell.org/package/pipes-transduce">pipes-transduce</a> package.</p>
<h1>Summary</h1>
<p>While strengthening types to catch more errors, streaming libraries are still sufficiently
expressive to allow for many data flow patterns of streaming programs.
The basic compositional forms are complemented by additional concepts to cover even more
patterns. Leftovers, for instance, support parsing
challenges that cannot be easily delegated to standard parsing libraries while still keeping
an upper bound on the consumed memory. In turn, <code>ZipSink</code>s simplify the sharing of a single input stream
by multiple stream processors.</p>Thu, 05 Oct 2017 00:00:00 +0000Jeremy Gibbons: Metamorphismshttp://patternsinfp.wordpress.com/?p=288
https://patternsinfp.wordpress.com/2017/10/04/metamorphisms/
<p>
It appears that I have insufficient time, or at least insufficient discipline, to contribute to this blog, except when I am on sabbatical. Which I now am… so let’s see if I can do better.</p>
<p></p><h2> Hylomorphisms </h2>
<p>
I don’t think I’ve written about them yet in this series—another story, for another day—but <em>hylomorphisms</em> consist of a fold after an unfold. One very simple example is the factorial function: <img src="https://s0.wp.com/latex.php?latex=%7Bn%21%7D&bg=ffffff&fg=000000&s=0" alt="{n!}" class="latex" title="{n!}" /> is the product of the predecessors <img src="https://s0.wp.com/latex.php?latex=%7B%5Bn%2C...%2C1%5D%7D&bg=ffffff&fg=000000&s=0" alt="{[n,...,1]}" class="latex" title="{[n,...,1]}" /> of <img src="https://s0.wp.com/latex.php?latex=%7Bn%7D&bg=ffffff&fg=000000&s=0" alt="{n}" class="latex" title="{n}" />. The predecessors can be computed with an unfold: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bpreds%7D+%26%3A%3A%26+%5Cmathit%7BInteger%7D+%5Crightarrow+%5B%5Cmathit%7BInteger%7D%5D+%5C%5C+%5Cmathit%7Bpreds%7D+%26%3D%26+%5Cmathit%7Bunfoldr%7D%5C%3B%5Cmathit%7Bstep%7D+%5C%3B+%5Cmathbf%7Bwhere%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bstep%7D%5C%3B0+%26%3D%26+%5Cmathit%7BNothing%7D+%5C%5C+%5Cmathit%7Bstep%7D%5C%3Bn+%26%3D%26+%5Cmathit%7BJust%7D%5C%3B%28n%2C+n-1%29+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{preds} &::& \mathit{Integer} \rightarrow [\mathit{Integer}] \\ \mathit{preds} &=& \mathit{unfoldr}\;\mathit{step} \; \mathbf{where} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{step}\;0 &=& \mathit{Nothing} \\ \mathit{step}\;n &=& \mathit{Just}\;(n, n-1) \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{preds} &::& \mathit{Integer} \rightarrow [\mathit{Integer}] \\ \mathit{preds} &=& \mathit{unfoldr}\;\mathit{step} \; \mathbf{where} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{step}\;0 &=& \mathit{Nothing} \\ \mathit{step}\;n &=& \mathit{Just}\;(n, n-1) \end{array} \end{array} " />
</p></blockquote>
<p> and the product as a fold: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bprod%7D+%26%3A%3A%26+%5B%5Cmathit%7BInteger%7D%5D+%5Crightarrow+%5Cmathit%7BInteger%7D+%5C%5C+%5Cmathit%7Bprod%7D+%26%3D%26+%5Cmathit%7Bfoldr%7D%5C%3B%28%5Ctimes%29%5C%3B1+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{prod} &::& [\mathit{Integer}] \rightarrow \mathit{Integer} \\ \mathit{prod} &=& \mathit{foldr}\;(\times)\;1 \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{prod} &::& [\mathit{Integer}] \rightarrow \mathit{Integer} \\ \mathit{prod} &=& \mathit{foldr}\;(\times)\;1 \end{array} " />
</p></blockquote>
<p> and then factorial is their composition: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bfactorial%7D+%26%3A%3A%26+%5Cmathit%7BInteger%7D+%5Crightarrow+%5Cmathit%7BInteger%7D+%5C%5C+%5Cmathit%7Bfactorial%7D+%26%3D%26+%5Cmathit%7Bprod%7D+%5Ccdot+%5Cmathit%7Bpreds%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{factorial} &::& \mathit{Integer} \rightarrow \mathit{Integer} \\ \mathit{factorial} &=& \mathit{prod} \cdot \mathit{preds} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{factorial} &::& \mathit{Integer} \rightarrow \mathit{Integer} \\ \mathit{factorial} &=& \mathit{prod} \cdot \mathit{preds} \end{array} " />
</p></blockquote>
<p> Another example is a tree-based sorting algorithm that resembles Hoare’s quicksort: from the input list, grow a binary search tree, as an unfold, and then flattern that tree back to a sorted list, as a fold. This is a divide-and-conquer algorithm; in general, these can be modelled as unfolding a tree of subproblems by repeatedly dividing the problem, then collecting the solution to the original problem by folding together the solutions to subproblems.</p>
<p></p><h2> An unfold after a fold </h2>
<p>
The post is about the opposite composition, an unfold after a fold. Some examples: </p>
<ul>
<li> <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bregroup%7D%5C%3Bn+%3D+%5Cmathit%7Bgroup%7D%5C%3Bn+%5Ccdot+%5Cmathit%7Bconcat%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{regroup}\;n = \mathit{group}\;n \cdot \mathit{concat}}" class="latex" title="{\mathit{regroup}\;n = \mathit{group}\;n \cdot \mathit{concat}}" /> to reformat a list of lists to a given length;
</li><li> <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bheapsort%7D+%3D+%5Cmathit%7BflattenHeap%7D+%5Ccdot+%5Cmathit%7BbuildHeap%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{heapsort} = \mathit{flattenHeap} \cdot \mathit{buildHeap}}" class="latex" title="{\mathit{heapsort} = \mathit{flattenHeap} \cdot \mathit{buildHeap}}" /> to sort a list;
</li><li> <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bbaseconv%7D%5C%3B%28b%2Cc%29+%3D+%5Cmathit%7BtoBase%7D%5C%3Bb+%5Ccdot+%5Cmathit%7BfromBase%7D%5C%3Bc%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{baseconv}\;(b,c) = \mathit{toBase}\;b \cdot \mathit{fromBase}\;c}" class="latex" title="{\mathit{baseconv}\;(b,c) = \mathit{toBase}\;b \cdot \mathit{fromBase}\;c}" /> to convert a fraction from base <img src="https://s0.wp.com/latex.php?latex=%7Bc%7D&bg=ffffff&fg=000000&s=0" alt="{c}" class="latex" title="{c}" /> to base <img src="https://s0.wp.com/latex.php?latex=%7Bb%7D&bg=ffffff&fg=000000&s=0" alt="{b}" class="latex" title="{b}" />;
</li><li> <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7BarithCode%7D+%3D+%5Cmathit%7BtoBits%7D+%5Ccdot+%5Cmathit%7Bnarrow%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{arithCode} = \mathit{toBits} \cdot \mathit{narrow}}" class="latex" title="{\mathit{arithCode} = \mathit{toBits} \cdot \mathit{narrow}}" /> to encode a text in binary by “arithmetic coding”.
</li></ul>
<p> In each of these cases, the first phase is a fold, which consumes some structured representation of a value into an intermediate unstructured format, and the second phase is an unfold, which generates a new structured representation. Their composition effects a change of representation, so we call them <a href="https://www.cs.ox.ac.uk/publications/publication380-abstract.html">metamorphisms</a>. </p>
<p>
Hylomorphisms always <em>fuse</em>, and one can deforest the intermediate <em>virtual data structure</em>. For example, one need not construct the intermediate list in the factorial function; since each cell gets constructed in the unfold only to be immediately deconstructed in the fold, one can cut to the chase and go straight to the familiar recursive definition. For the base case, we have: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7Bll%7D+%26+%5Cmathit%7Bfactorial%7D%5C%3B0+%5C%5C+%3D+%26+%5Cqquad+%5C%7B+%5Cmathit%7Bfactorial%7D+%5C%7D+%5C%5C+%26+%5Cmathit%7Bprod%7D%5C%3B%28%5Cmathit%7Bpreds%7D%5C%3B0%29+%5C%5C+%3D+%26+%5Cqquad+%5C%7B+%5Cmathit%7Bpreds%7D+%5C%7D+%5C%5C+%26+%5Cmathit%7Bprod%7D%5C%3B%5B%5C%2C%5D+%5C%5C+%3D+%26+%5Cqquad+%5C%7B+%5Cmathit%7Bprod%7D+%5C%7D+%5C%5C+%26+1+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{ll} & \mathit{factorial}\;0 \\ = & \qquad \{ \mathit{factorial} \} \\ & \mathit{prod}\;(\mathit{preds}\;0) \\ = & \qquad \{ \mathit{preds} \} \\ & \mathit{prod}\;[\,] \\ = & \qquad \{ \mathit{prod} \} \\ & 1 \end{array} " class="latex" title="\displaystyle \begin{array}{ll} & \mathit{factorial}\;0 \\ = & \qquad \{ \mathit{factorial} \} \\ & \mathit{prod}\;(\mathit{preds}\;0) \\ = & \qquad \{ \mathit{preds} \} \\ & \mathit{prod}\;[\,] \\ = & \qquad \{ \mathit{prod} \} \\ & 1 \end{array} " />
</p></blockquote>
<p> and for non-zero argument <img src="https://s0.wp.com/latex.php?latex=%7Bn%7D&bg=ffffff&fg=000000&s=0" alt="{n}" class="latex" title="{n}" />, we have: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7Bll%7D+%26+%5Cmathit%7Bfactorial%7D%5C%3Bn+%5C%5C+%3D+%26+%5Cqquad+%5C%7B+%5Cmathit%7Bfactorial%7D+%5C%7D+%5C%5C+%26+%5Cmathit%7Bprod%7D%5C%3B%28%5Cmathit%7Bpreds%7D%5C%3Bn%29+%5C%5C+%3D+%26+%5Cqquad+%5C%7B+%5Cmathit%7Bpreds%7D+%5C%7D+%5C%5C+%26+%5Cmathit%7Bprod%7D%5C%3B%28n+%3A+%5Cmathit%7Bpreds%7D%5C%3B%28n-1%29%29+%5C%5C+%3D+%26+%5Cqquad+%5C%7B+%5Cmathit%7Bprod%7D+%5C%7D+%5C%5C+%26+n+%5Ctimes+%5Cmathit%7Bprod%7D%5C%3B%28%5Cmathit%7Bpreds%7D%5C%3B%28n-1%29%29+%5C%5C+%3D+%26+%5Cqquad+%5C%7B+%5Cmathit%7Bfactorial%7D+%5C%7D+%5C%5C+%26+n+%5Ctimes+%5Cmathit%7Bfactorial%7D%5C%3B%28n-1%29+%5C%5C+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{ll} & \mathit{factorial}\;n \\ = & \qquad \{ \mathit{factorial} \} \\ & \mathit{prod}\;(\mathit{preds}\;n) \\ = & \qquad \{ \mathit{preds} \} \\ & \mathit{prod}\;(n : \mathit{preds}\;(n-1)) \\ = & \qquad \{ \mathit{prod} \} \\ & n \times \mathit{prod}\;(\mathit{preds}\;(n-1)) \\ = & \qquad \{ \mathit{factorial} \} \\ & n \times \mathit{factorial}\;(n-1) \\ \end{array} " class="latex" title="\displaystyle \begin{array}{ll} & \mathit{factorial}\;n \\ = & \qquad \{ \mathit{factorial} \} \\ & \mathit{prod}\;(\mathit{preds}\;n) \\ = & \qquad \{ \mathit{preds} \} \\ & \mathit{prod}\;(n : \mathit{preds}\;(n-1)) \\ = & \qquad \{ \mathit{prod} \} \\ & n \times \mathit{prod}\;(\mathit{preds}\;(n-1)) \\ = & \qquad \{ \mathit{factorial} \} \\ & n \times \mathit{factorial}\;(n-1) \\ \end{array} " />
</p></blockquote>
<p>
In contrast, metamorphisms only fuse under certain conditions. However, when they do fuse, they also allow infinite representations to be processed, as we shall see.</p>
<p>
Fusion seems to depend on the fold being tail-recursive; that is, a <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bfoldl%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{foldl}}" class="latex" title="{\mathit{foldl}}" />: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bfoldl%7D+%26%3A%3A%26+%28%5Cbeta+%5Crightarrow+%5Calpha+%5Crightarrow+%5Cbeta%29+%5Crightarrow+%5Cbeta+%5Crightarrow+%5B%5Calpha%5D+%5Crightarrow+%5Cbeta+%5C%5C+%5Cmathit%7Bfoldl%7D%5C%3Bf%5C%3Bb%5C%3B%28a%3Ax%29+%26%3D%26+%5Cmathit%7Bfoldl%7D%5C%3Bf%5C%3B%28f%5C%3Bb%5C%3Ba%29%5C%3Bx+%5C%5C+%5Cmathit%7Bfoldl%7D%5C%3Bf%5C%3Bb%5C%3B%5B%5C%2C%5D+%26%3D%26+b+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{foldl} &::& (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow \beta \\ \mathit{foldl}\;f\;b\;(a:x) &=& \mathit{foldl}\;f\;(f\;b\;a)\;x \\ \mathit{foldl}\;f\;b\;[\,] &=& b \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{foldl} &::& (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow \beta \\ \mathit{foldl}\;f\;b\;(a:x) &=& \mathit{foldl}\;f\;(f\;b\;a)\;x \\ \mathit{foldl}\;f\;b\;[\,] &=& b \end{array} " />
</p></blockquote>
<p> For the unfold phase, we will use the usual list unfold: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bunfoldr%7D+%26%3A%3A%26+%28%5Cbeta+%5Crightarrow+%5Cmathsf%7BMaybe%7D%5C%3B%28%5Cgamma%2C%5Cbeta%29%29+%5Crightarrow+%5Cbeta+%5Crightarrow+%5B%5Cgamma%5D+%5C%5C+%5Cmathit%7Bunfoldr%7D%5C%3Bg%5C%3Bb+%26%3D%26+%5Cmathbf%7Bcase%7D%5C%3Bg%5C%3Bb%5C%3B%5Cmathbf%7Bof%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7BJust%7D%5C%3B%28c%2Cb%27%29+%26%5Crightarrow%26+c+%3A+%5Cmathit%7Bunfoldr%7D%5C%3Bg%5C%3Bb%27+%5C%5C+%5Cmathit%7BNothing%7D+%26%5Crightarrow%26+%5B%5C%2C%5D+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{unfoldr} &::& (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow \beta \rightarrow [\gamma] \\ \mathit{unfoldr}\;g\;b &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{unfoldr}\;g\;b' \\ \mathit{Nothing} &\rightarrow& [\,] \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{unfoldr} &::& (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow \beta \rightarrow [\gamma] \\ \mathit{unfoldr}\;g\;b &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{unfoldr}\;g\;b' \\ \mathit{Nothing} &\rightarrow& [\,] \end{array} \end{array} " />
</p></blockquote>
<p> We define a metamorphism as their composition: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7Bl%7D+%5Cmathit%7Bmeta%7D+%3A%3A+%28%5Cbeta+%5Crightarrow+%5Cmathsf%7BMaybe%7D%5C%3B%28%5Cgamma%2C%5Cbeta%29%29+%5Crightarrow+%28%5Cbeta+%5Crightarrow+%5Calpha+%5Crightarrow+%5Cbeta%29+%5Crightarrow+%5Cbeta+%5Crightarrow+%5B%5Calpha%5D+%5Crightarrow+%5B%5Cgamma%5D+%5C%5C+%5Cmathit%7Bmeta%7D%5C%3Bg%5C%3Bf%5C%3Bb+%3D+%5Cmathit%7Bunfoldr%7D%5C%3Bg+%5Ccdot+%5Cmathit%7Bfoldl%7D%5C%3Bf%5C%3Bb+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{l} \mathit{meta} :: (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow [\gamma] \\ \mathit{meta}\;g\;f\;b = \mathit{unfoldr}\;g \cdot \mathit{foldl}\;f\;b \end{array} " class="latex" title="\displaystyle \begin{array}{l} \mathit{meta} :: (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow [\gamma] \\ \mathit{meta}\;g\;f\;b = \mathit{unfoldr}\;g \cdot \mathit{foldl}\;f\;b \end{array} " />
</p></blockquote>
<p> This transforms input of type <img src="https://s0.wp.com/latex.php?latex=%7B%5BA%5D%7D&bg=ffffff&fg=000000&s=0" alt="{[A]}" class="latex" title="{[A]}" /> to output of type <img src="https://s0.wp.com/latex.php?latex=%7B%5BC%5D%7D&bg=ffffff&fg=000000&s=0" alt="{[C]}" class="latex" title="{[C]}" />: in the first phase, <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bfoldl%7D%5C%3Bf%5C%3Bb%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{foldl}\;f\;b}" class="latex" title="{\mathit{foldl}\;f\;b}" />, it consumes all the input into an intermediate value of type <img src="https://s0.wp.com/latex.php?latex=%7BB%7D&bg=ffffff&fg=000000&s=0" alt="{B}" class="latex" title="{B}" />; in the second phase, <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bunfoldr%7D%5C%3Bg%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{unfoldr}\;g}" class="latex" title="{\mathit{unfoldr}\;g}" />, it produces all the output.</p>
<p></p><h2> Streaming </h2>
<p>
Under certain conditions, it is possible to fuse these two phases—this time, not in order to eliminate an intermediate data structure (after all, the intermediate type <img src="https://s0.wp.com/latex.php?latex=%7BB%7D&bg=ffffff&fg=000000&s=0" alt="{B}" class="latex" title="{B}" /> need not be structured), but rather in order to allow some production steps to happen before all the consumption steps are complete. </p>
<p>
To that end, we define the <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bstream%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{stream}}" class="latex" title="{\mathit{stream}}" /> function as follows: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmulticolumn%7B3%7D%7B%40%7B%7Dl%7D%7B%5Cmathit%7Bstream%7D+%3A%3A+%28%5Cbeta+%5Crightarrow+%5Cmathsf%7BMaybe%7D%5C%3B%28%5Cgamma%2C%5Cbeta%29%29+%5Crightarrow+%28%5Cbeta+%5Crightarrow+%5Calpha+%5Crightarrow+%5Cbeta%29+%5Crightarrow+%5Cbeta+%5Crightarrow+%5B%5Calpha%5D+%5Crightarrow+%5B%5Cgamma%5D%7D+%5C%5C+%5Cmathit%7Bstream%7D%5C%3Bg%5C%3Bf%5C%3Bb%5C%3Bx+%26%3D%26+%5Cmathbf%7Bcase%7D%5C%3Bg%5C%3Bb%5C%3B%5Cmathbf%7Bof%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7BJust%7D%5C%3B%28c%2Cb%27%29+%26%5Crightarrow%26+c+%3A+%5Cmathit%7Bstream%7D%5C%3Bg%5C%3Bf%5C%3Bb%27%5C%3Bx+%5C%5C+%5Cmathit%7BNothing%7D+%26%5Crightarrow%26+%5Cmathbf%7Bcase%7D%5C%3Bx%5C%3B%5Cmathbf%7Bof%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+a%3Ax%27+%26%5Crightarrow%26+%5Cmathit%7Bstream%7D%5C%3Bg%5C%3Bf%5C%3B%28f%5C%3Bb%5C%3Ba%29%5C%3Bx%27+%5C%5C+%7B%5B%5C%2C%5D%7D+%26%5Crightarrow%26+%5B%5C%2C%5D+%5Cend%7Barray%7D+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \multicolumn{3}{@{}l}{\mathit{stream} :: (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow [\gamma]} \\ \mathit{stream}\;g\;f\;b\;x &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{stream}\;g\;f\;b'\;x \\ \mathit{Nothing} &\rightarrow& \mathbf{case}\;x\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} a:x' &\rightarrow& \mathit{stream}\;g\;f\;(f\;b\;a)\;x' \\ {[\,]} &\rightarrow& [\,] \end{array} \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \multicolumn{3}{@{}l}{\mathit{stream} :: (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow [\gamma]} \\ \mathit{stream}\;g\;f\;b\;x &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{stream}\;g\;f\;b'\;x \\ \mathit{Nothing} &\rightarrow& \mathbf{case}\;x\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} a:x' &\rightarrow& \mathit{stream}\;g\;f\;(f\;b\;a)\;x' \\ {[\,]} &\rightarrow& [\,] \end{array} \end{array} \end{array} " />
</p></blockquote>
<p> This takes the same arguments as <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bmeta%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{meta}}" class="latex" title="{\mathit{meta}}" />. It maintains a current state <img src="https://s0.wp.com/latex.php?latex=%7Bb%7D&bg=ffffff&fg=000000&s=0" alt="{b}" class="latex" title="{b}" />, and produces an output element <img src="https://s0.wp.com/latex.php?latex=%7Bc%7D&bg=ffffff&fg=000000&s=0" alt="{c}" class="latex" title="{c}" /> when it can; and when it can’t produce, it consumes an input element instead. In more detail, it examines the current state <img src="https://s0.wp.com/latex.php?latex=%7Bb%7D&bg=ffffff&fg=000000&s=0" alt="{b}" class="latex" title="{b}" /> using function <img src="https://s0.wp.com/latex.php?latex=%7Bg%7D&bg=ffffff&fg=000000&s=0" alt="{g}" class="latex" title="{g}" />, which is like the body of an unfold; this may produce a first element <img src="https://s0.wp.com/latex.php?latex=%7Bc%7D&bg=ffffff&fg=000000&s=0" alt="{c}" class="latex" title="{c}" /> of the result and a new state <img src="https://s0.wp.com/latex.php?latex=%7Bb%27%7D&bg=ffffff&fg=000000&s=0" alt="{b'}" class="latex" title="{b'}" />; when it yields no element, the next element <img src="https://s0.wp.com/latex.php?latex=%7Ba%7D&bg=ffffff&fg=000000&s=0" alt="{a}" class="latex" title="{a}" /> of the input is consumed using function <img src="https://s0.wp.com/latex.php?latex=%7Bf%7D&bg=ffffff&fg=000000&s=0" alt="{f}" class="latex" title="{f}" />, which is like the body of a <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bfoldl%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{foldl}}" class="latex" title="{\mathit{foldl}}" />; and when no input remains either, we are done.</p>
<p>
The <em>streaming condition</em> for <img src="https://s0.wp.com/latex.php?latex=%7Bf%7D&bg=ffffff&fg=000000&s=0" alt="{f}" class="latex" title="{f}" /> and <img src="https://s0.wp.com/latex.php?latex=%7Bg%7D&bg=ffffff&fg=000000&s=0" alt="{g}" class="latex" title="{g}" /> is that </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++g%5C%3Bb+%3D+%5Cmathit%7BJust%7D%5C%3B%28c%2Cb%27%29+%5Cquad%5CRightarrow%5Cquad+%5Cforall+a+%5Cmathbin%7B.%7D+g%5C%3B%28f%5C%3Bb%5C%3Ba%29+%3D+%5Cmathit%7BJust%7D%5C%3B%28c%2C+f%5C%3Bb%27%5C%3Ba%29+&bg=ffffff&fg=000000&s=0" alt="\displaystyle g\;b = \mathit{Just}\;(c,b') \quad\Rightarrow\quad \forall a \mathbin{.} g\;(f\;b\;a) = \mathit{Just}\;(c, f\;b'\;a) " class="latex" title="\displaystyle g\;b = \mathit{Just}\;(c,b') \quad\Rightarrow\quad \forall a \mathbin{.} g\;(f\;b\;a) = \mathit{Just}\;(c, f\;b'\;a) " />
</p></blockquote>
<p> Consider a state <img src="https://s0.wp.com/latex.php?latex=%7Bb%7D&bg=ffffff&fg=000000&s=0" alt="{b}" class="latex" title="{b}" /> from which the body <img src="https://s0.wp.com/latex.php?latex=%7Bg%7D&bg=ffffff&fg=000000&s=0" alt="{g}" class="latex" title="{g}" /> of the unfold is productive, yielding some <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7BJust%7D%5C%3B%28c%2Cb%27%29%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{Just}\;(c,b')}" class="latex" title="{\mathit{Just}\;(c,b')}" />. From here we have two choices: we can either produce the output <img src="https://s0.wp.com/latex.php?latex=%7Bc%7D&bg=ffffff&fg=000000&s=0" alt="{c}" class="latex" title="{c}" />, move to intermediate state <img src="https://s0.wp.com/latex.php?latex=%7Bb%27%7D&bg=ffffff&fg=000000&s=0" alt="{b'}" class="latex" title="{b'}" />, then consume the next input <img src="https://s0.wp.com/latex.php?latex=%7Ba%7D&bg=ffffff&fg=000000&s=0" alt="{a}" class="latex" title="{a}" /> to yield a final state <img src="https://s0.wp.com/latex.php?latex=%7Bf%5C%3Bb%27%5C%3Ba%7D&bg=ffffff&fg=000000&s=0" alt="{f\;b'\;a}" class="latex" title="{f\;b'\;a}" />; or we can consume first to get the intermediate state <img src="https://s0.wp.com/latex.php?latex=%7Bf%5C%3Bb%5C%3Ba%7D&bg=ffffff&fg=000000&s=0" alt="{f\;b\;a}" class="latex" title="{f\;b\;a}" />, and again try to produce. The streaming condition says that this intermediate state <img src="https://s0.wp.com/latex.php?latex=%7Bf%5C%3Bb%5C%3Ba%7D&bg=ffffff&fg=000000&s=0" alt="{f\;b\;a}" class="latex" title="{f\;b\;a}" /> will again be productive, and will yield the same output <img src="https://s0.wp.com/latex.php?latex=%7Bc%7D&bg=ffffff&fg=000000&s=0" alt="{c}" class="latex" title="{c}" /> and the same final state <img src="https://s0.wp.com/latex.php?latex=%7Bf%5C%3Bb%27%5C%3Ba%7D&bg=ffffff&fg=000000&s=0" alt="{f\;b'\;a}" class="latex" title="{f\;b'\;a}" />. That is, instead of consuming all the inputs first, and then producing all the outputs, it is possible to produce some of the outputs early, without jeopardizing the overall result. Provided that the streaming condition holds for <img src="https://s0.wp.com/latex.php?latex=%7Bf%7D&bg=ffffff&fg=000000&s=0" alt="{f}" class="latex" title="{f}" /> and <img src="https://s0.wp.com/latex.php?latex=%7Bg%7D&bg=ffffff&fg=000000&s=0" alt="{g}" class="latex" title="{g}" />, then </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathit%7Bstream%7D%5C%3Bg%5C%3Bf%5C%3Bb%5C%3Bx+%3D+%5Cmathit%7Bmeta%7D%5C%3Bg%5C%3Bf%5C%3Bb%5C%3Bx+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \mathit{stream}\;g\;f\;b\;x = \mathit{meta}\;g\;f\;b\;x " class="latex" title="\displaystyle \mathit{stream}\;g\;f\;b\;x = \mathit{meta}\;g\;f\;b\;x " />
</p></blockquote>
<p> for all finite lists <img src="https://s0.wp.com/latex.php?latex=%7Bx%7D&bg=ffffff&fg=000000&s=0" alt="{x}" class="latex" title="{x}" />.</p>
<p>
As a simple example, consider the `buffering’ process <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bmeta%7D%5C%3B%5Cmathit%7Buncons%7D%5C%3B%28%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%29%5C%3B%5B%5C%2C%5D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{meta}\;\mathit{uncons}\;(\mathbin{{+}\!\!\!{+}})\;[\,]}" class="latex" title="{\mathit{meta}\;\mathit{uncons}\;(\mathbin{{+}\!\!\!{+}})\;[\,]}" />, where </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Buncons%7D%5C%3Bx+%26%3D%26+%5Cmathbf%7Bcase%7D%5C%3Bx%5C%3B%5Cmathbf%7Bof%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5B%5C%2C%5D+%26%5Crightarrow%26+%5Cmathit%7BNothing%7D+%5C%5C+c%3Ax%27+%26%5Crightarrow%26+%5Cmathit%7BJust%7D%5C%3B%28c%2Cx%27%29+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{uncons}\;x &=& \mathbf{case}\;x\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} [\,] &\rightarrow& \mathit{Nothing} \\ c:x' &\rightarrow& \mathit{Just}\;(c,x') \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{uncons}\;x &=& \mathbf{case}\;x\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} [\,] &\rightarrow& \mathit{Nothing} \\ c:x' &\rightarrow& \mathit{Just}\;(c,x') \end{array} \end{array} " />
</p></blockquote>
<p> Note that <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bunfoldr%7D%5C%3B%5Cmathit%7Buncons%7D+%3D+%5Cmathit%7Bid%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{unfoldr}\;\mathit{uncons} = \mathit{id}}" class="latex" title="{\mathit{unfoldr}\;\mathit{uncons} = \mathit{id}}" />, so <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bmeta%7D%5C%3B%5Cmathit%7Buncons%7D%5C%3B%28%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%29%5C%3B%5B%5C%2C%5D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{meta}\;\mathit{uncons}\;(\mathbin{{+}\!\!\!{+}})\;[\,]}" class="latex" title="{\mathit{meta}\;\mathit{uncons}\;(\mathbin{{+}\!\!\!{+}})\;[\,]}" /> is just a complicated way of writing <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bconcat%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{concat}}" class="latex" title="{\mathit{concat}}" /> as a <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bfoldl%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{foldl}}" class="latex" title="{\mathit{foldl}}" />. But the streaming condition holds for <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathbin{{+}\!\!\!{+}}}" class="latex" title="{\mathbin{{+}\!\!\!{+}}}" /> and <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Buncons%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{uncons}}" class="latex" title="{\mathit{uncons}}" /> (as you may check), so <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bconcat%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{concat}}" class="latex" title="{\mathit{concat}}" /> may be streamed. Operationally, the streaming version of <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bconcat%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{concat}}" class="latex" title="{\mathit{concat}}" /> consumes one list from the input list of lists, then peels off and produces its elements one by one; when they have all been delivered, it consumes the next input list, and so on.</p>
<p></p><h2> Flushing </h2>
<p>
The streaming version of <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bconcat%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{concat}}" class="latex" title="{\mathit{concat}}" /> is actually rather special, because the production steps can always completely exhaust the intermediate state. In contrast, consider the `regrouping’ example <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bregroup%7D%5C%3Bn+%3D+%5Cmathit%7Bmeta%7D%5C%3B%28%5Cmathit%7Bchunk%7D%5C%3Bn%29%5C%3B%28%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%29%5C%3B%5B%5C%2C%5D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{regroup}\;n = \mathit{meta}\;(\mathit{chunk}\;n)\;(\mathbin{{+}\!\!\!{+}})\;[\,]}" class="latex" title="{\mathit{regroup}\;n = \mathit{meta}\;(\mathit{chunk}\;n)\;(\mathbin{{+}\!\!\!{+}})\;[\,]}" /> where </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bchunk%7D%5C%3Bn%5C%3B%5B%5C%2C%5D+%26%3D%26+%5Cmathit%7BNothing%7D+%5C%5C+%5Cmathit%7Bchunk%7D%5C%3Bn%5C%3Bx+%26%3D%26+%5Cmathit%7BJust%7D%5C%3B%28%5Cmathit%7BsplitAt%7D%5C%3Bn%5C%3Bx%29+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{chunk}\;n\;[\,] &=& \mathit{Nothing} \\ \mathit{chunk}\;n\;x &=& \mathit{Just}\;(\mathit{splitAt}\;n\;x) \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{chunk}\;n\;[\,] &=& \mathit{Nothing} \\ \mathit{chunk}\;n\;x &=& \mathit{Just}\;(\mathit{splitAt}\;n\;x) \end{array} " />
</p></blockquote>
<p> from the introduction (here, <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7BsplitAt%7D%5C%3Bn%5C%3Bx%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{splitAt}\;n\;x}" class="latex" title="{\mathit{splitAt}\;n\;x}" /> yields <img src="https://s0.wp.com/latex.php?latex=%7B%28y%2Cz%29%7D&bg=ffffff&fg=000000&s=0" alt="{(y,z)}" class="latex" title="{(y,z)}" /> where <img src="https://s0.wp.com/latex.php?latex=%7By+%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D+z+%3D+x%7D&bg=ffffff&fg=000000&s=0" alt="{y \mathbin{{+}\!\!\!{+}} z = x}" class="latex" title="{y \mathbin{{+}\!\!\!{+}} z = x}" />, with <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Blength%7D%5C%3By%3Dn%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{length}\;y=n}" class="latex" title="{\mathit{length}\;y=n}" /> when <img src="https://s0.wp.com/latex.php?latex=%7Bn+%5Cle+%5Cmathit%7Blength%7D%5C%3Bx%7D&bg=ffffff&fg=000000&s=0" alt="{n \le \mathit{length}\;x}" class="latex" title="{n \le \mathit{length}\;x}" /> and <img src="https://s0.wp.com/latex.php?latex=%7By%3Dx%7D&bg=ffffff&fg=000000&s=0" alt="{y=x}" class="latex" title="{y=x}" /> otherwise). This transforms an input list of lists into an output list of lists, where each output `chunk’ except perhaps the last has length <img src="https://s0.wp.com/latex.php?latex=%7Bn%7D&bg=ffffff&fg=000000&s=0" alt="{n}" class="latex" title="{n}" />—if the content doesn’t divide up evenly, then the last chunk is short. One might hope to be able to stream <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bregroup%7D%5C%3Bn%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{regroup}\;n}" class="latex" title="{\mathit{regroup}\;n}" />, but it doesn’t quite work with the formulation so far. The problem is that <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bchunk%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{chunk}}" class="latex" title="{\mathit{chunk}}" /> is too aggressive, and will produce short chunks when there is still some input to consume. (Indeed, the streaming condition does not for <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathbin{{+}\!\!\!{+}}}" class="latex" title="{\mathbin{{+}\!\!\!{+}}}" /> and <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bchunk%7D%5C%3Bn%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{chunk}\;n}" class="latex" title="{\mathit{chunk}\;n}" />—why not?) One might try the more cautious producer <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bchunk%27%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{chunk'}}" class="latex" title="{\mathit{chunk'}}" />: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlclcl%40%7B%7D%7D+%5Cmathit%7Bchunk%27%7D%5C%3Bn%5C%3Bx+%26%5Cmid%26+n+%5Cle+%5Cmathit%7Blength%7D%5C%3Bx+%26%3D%26+%5Cmathit%7BJust%7D%5C%3B%28%5Cmathit%7BsplitAt%7D%5C%3Bn%5C%3Bx%29+%5C%5C+%26%5Cmid%26+%5Cmathbf%7Botherwise%7D+%26%3D%26+%5Cmathit%7BNothing%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lclcl@{}} \mathit{chunk'}\;n\;x &\mid& n \le \mathit{length}\;x &=& \mathit{Just}\;(\mathit{splitAt}\;n\;x) \\ &\mid& \mathbf{otherwise} &=& \mathit{Nothing} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lclcl@{}} \mathit{chunk'}\;n\;x &\mid& n \le \mathit{length}\;x &=& \mathit{Just}\;(\mathit{splitAt}\;n\;x) \\ &\mid& \mathbf{otherwise} &=& \mathit{Nothing} \end{array} " />
</p></blockquote>
<p> But this never produces a short chunk, and so if the content doesn’t divide up evenly then the last few elements will not be extracted from the intermediate state and will be lost.</p>
<p>
We need to combine these two producers somehow: the streaming process should behave cautiously while there is still remaining input, which might influence the next output; but it should then switch to a more aggressive strategy once the input is finished, in order to flush out the contents of the intermediate state. To achieve this, we define a more general <em>flushing stream</em> operator: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmulticolumn%7B3%7D%7B%40%7B%7Dl%40%7B%7D%7D%7B%5Cmathit%7Bfstream%7D+%3A%3A+%28%5Cbeta+%5Crightarrow+%5Cmathsf%7BMaybe%7D%5C%3B%28%5Cgamma%2C%5Cbeta%29%29+%5Crightarrow+%28%5Cbeta+%5Crightarrow+%5B%5Cgamma%5D%29+%5Crightarrow+%28%5Cbeta+%5Crightarrow+%5Calpha+%5Crightarrow+%5Cbeta%29+%5Crightarrow+%5Cbeta+%5Crightarrow+%5B%5Calpha%5D+%5Crightarrow+%5B%5Cgamma%5D%7D+%5C%5C+%5Cmathit%7Bfstream%7D%5C%3Bg%5C%3Bh%5C%3Bf%5C%3Bb%5C%3Bx+%26%3D%26+%5Cmathbf%7Bcase%7D%5C%3Bg%5C%3Bb%5C%3B%5Cmathbf%7Bof%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7BJust%7D%5C%3B%28c%2Cb%27%29+%26%5Crightarrow%26+c+%3A+%5Cmathit%7Bfstream%7D%5C%3Bg%5C%3Bh%5C%3Bf%5C%3Bb%27%5C%3Bx+%5C%5C+%5Cmathit%7BNothing%7D+%26%5Crightarrow%26+%5Cmathbf%7Bcase%7D%5C%3Bx%5C%3B%5Cmathbf%7Bof%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+a%3Ax%27+%26%5Crightarrow%26+%5Cmathit%7Bfstream%7D%5C%3Bg%5C%3Bh%5C%3Bf%5C%3B%28f%5C%3Bb%5C%3Ba%29%5C%3Bx%27+%5C%5C+%7B%5B%5C%2C%5D%7D+%26%5Crightarrow%26+h%5C%3Bb+%5Cend%7Barray%7D+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \multicolumn{3}{@{}l@{}}{\mathit{fstream} :: (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow [\gamma]) \rightarrow (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow [\gamma]} \\ \mathit{fstream}\;g\;h\;f\;b\;x &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{fstream}\;g\;h\;f\;b'\;x \\ \mathit{Nothing} &\rightarrow& \mathbf{case}\;x\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} a:x' &\rightarrow& \mathit{fstream}\;g\;h\;f\;(f\;b\;a)\;x' \\ {[\,]} &\rightarrow& h\;b \end{array} \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \multicolumn{3}{@{}l@{}}{\mathit{fstream} :: (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow [\gamma]) \rightarrow (\beta \rightarrow \alpha \rightarrow \beta) \rightarrow \beta \rightarrow [\alpha] \rightarrow [\gamma]} \\ \mathit{fstream}\;g\;h\;f\;b\;x &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{fstream}\;g\;h\;f\;b'\;x \\ \mathit{Nothing} &\rightarrow& \mathbf{case}\;x\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} a:x' &\rightarrow& \mathit{fstream}\;g\;h\;f\;(f\;b\;a)\;x' \\ {[\,]} &\rightarrow& h\;b \end{array} \end{array} \end{array} " />
</p></blockquote>
<p> This takes an additional argument <img src="https://s0.wp.com/latex.php?latex=%7Bh+%3A%3A+%5Cbeta+%5Crightarrow+%5B%5Cgamma%5D%7D&bg=ffffff&fg=000000&s=0" alt="{h :: \beta \rightarrow [\gamma]}" class="latex" title="{h :: \beta \rightarrow [\gamma]}" />; when the cautious producer <img src="https://s0.wp.com/latex.php?latex=%7Bg%7D&bg=ffffff&fg=000000&s=0" alt="{g}" class="latex" title="{g}" /> is unproductive, and there is no remaining input to consume, it uses <img src="https://s0.wp.com/latex.php?latex=%7Bh%7D&bg=ffffff&fg=000000&s=0" alt="{h}" class="latex" title="{h}" /> to flush out the remaining output elements from the state. Clearly, specializing to <img src="https://s0.wp.com/latex.php?latex=%7Bh%5C%3Bb%3D%5B%5C%2C%5D%7D&bg=ffffff&fg=000000&s=0" alt="{h\;b=[\,]}" class="latex" title="{h\;b=[\,]}" /> retrieves the original <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bstream%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{stream}}" class="latex" title="{\mathit{stream}}" /> operator.</p>
<p>
The corresponding metamorphism uses an <em><a href="https://www.mii.lt/informatica/htm/INFO141.htm">apomorphism</a></em> in place of the unfold. Define </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bapo%7D+%26%3A%3A%26+%28%5Cbeta+%5Crightarrow+%5Cmathsf%7BMaybe%7D%5C%3B%28%5Cgamma%2C%5Cbeta%29%29+%5Crightarrow+%28%5Cbeta+%5Crightarrow+%5B%5Cgamma%5D%29+%5Crightarrow+%5Cbeta+%5Crightarrow+%5B%5Cgamma%5D+%5C%5C+%5Cmathit%7Bapo%7D%5C%3Bg%5C%3Bh%5C%3Bb+%26%3D%26+%5Cmathbf%7Bcase%7D%5C%3Bg%5C%3Bb%5C%3B%5Cmathbf%7Bof%7D+%5C%5C+%26+%26+%5Cquad+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7BJust%7D%5C%3B%28c%2Cb%27%29+%26%5Crightarrow%26+c+%3A+%5Cmathit%7Bapo%7D%5C%3Bg%5C%3Bh%5C%3Bb%27+%5C%5C+%5Cmathit%7BNothing%7D+%26%5Crightarrow%26+h%5C%3Bb+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{apo} &::& (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow [\gamma]) \rightarrow \beta \rightarrow [\gamma] \\ \mathit{apo}\;g\;h\;b &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{apo}\;g\;h\;b' \\ \mathit{Nothing} &\rightarrow& h\;b \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{apo} &::& (\beta \rightarrow \mathsf{Maybe}\;(\gamma,\beta)) \rightarrow (\beta \rightarrow [\gamma]) \rightarrow \beta \rightarrow [\gamma] \\ \mathit{apo}\;g\;h\;b &=& \mathbf{case}\;g\;b\;\mathbf{of} \\ & & \quad \begin{array}[t]{@{}lcl@{}} \mathit{Just}\;(c,b') &\rightarrow& c : \mathit{apo}\;g\;h\;b' \\ \mathit{Nothing} &\rightarrow& h\;b \end{array} \end{array} " />
</p></blockquote>
<p> Then <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bapo%7D%5C%3Bg%5C%3Bh%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{apo}\;g\;h}" class="latex" title="{\mathit{apo}\;g\;h}" /> behaves like <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bunfoldr%7D%5C%3Bg%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{unfoldr}\;g}" class="latex" title="{\mathit{unfoldr}\;g}" />, except that if and when <img src="https://s0.wp.com/latex.php?latex=%7Bg%7D&bg=ffffff&fg=000000&s=0" alt="{g}" class="latex" title="{g}" /> stops being productive it finishes up by applying <img src="https://s0.wp.com/latex.php?latex=%7Bh%7D&bg=ffffff&fg=000000&s=0" alt="{h}" class="latex" title="{h}" /> to the final state. Similarly, define flushing metamorphisms: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathit%7Bfmeta%7D%5C%3Bg%5C%3Bh%5C%3Bf%5C%3Bb+%3D+%5Cmathit%7Bapo%7D%5C%3Bg%5C%3Bh+%5Ccdot+%5Cmathit%7Bfoldl%7D%5C%3Bf%5C%3Bb+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \mathit{fmeta}\;g\;h\;f\;b = \mathit{apo}\;g\;h \cdot \mathit{foldl}\;f\;b " class="latex" title="\displaystyle \mathit{fmeta}\;g\;h\;f\;b = \mathit{apo}\;g\;h \cdot \mathit{foldl}\;f\;b " />
</p></blockquote>
<p> Then we have </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathit%7Bfstream%7D%5C%3Bg%5C%3Bh%5C%3Bf%5C%3Bb%5C%3Bx+%3D+%5Cmathit%7Bfmeta%7D%5C%3Bg%5C%3Bh%5C%3Bf%5C%3Bb%5C%3Bx+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \mathit{fstream}\;g\;h\;f\;b\;x = \mathit{fmeta}\;g\;h\;f\;b\;x " class="latex" title="\displaystyle \mathit{fstream}\;g\;h\;f\;b\;x = \mathit{fmeta}\;g\;h\;f\;b\;x " />
</p></blockquote>
<p> for all finite lists <img src="https://s0.wp.com/latex.php?latex=%7Bx%7D&bg=ffffff&fg=000000&s=0" alt="{x}" class="latex" title="{x}" /> if the streaming condition holds for <img src="https://s0.wp.com/latex.php?latex=%7Bf%7D&bg=ffffff&fg=000000&s=0" alt="{f}" class="latex" title="{f}" /> and <img src="https://s0.wp.com/latex.php?latex=%7Bg%7D&bg=ffffff&fg=000000&s=0" alt="{g}" class="latex" title="{g}" />. In particular, </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bregroup%7D%5C%3Bn%5C%3B%5Cmathit%7Bxs%7D+%26%3D%26+%5Cmathit%7Bfmeta%7D%5C%3B%28%5Cmathit%7Bchunk%27%7D%5C%3Bn%29%5C%3B%28%5Cmathit%7Bunfoldr%7D%5C%3B%28%5Cmathit%7Bchunk%7D%5C%3Bn%29%29%5C%3B%28%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%29%5C%3B%5B%5C%2C%5D%5C%3B%5Cmathit%7Bxs%7D+%5C%5C+%26%3D%26+%5Cmathit%7Bfstream%7D%5C%3B%28%5Cmathit%7Bchunk%27%7D%5C%3Bn%29%5C%3B%28%5Cmathit%7Bunfoldr%7D%5C%3B%28%5Cmathit%7Bchunk%7D%5C%3Bn%29%29%5C%3B%28%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%29%5C%3B%5B%5C%2C%5D%5C%3B%5Cmathit%7Bxs%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{regroup}\;n\;\mathit{xs} &=& \mathit{fmeta}\;(\mathit{chunk'}\;n)\;(\mathit{unfoldr}\;(\mathit{chunk}\;n))\;(\mathbin{{+}\!\!\!{+}})\;[\,]\;\mathit{xs} \\ &=& \mathit{fstream}\;(\mathit{chunk'}\;n)\;(\mathit{unfoldr}\;(\mathit{chunk}\;n))\;(\mathbin{{+}\!\!\!{+}})\;[\,]\;\mathit{xs} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{regroup}\;n\;\mathit{xs} &=& \mathit{fmeta}\;(\mathit{chunk'}\;n)\;(\mathit{unfoldr}\;(\mathit{chunk}\;n))\;(\mathbin{{+}\!\!\!{+}})\;[\,]\;\mathit{xs} \\ &=& \mathit{fstream}\;(\mathit{chunk'}\;n)\;(\mathit{unfoldr}\;(\mathit{chunk}\;n))\;(\mathbin{{+}\!\!\!{+}})\;[\,]\;\mathit{xs} \end{array} " />
</p></blockquote>
<p> on finite inputs <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bxs%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{xs}}" class="latex" title="{\mathit{xs}}" />: the streaming condition does hold for <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathbin%7B%7B%2B%7D%5C%21%5C%21%5C%21%7B%2B%7D%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathbin{{+}\!\!\!{+}}}" class="latex" title="{\mathbin{{+}\!\!\!{+}}}" /> and the more cautious <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bchunk%27%7D%5C%3Bn%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{chunk'}\;n}" class="latex" title="{\mathit{chunk'}\;n}" />, and once the input has been exhausted, the process can switch to the more aggressive <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bchunk%7D%5C%3Bn%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{chunk}\;n}" class="latex" title="{\mathit{chunk}\;n}" />. </p>
<p></p><h2> Infinite input </h2>
<p>
The main advantage of streaming is that it can allow the change-of-representation process also to work on infinite inputs. With the plain metamorphism, this is not possible: the <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bfoldl%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{foldl}}" class="latex" title="{\mathit{foldl}}" /> will yield no result on an infinite input, and so the <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bunfoldr%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{unfoldr}}" class="latex" title="{\mathit{unfoldr}}" /> will never get started, but the <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bstream%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{stream}}" class="latex" title="{\mathit{stream}}" /> may be able to produce some outputs before having consumed all the inputs. For example, the streaming version of <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bregroup%7D%5C%3Bn%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{regroup}\;n}" class="latex" title="{\mathit{regroup}\;n}" /> also works for infinite lists, providing that the input does not end with an infinite tail of empty lists. And of course, if the input never runs out, then there is no need ever to switch to the more aggressive flushing phase.</p>
<p>
As a more interesting example, consider converting a fraction from base 3 to base 7: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7BfromBase3%7D+%26%3D%26+%5Cmathit%7Bfoldr%7D%5C%3B%5Cmathit%7Bstepr%7D%5C%3B0+%5Cquad+%5Cmathbf%7Bwhere%7D%5C%3B%5Cmathit%7Bstepr%7D%5C%3Bd%5C%3Bx+%3D+%28d%2Bx%29%2F3+%5C%5C+%5Cmathit%7BtoBase7%7D+%26%3D%26+%5Cmathit%7Bunfoldr%7D%5C%3B%5Cmathit%7Bnext%7D+%5Cquad+%5Cmathbf%7Bwhere%7D%5C%3B+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bnext%7D%5C%3B0+%26%3D%26+%5Cmathit%7BNothing%7D+%5C%5C+%5Cmathit%7Bnext%7D%5C%3Bx+%26%3D%26+%5Cmathbf%7Blet%7D%5C%3By%3D7%5Ctimes+x%5C%3B%5Cmathbf%7Bin%7D%5C%3B%5Cmathit%7BJust%7D%5C%3B%28%5Clfloor+y%5Crfloor%2C+y-%5Clfloor+y%5Crfloor%29+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{fromBase3} &=& \mathit{foldr}\;\mathit{stepr}\;0 \quad \mathbf{where}\;\mathit{stepr}\;d\;x = (d+x)/3 \\ \mathit{toBase7} &=& \mathit{unfoldr}\;\mathit{next} \quad \mathbf{where}\; \begin{array}[t]{@{}lcl@{}} \mathit{next}\;0 &=& \mathit{Nothing} \\ \mathit{next}\;x &=& \mathbf{let}\;y=7\times x\;\mathbf{in}\;\mathit{Just}\;(\lfloor y\rfloor, y-\lfloor y\rfloor) \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{fromBase3} &=& \mathit{foldr}\;\mathit{stepr}\;0 \quad \mathbf{where}\;\mathit{stepr}\;d\;x = (d+x)/3 \\ \mathit{toBase7} &=& \mathit{unfoldr}\;\mathit{next} \quad \mathbf{where}\; \begin{array}[t]{@{}lcl@{}} \mathit{next}\;0 &=& \mathit{Nothing} \\ \mathit{next}\;x &=& \mathbf{let}\;y=7\times x\;\mathbf{in}\;\mathit{Just}\;(\lfloor y\rfloor, y-\lfloor y\rfloor) \end{array} \end{array} " />
</p></blockquote>
<p> We assume that the input digits are all either 0, 1 or 2, so that the number being represented is in the unit interval.</p>
<p>
The fold in <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7BfromBase3%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{fromBase3}}" class="latex" title="{\mathit{fromBase3}}" /> is of the wrong kind; but we have also </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7BfromBase3%7D+%26%3D%26+%5Cmathit%7Bextract%7D+%5Ccdot+%5Cmathit%7Bfoldl%7D%5C%3B%5Cmathit%7Bstepl%7D%5C%3B%280%2C1%29+%5Cquad+%5Cmathbf%7Bwhere%7D%5C%3B+%5Cmathit%7Bstepl%7D%5C%3B%28u%2Cv%29%5C%3Bd+%3D+%28u+%5Ctimes+3+%2B+d%2C+v+%2F+3%29+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{fromBase3} &=& \mathit{extract} \cdot \mathit{foldl}\;\mathit{stepl}\;(0,1) \quad \mathbf{where}\; \mathit{stepl}\;(u,v)\;d = (u \times 3 + d, v / 3) \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{fromBase3} &=& \mathit{extract} \cdot \mathit{foldl}\;\mathit{stepl}\;(0,1) \quad \mathbf{where}\; \mathit{stepl}\;(u,v)\;d = (u \times 3 + d, v / 3) \end{array} " />
</p></blockquote>
<p> Here, the intermediate state <img src="https://s0.wp.com/latex.php?latex=%7B%28u%2Cv%29%7D&bg=ffffff&fg=000000&s=0" alt="{(u,v)}" class="latex" title="{(u,v)}" /> can be seen as a defunctionalized representation of the function <img src="https://s0.wp.com/latex.php?latex=%7B%28v%5Ctimes%29+%5Ccdot+%28u%2B%29%7D&bg=ffffff&fg=000000&s=0" alt="{(v\times) \cdot (u+)}" class="latex" title="{(v\times) \cdot (u+)}" />, and <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bextract%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{extract}}" class="latex" title="{\mathit{extract}}" /> applies this function to <img src="https://s0.wp.com/latex.php?latex=%7B0%7D&bg=ffffff&fg=000000&s=0" alt="{0}" class="latex" title="{0}" />: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bapply%7D%5C%3B%28u%2Cv%29%5C%3Bx+%26%3D%26+v+%5Ctimes+%28u+%2B+x%29+%5C%5C+%5Cmathit%7Bextract%7D%5C%3B%28u%2Cv%29+%26%3D%26+%5Cmathit%7Bapply%7D%5C%3B%28u%2Cv%29%5C%3B0+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{apply}\;(u,v)\;x &=& v \times (u + x) \\ \mathit{extract}\;(u,v) &=& \mathit{apply}\;(u,v)\;0 \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{apply}\;(u,v)\;x &=& v \times (u + x) \\ \mathit{extract}\;(u,v) &=& \mathit{apply}\;(u,v)\;0 \end{array} " />
</p></blockquote>
<p> Now there is an extra function <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bextract%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{extract}}" class="latex" title="{\mathit{extract}}" /> between the <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bfoldl%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{foldl}}" class="latex" title="{\mathit{foldl}}" /> and the <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bunfoldr%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{unfoldr}}" class="latex" title="{\mathit{unfoldr}}" />; but that’s no obstacle, because it fuses with the <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bunfoldr%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{unfoldr}}" class="latex" title="{\mathit{unfoldr}}" />: </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7BtoBase7%7D+%5Ccdot+%5Cmathit%7Bextract%7D+%26%3D%26+%5Cmathit%7Bunfoldr%7D%5C%3B%5Cmathit%7Bnext%27%7D+%5Cquad+%5Cmathbf%7Bwhere%7D%5C%3B+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bnext%27%7D%5C%3B%280%2Cv%29+%26%3D%26+%5Cmathit%7BNothing%7D+%5C%5C+%5Cmathit%7Bnext%27%7D%5C%3B%28u%2Cv%29+%26%3D%26+%5Cbegin%7Barray%7D%5Bt%5D%7B%40%7B%7Dl%7D+%5Cmathbf%7Blet%7D%5C%3By+%3D+%5Clfloor%7B7+%5Ctimes+u+%5Ctimes+v%7D%5Crfloor%5C%3B%5Cmathbf%7Bin%7D+%5C%5C+%5Cmathit%7BJust%7D%5C%3B%28y%2C%28u+-+y%2F%28v+%5Ctimes+7%29%2C+v+%5Ctimes+7%29%29+%5Cend%7Barray%7D+%5Cend%7Barray%7D+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{toBase7} \cdot \mathit{extract} &=& \mathit{unfoldr}\;\mathit{next'} \quad \mathbf{where}\; \begin{array}[t]{@{}lcl@{}} \mathit{next'}\;(0,v) &=& \mathit{Nothing} \\ \mathit{next'}\;(u,v) &=& \begin{array}[t]{@{}l} \mathbf{let}\;y = \lfloor{7 \times u \times v}\rfloor\;\mathbf{in} \\ \mathit{Just}\;(y,(u - y/(v \times 7), v \times 7)) \end{array} \end{array} \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{toBase7} \cdot \mathit{extract} &=& \mathit{unfoldr}\;\mathit{next'} \quad \mathbf{where}\; \begin{array}[t]{@{}lcl@{}} \mathit{next'}\;(0,v) &=& \mathit{Nothing} \\ \mathit{next'}\;(u,v) &=& \begin{array}[t]{@{}l} \mathbf{let}\;y = \lfloor{7 \times u \times v}\rfloor\;\mathbf{in} \\ \mathit{Just}\;(y,(u - y/(v \times 7), v \times 7)) \end{array} \end{array} \end{array} " />
</p></blockquote>
<p> However, the streaming condition does not hold for <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bstepl%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{stepl}}" class="latex" title="{\mathit{stepl}}" /> and <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bnext%27%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{next'}}" class="latex" title="{\mathit{next'}}" />. For example, </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7B%40%7B%7Dlcl%40%7B%7D%7D+%5Cmathit%7Bnext%27%7D%5C%3B%281%2C%7B%7B%7D%5E%7B1%5C%21%7D%2F_%7B%5C%213%7D%7D%29+%26%3D%26+%5Cmathit%7BJust%7D%5C%3B%282%2C+%28%7B%7B%7D%5E%7B1%5C%21%7D%2F_%7B%5C%217%7D%7D%2C%7B%7B%7D%5E%7B7%5C%21%7D%2F_%7B%5C%213%7D%7D%29%29+%5C%5C+%5Cmathit%7Bnext%27%7D%5C%3B%28%5Cmathit%7Bstepl%7D%5C%3B%281%2C%7B%7B%7D%5E%7B1%5C%21%7D%2F_%7B%5C%213%7D%7D%29%5C%3B1%29+%26%3D%26+%5Cmathit%7Bnext%27%7D%5C%3B%284%2C%7B%7B%7D%5E%7B1%5C%21%7D%2F_%7B%5C%219%7D%7D%29+%5C%5C+%26%3D%26+%5Cmathit%7BJust%7D%5C%3B%283%2C%28%7B%7B%7D%5E%7B1%5C%21%7D%2F_%7B%5C%217%7D%7D%2C%7B%7B%7D%5E%7B7%5C%21%7D%2F_%7B%5C%219%7D%7D%29%29+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \begin{array}{@{}lcl@{}} \mathit{next'}\;(1,{{}^{1\!}/_{\!3}}) &=& \mathit{Just}\;(2, ({{}^{1\!}/_{\!7}},{{}^{7\!}/_{\!3}})) \\ \mathit{next'}\;(\mathit{stepl}\;(1,{{}^{1\!}/_{\!3}})\;1) &=& \mathit{next'}\;(4,{{}^{1\!}/_{\!9}}) \\ &=& \mathit{Just}\;(3,({{}^{1\!}/_{\!7}},{{}^{7\!}/_{\!9}})) \end{array} " class="latex" title="\displaystyle \begin{array}{@{}lcl@{}} \mathit{next'}\;(1,{{}^{1\!}/_{\!3}}) &=& \mathit{Just}\;(2, ({{}^{1\!}/_{\!7}},{{}^{7\!}/_{\!3}})) \\ \mathit{next'}\;(\mathit{stepl}\;(1,{{}^{1\!}/_{\!3}})\;1) &=& \mathit{next'}\;(4,{{}^{1\!}/_{\!9}}) \\ &=& \mathit{Just}\;(3,({{}^{1\!}/_{\!7}},{{}^{7\!}/_{\!9}})) \end{array} " />
</p></blockquote>
<p> That is, <img src="https://s0.wp.com/latex.php?latex=%7B0.1_3+%5Csimeq+0.222_7%7D&bg=ffffff&fg=000000&s=0" alt="{0.1_3 \simeq 0.222_7}" class="latex" title="{0.1_3 \simeq 0.222_7}" />, but <img src="https://s0.wp.com/latex.php?latex=%7B0.11_3+%5Csimeq+0.305_7%7D&bg=ffffff&fg=000000&s=0" alt="{0.11_3 \simeq 0.305_7}" class="latex" title="{0.11_3 \simeq 0.305_7}" />, so it is premature to produce the first digit 2 in base 7 having consumed only the first digit 1 in base 3. The producer <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bnext%27%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{next'}}" class="latex" title="{\mathit{next'}}" /> is too aggressive; it should be more cautious while input remains that might invalidate a produced digit.</p>
<p>
Fortunately, on the assumption that the input digits are all 0, 1, or 2, the unconsumed input—a tail of the original input—again represents a number in the unit interval; so from the state <img src="https://s0.wp.com/latex.php?latex=%7B%28u%2Cv%29%7D&bg=ffffff&fg=000000&s=0" alt="{(u,v)}" class="latex" title="{(u,v)}" /> the range of possible unproduced outputs represents a number between <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bapply%7D%5C%3B%28u%2Cv%29%5C%3B0%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{apply}\;(u,v)\;0}" class="latex" title="{\mathit{apply}\;(u,v)\;0}" /> and <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bapply%7D%5C%3B%28u%2Cv%29%5C%3B1%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{apply}\;(u,v)\;1}" class="latex" title="{\mathit{apply}\;(u,v)\;1}" />. If these both start with the same digit in base 7, then (and only then) is it safe to produce that digit. So we define </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathit%7Bnext%27%27%7D%5C%3B%28u%2Cv%29+%3D+%5Cmathbf%7Bif%7D%5C%3B%5Clfloor%7Bu+%5Ctimes+v+%5Ctimes+7%7D%5Crfloor+%3D+%5Clfloor%7B%28u%2B1%29+%5Ctimes+v+%5Ctimes+7%7D%5Crfloor%5C%3B%5Cmathbf%7Bthen%7D%5C%3B%5Cmathit%7Bnext%27%7D%5C%3B%28u%2Cv%29%5C%3B%5Cmathbf%7Belse%7D%5C%3B%5Cmathit%7BNothing%7D+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \mathit{next''}\;(u,v) = \mathbf{if}\;\lfloor{u \times v \times 7}\rfloor = \lfloor{(u+1) \times v \times 7}\rfloor\;\mathbf{then}\;\mathit{next'}\;(u,v)\;\mathbf{else}\;\mathit{Nothing} " class="latex" title="\displaystyle \mathit{next''}\;(u,v) = \mathbf{if}\;\lfloor{u \times v \times 7}\rfloor = \lfloor{(u+1) \times v \times 7}\rfloor\;\mathbf{then}\;\mathit{next'}\;(u,v)\;\mathbf{else}\;\mathit{Nothing} " />
</p></blockquote>
<p> and we have </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathit%7Bunfoldr%7D%5C%3B%5Cmathit%7Bnext%27%7D+%3D+%5Cmathit%7Bapo%7D%5C%3B%5Cmathit%7Bnext%27%27%7D%5C%3B%28%5Cmathit%7Bunfoldr%7D%5C%3B%5Cmathit%7Bnext%27%7D%29+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \mathit{unfoldr}\;\mathit{next'} = \mathit{apo}\;\mathit{next''}\;(\mathit{unfoldr}\;\mathit{next'}) " class="latex" title="\displaystyle \mathit{unfoldr}\;\mathit{next'} = \mathit{apo}\;\mathit{next''}\;(\mathit{unfoldr}\;\mathit{next'}) " />
</p></blockquote>
<p> Now, the streaming condition holds for <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bstepl%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{stepl}}" class="latex" title="{\mathit{stepl}}" /> and <img src="https://s0.wp.com/latex.php?latex=%7B%5Cmathit%7Bnext%27%27%7D%7D&bg=ffffff&fg=000000&s=0" alt="{\mathit{next''}}" class="latex" title="{\mathit{next''}}" /> (as you may check), and therefore </p>
<blockquote><p>
<img src="https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathit%7BtoBase7%7D%5C%3B%28%5Cmathit%7BfromBase3%7D%5C%3Bx%29+%3D+%5Cmathit%7Bfstream%7D%5C%3B%5Cmathit%7Bnext%27%27%7D%5C%3B%28%5Cmathit%7Bunfoldr%7D%5C%3B%5Cmathit%7Bnext%27%7D%29%5C%3B%5Cmathit%7Bstepl%7D%5C%3B%280%2C1%29%5C%3Bx+&bg=ffffff&fg=000000&s=0" alt="\displaystyle \mathit{toBase7}\;(\mathit{fromBase3}\;x) = \mathit{fstream}\;\mathit{next''}\;(\mathit{unfoldr}\;\mathit{next'})\;\mathit{stepl}\;(0,1)\;x " class="latex" title="\displaystyle \mathit{toBase7}\;(\mathit{fromBase3}\;x) = \mathit{fstream}\;\mathit{next''}\;(\mathit{unfoldr}\;\mathit{next'})\;\mathit{stepl}\;(0,1)\;x " />
</p></blockquote>
<p> on finite digit sequences <img src="https://s0.wp.com/latex.php?latex=%7Bx%7D&bg=ffffff&fg=000000&s=0" alt="{x}" class="latex" title="{x}" /> in base 3. Moreover, the streaming program works also on <em>infinite</em> digit sequences, where the original does not. </p>
<p>
(Actually, the only way this could possibly produce a finite output in base 7 would be for the input to be all zeroes. Why? If we are happy to rule out this case, we could consider only the case of taking infinite input to infinite output, and not have to worry about reaching the end of the input or flushing the state.)</p>
<p><img src="https://pixel.wp.com/b.gif?host=patternsinfp.wordpress.com&blog=15593982&post=288&subd=patternsinfp&ref=&feed=1" alt="" height="1" border="0" width="1" /></p>Wed, 04 Oct 2017 09:45:14 +0000Douglas M. Auclair (geophf): September 2017 1HaskellADay 1Liner problems and solutionstag:blogger.com,1999:blog-4650294074444534066.post-8610335132545287661
http://logicaltypes.blogspot.com/2017/10/september-2017-1haskelladay-1liner.html
<ul style="color: #333333; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18px; margin: 0.5em 0px; padding: 0px 2.5em;"><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 26th, 2017:<br />art2RawNames art = Raw (artId art) <$> (Map.lookup "Person" $ metadata art)<br />curry away the 'art' var. ref: <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D26/Exercise.hs#L84-L85">Y2017.M09.D26.Exercise</a> </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 19th, 2017: The #1Liner to follow (next tweet) is a question based off <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D18/Solution.hs#L70-L71">a code snippet</a> from Y2017.M09.D18.Solution. What is a better/more elegant definition?<br /><blockquote>fmap (wordStrength . wc) (BL.readFile filename) >>= \dict -><br />return (Map.insert filename dict m)</blockquote>Reformulate. Curry the dict-ref.</li></ul>Tue, 03 Oct 2017 18:58:04 +0000noreply@blogger.com (geophf)Douglas M. Auclair (geophf): September 2017 1HaskellADay problems and solutionstag:blogger.com,1999:blog-4650294074444534066.post-400621163054099473
http://logicaltypes.blogspot.com/2017/09/september-2017-1haskelladay-problems.html
<ul style="color: #333333; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 1.4; margin: 0.5em 0px; padding: 0px 2.5em;"><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 29th, 2017: Friday's #haskell problem <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D29/Exercise.hs">uses name-parsing to post parsed names into PostgreSQL database</a> and join them to articles. Today's #haskell solution <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D29/Solution.hs">extracts rows of raw names associated with articles, parses then stores them</a> in PostgreSQL.<div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://3.bp.blogspot.com/-O2HI5yagWoc/Wc6tpH7xyTI/AAAAAAAAB34/L36P8RY9L20oJjugwLnlSkcKgQ2i54GZACLcBGAs/s1600/person-table-populated.png"><img src="https://3.bp.blogspot.com/-O2HI5yagWoc/Wc6tpH7xyTI/AAAAAAAAB34/L36P8RY9L20oJjugwLnlSkcKgQ2i54GZACLcBGAs/s320/person-table-populated.png" style="" height="244" border="0" width="320" /></a></div></li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 28th, 2017: Thursday's #haskell problem <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D28/Exercise.hs">inserts reified articles into PostgreSQL database</a>, including names, and throws confetti! Today's #haskell solution <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D28/Solution.hs">inserts the articles from our compressed archive</a>. YAY! Throw confetti! <div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://3.bp.blogspot.com/-PlxoXB2aBz4/WcwmC6VA5eI/AAAAAAAAB3I/99Q_qUKzEtAnOanWHRkRK6xZONkcOEHDQCLcBGAs/s1600/YAY-confetti.png"><img src="https://3.bp.blogspot.com/-PlxoXB2aBz4/WcwmC6VA5eI/AAAAAAAAB3I/99Q_qUKzEtAnOanWHRkRK6xZONkcOEHDQCLcBGAs/s320/YAY-confetti.png" style="" height="238" border="0" width="320" /></a><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://4.bp.blogspot.com/-sE-FrI4S2o4/Wc1TsRhkLXI/AAAAAAAAB3g/r4NWcieO7t4qZI63nXEpl2EJnr5PSK3LACLcBGAs/s1600/muddy-waters-blues.png"><img src="https://4.bp.blogspot.com/-sE-FrI4S2o4/Wc1TsRhkLXI/AAAAAAAAB3g/r4NWcieO7t4qZI63nXEpl2EJnr5PSK3LACLcBGAs/s320/muddy-waters-blues.png" style="" height="144" border="0" width="320" /></a></div></li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 27th, 2017: Wednesday's #haskell problem looks at <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D27/Exercise.hs">parsing names in various formats</a>. Today's #haskell solution solves the name-parsing problem by <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D27/Solution.hs">parsing each name-form in turn</a>. </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 26th, 2017: Tuesday's #haskell problem looks at <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D26/Exercise.hs">staging data on PostgreSQL for deferred processing</a> using Haskell for ETL. Today's #haskell solution <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D26/Solution.hs">transforms RawNames to JSON and uploads it to a staging table on the PostgreSQL database</a>.<div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://4.bp.blogspot.com/-Eo7B6S7RHrM/WclvDJXgMBI/AAAAAAAAB2Y/Zq_YAp-_hV0r7P1v9XRU3O6gBcWAQD8tQCLcBGAs/s1600/nyt-name-stg.png"><img src="https://4.bp.blogspot.com/-Eo7B6S7RHrM/WclvDJXgMBI/AAAAAAAAB2Y/Zq_YAp-_hV0r7P1v9XRU3O6gBcWAQD8tQCLcBGAs/s320/nyt-name-stg.png" style="" height="130" border="0" width="320" /></a> <a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://2.bp.blogspot.com/-RoqB3vypBTI/WcrWaI7DzhI/AAAAAAAAB2w/npxrLRSn2qgQaVtiM6BbLgx0yEy5VWbCQCLcBGAs/s1600/nyt-names-etl.png"><img src="https://2.bp.blogspot.com/-RoqB3vypBTI/WcrWaI7DzhI/AAAAAAAAB2w/npxrLRSn2qgQaVtiM6BbLgx0yEy5VWbCQCLcBGAs/s320/nyt-names-etl.png" style="" height="222" border="0" width="320" /></a></div></li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 25th, 2017: Friday we took a compressed archive and broke that into individual articles. Today we <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D25/Exercise.hs">divide articles into sections</a>. Today's #haskell solution <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D25/Solution.hs">uncovers structure bit by bit</a> as we reify an article structure from a block of text. </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 22nd, 2017: Today's #haskell problem we're going to <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D22/Exercise.hs">scan into an 'unstructured' set of documents and start to look for structure</a>. Welp, today's #haskell solution is a 'start': from <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D22/Solution.hs">one blob to 11 blocks of raw text</a>. We'll dig in deeper next week. </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 21st, 2017: TOMORROW's #haskell problem (announced a wee bit early) queries PostgreSQL database to <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D21/Exercise.hs">fetch keyword dictionary state</a>. Today's #haskell solution <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D21/Solution.hs">picks up where we left off yesterday then computes the top keywords across articles</a>. <div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://2.bp.blogspot.com/-7LaLMFaNjeM/WcQbuhCHoKI/AAAAAAAAB2A/jGW-120uiA08BYqEicbOPbgtyLsruMjSACLcBGAs/s1600/01-top10-keywords.png"><img src="https://2.bp.blogspot.com/-7LaLMFaNjeM/WcQbuhCHoKI/AAAAAAAAB2A/jGW-120uiA08BYqEicbOPbgtyLsruMjSACLcBGAs/s320/01-top10-keywords.png" style="" height="247" border="0" width="320" /></a></div></li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 20th, 2017: Today's #haskell problem asks to push <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D20/Exercise.hs">keywords extracted from article onto a PostgreSQL</a> data table. Today's #haskell solution pushes keywords – <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D20/Solution.hs">PUSHES KEYWORDS REAL GOOD</a> – to a PostgreSQL database. <div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://4.bp.blogspot.com/-sagqKCf7mxI/WcLOADBn7QI/AAAAAAAAB1k/dZEHL_PpqzUHq90ZMgEk0IhGDXIzeOJ0QCLcBGAs/s1600/01-key-dictionary.png"><img src="https://4.bp.blogspot.com/-sagqKCf7mxI/WcLOADBn7QI/AAAAAAAAB1k/dZEHL_PpqzUHq90ZMgEk0IhGDXIzeOJ0QCLcBGAs/s320/01-key-dictionary.png" style="" height="221" border="0" width="320" /></a></div><br /><div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://4.bp.blogspot.com/-O-tjdvBOX_4/WcLOB4Ng5QI/AAAAAAAAB1o/aBsCR8V1cIIHrzew53_gXb6-PB3UO9FGgCLcBGAs/s1600/02-keywords-in-articles.png"><img src="https://4.bp.blogspot.com/-O-tjdvBOX_4/WcLOB4Ng5QI/AAAAAAAAB1o/aBsCR8V1cIIHrzew53_gXb6-PB3UO9FGgCLcBGAs/s320/02-keywords-in-articles.png" style="" height="223" border="0" width="320" /></a></div></li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 19th, 2017: So we've encoded keywords into Ints, today's #haskell problem <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D19/Exercise.hs">decodes those ints back to the keywords</a>. Today's #haskell solution <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D19/Solution.hs">'reverses the arrows' of the Map</a>, decoding a keyword from an index </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 18th, 2017: Today's #haskell problem computes the <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D18/Exercise.hs">relative word 'strength'</a> of each word in a document. Today's #haskell solution breaks the <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D18/Solution.hs">word-strength problem into small pieces then foldM's over the small pieces</a>. </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 15th, 2017: Today's #haskell problem takes a look at <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D15/Exercise.hs">KEA/Keyword Extraction Algorithm</a>. Ooh, doggies! <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D15/Solution.hs">State IO Map KeyWord something-or-other</a>for today's #haskell solution.</li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 14th, 2017: Today's #haskell problem involves using the <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D14/Exercise.hs">simple interface to PostgreSQL</a> to insert rows of data. Ooh! Exciting! Today's #haskell solution uses <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D14/Solution.hs">ToRow and ToField to insert Haskell values</a> into #PostgreSQL #DaaS instance! lolneat! <div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://1.bp.blogspot.com/-aaf6VZGbMxQ/WbsneXHViZI/AAAAAAAAB1I/rO4EmBnMkd4zhAC2z4-z5uczHADFV02MwCLcBGAs/s1600/db1-row-sample.png"><img src="https://1.bp.blogspot.com/-aaf6VZGbMxQ/WbsneXHViZI/AAAAAAAAB1I/rO4EmBnMkd4zhAC2z4-z5uczHADFV02MwCLcBGAs/s320/db1-row-sample.png" style="" height="257" border="0" width="320" /></a></div></li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 13th, 2017: Today's #haskell problem asks the article names have <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D13/Exercise.hs">encoded Julian dates</a> ... OR DO THEY? So, in solving today's #haskell problem, I realized THOSE AREN'T JULIAN DATES! THOSE ARE <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D13/Solution.hs">DATE DATES</a>! </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 8th, 2017: Continuing our ETL exploration with a <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D08/Exercise.hs">load/compress/store</a>for today's #haskell exercise. Today's #haskell <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D08/Solution.hs">solution shows load/compress/store</a> ... that's a word now. </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 5th, 2017: Today's #haskell exercise is to <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D05/Exercise.hs">build a directory web service</a>using the Snap framework [or a framework you prefer] </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 4th, 2017: Today's #haskell problem reads in source documents and does some <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D04/Exercise.hs">word frequency analyses</a>. Today's #haskell solution shows, surprisingly, <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D04/Solution.hs">'the' is the most popular English word</a>. Or, not so surprisingly. </li><li style="border: none; margin: 0px 0px 0.25em; padding: 0.25em 0px;">September 1st, 2017: Today's #haskell problem #SPARQL-queries wikidata.org to fetch <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D01/Exercise.hs">countries' populations</a> for FB analysis. Yesterday, India won FB by population, today we have <a style="color: #7d181e; text-decoration: none;" href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M09/D01/Solution.hs">a surprise winner in Thailand for FB by percentage user base</a>. </li><div style="clear: both; text-align: center;" class="separator"><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://4.bp.blogspot.com/-44teg8Z4TkY/Wal2_ot0B4I/AAAAAAAAB0Y/U5s6ddnsvQwrJnsGcDxX6HGkcuozMHl4QCLcBGAs/s1600/population-by-country.png"><img src="https://4.bp.blogspot.com/-44teg8Z4TkY/Wal2_ot0B4I/AAAAAAAAB0Y/U5s6ddnsvQwrJnsGcDxX6HGkcuozMHl4QCLcBGAs/s320/population-by-country.png" style="" height="210" border="0" width="320" /></a><a style="color: #7d181e; margin-left: 1em; margin-right: 1em; text-decoration: none;" href="https://4.bp.blogspot.com/-MAHR2c-hKvE/War74y5bdyI/AAAAAAAAB0w/ZfT1ml8HblsuxQbYgmuzvDW_j5pBNlIpQCLcBGAs/s1600/percent-fb-users-by-country.png"><img src="https://4.bp.blogspot.com/-MAHR2c-hKvE/War74y5bdyI/AAAAAAAAB0w/ZfT1ml8HblsuxQbYgmuzvDW_j5pBNlIpQCLcBGAs/s320/percent-fb-users-by-country.png" style="" height="186" border="0" width="320" /></a></div></ul>Sat, 30 Sep 2017 14:24:05 +0000noreply@blogger.com (geophf)FP Complete: Yesod Tutorial 1. My First Web Sitehttps://www.fpcomplete.com/blog/2012/10/yesod-tutorial-1-my-first-web-site
https://www.fpcomplete.com/blog/2012/10/yesod-tutorial-1-my-first-web-site
<p>This is the first in the series of tutorials introducing a new approach to web development using Haskell and Yesod. Haskell is a functional language and Yesod is a web development framework written in Haskell by Michael Snoyman who also wrote a book about it. Michael is a member of the FP Complete team and he provided a lot of valuable feedback for this tutorial.</p>
<img src="http://track.hubspot.com/__ptq.gif?a=2814979&k=14&r=https%3A%2F%2Fwww.fpcomplete.com%2Fblog%2F2012%2F10%2Fyesod-tutorial-1-my-first-web-site&bu=https%253A%252F%252Fwww.fpcomplete.com%252Fblog&bvt=rss" alt="" height="1" style="width: 1px!important;" width="1" />Thu, 28 Sep 2017 23:34:38 +0000bartosz@fpcomplete.com (Bartosz Milewski)FP Complete: It's time for Functional Programminghttps://www.fpcomplete.com/blog/2012/03/onamission
https://www.fpcomplete.com/blog/2012/03/onamission
<em>by Aaron Contorer - CEO</em>
<p>In 1999 I went to Bill Gates with an idea to create a software tools group dedicated to shipping complex software faster. Engineers’ time is valuable, and more importantly, software that ships on time with fewer defects is worth a lot. I organized a team that analyzed what was missing from the old toolsets for our most valuable products. Based on developers’ input, we conceived and delivered five in-house tools within our first year, spanning areas from build to source control to localization. I'm proud of the talented and motivated people who chose to be on that team, and the positive impact it had: If you have a copy of Windows or Office today, it probably arrived on your desk a little sooner and a little better thanks to the Productivity Tools Team. From a business point of view, Microsoft got its tools investment back many times over.</p>
<img src="http://track.hubspot.com/__ptq.gif?a=2814979&k=14&r=https%3A%2F%2Fwww.fpcomplete.com%2Fblog%2F2012%2F03%2Fonamission&bu=https%253A%252F%252Fwww.fpcomplete.com%252Fblog&bvt=rss" alt="" height="1" style="width: 1px!important;" width="1" />Thu, 28 Sep 2017 23:34:07 +0000aaron@fpcomplete.com (Aaron Contorer)FP Complete: Asynchronous API in C++ and the Continuation Monadhttps://www.fpcomplete.com/blog/2012/06/asynchronous-api-in-c-and-the-continuation-monad
https://www.fpcomplete.com/blog/2012/06/asynchronous-api-in-c-and-the-continuation-monad
The latest C++11 Standard was a brave attempt, after many years of neglect, at catching up with the reality of concurrent programming. The result is a new memory model -- a solid foundation for concurrency -- and a collection of random concurrency features, many of them either obsolete or inadequate. But there is hope on the horizon: the next C++ Standard, on which the work started even before C++11 saw the light of day. In my last blog I
<a href="http://bartoszmilewski.com/2012/05/11/the-future-of-c-concurrency-and-parallelism/" title="C++ Standard Meeting report">reported</a> on the first meeting of the C++ study group on concurrency and parallelism that took place in Bellevue, WA.
<div style="float: right; text-align: center; width: 80px;">
<img src="http://upload.wikimedia.org/wikipedia/en/thumb/9/98/ErikMeijer.jpg/159px-ErikMeijer.jpg" alt="Erik Meijer" /> Erik Meijer
</div> One of the hot topics at that meeting was building better support for asynchronous calls, either through libraries or new language extensions. The C# group dazzled us with their very mature solution. Honestly, I am impressed with their work. And I know
<img src="http://track.hubspot.com/__ptq.gif?a=2814979&k=14&r=https%3A%2F%2Fwww.fpcomplete.com%2Fblog%2F2012%2F06%2Fasynchronous-api-in-c-and-the-continuation-monad&bu=https%253A%252F%252Fwww.fpcomplete.com%252Fblog&bvt=rss" alt="" height="1" style="width: 1px!important;" width="1" />Thu, 28 Sep 2017 23:33:35 +0000bartosz@fpcomplete.com (Bartosz Milewski)FP Complete: The Downfall of Imperative Programminghttps://www.fpcomplete.com/blog/2012/04/the-downfall-of-imperative-programming
https://www.fpcomplete.com/blog/2012/04/the-downfall-of-imperative-programming
<p>Imperative programming is in my bloodstream. I've been a C++ programmer for most of my life. I wrote a book about C++. I helped Andrei and Walter design an imperative language D. If I dabbled in functional programming, it was mostly to make my imperative programs better.</p>
<img src="http://track.hubspot.com/__ptq.gif?a=2814979&k=14&r=https%3A%2F%2Fwww.fpcomplete.com%2Fblog%2F2012%2F04%2Fthe-downfall-of-imperative-programming&bu=https%253A%252F%252Fwww.fpcomplete.com%252Fblog&bvt=rss" alt="" height="1" style="width: 1px!important;" width="1" />Thu, 28 Sep 2017 23:33:24 +0000bartosz@fpcomplete.com (Bartosz Milewski)FP Complete: The Functor Pattern in C++https://www.fpcomplete.com/blog/2012/07/the-functor-pattern-in-c
https://www.fpcomplete.com/blog/2012/07/the-functor-pattern-in-c
<p>The <a href="http://en.wikipedia.org/wiki/Design_Patterns">Gang of Four book</a> introduced many a C++ programmer to the world of patterns. Those patterns are mostly based on everyday examples, although there are a few more abstract ones like Strategy. In Haskell (and other functional languages) most patterns are very abstract. They are often based on mathematical constructs; category theory being a major source of ideas.</p>
<img src="http://track.hubspot.com/__ptq.gif?a=2814979&k=14&r=https%3A%2F%2Fwww.fpcomplete.com%2Fblog%2F2012%2F07%2Fthe-functor-pattern-in-c&bu=https%253A%252F%252Fwww.fpcomplete.com%252Fblog&bvt=rss" alt="" height="1" style="width: 1px!important;" width="1" />Thu, 28 Sep 2017 23:33:21 +0000bartosz@fpcomplete.com (Bartosz Milewski)Functional Jobs: Engineering Manager at Takt (Full-time)urn:uuid:160e3d95-bed3-c4c1-7f2c-4adea8340f49
https://functionaljobs.com/jobs/9039-engineering-manager-at-takt
<p>Takt’s Engineering organization is growing rapidly, and we’re looking for Engineering Managers to lead and scale the team. Takt distills complex customer data into uniquely tailored experiences; we orchestrate physical and digital exchanges into one seamless journey. Our business is building lasting, trusted relationships between people and brands — and making it look easy.</p>
<p>As an Engineering Manager at Takt, you deeply value and understand the work your team is accomplishing, and serve to inspired and develop each individual while ensuring business objectives are met. As we continue to expand our business, your team will build a rich, complex real-time product that serves our enterprise partners and their end customers while you enrich the lives and careers of your team. </p>
<p>Key Responsibilities:</p>
<ul>
<li>Build and manage a team of up to 8 software engineers, including coaching, mentorship, and performance management</li>
<li>Develop and optimize development processes and practices to maintain the balance between building a complex innovative product with shipping quickly</li>
<li>Instill a pragmatic, delivery-focused support-oriented mindset that values long-term efficiency and value</li>
<li>Collaborate with engineers and software architects to design and develop software components for current and prospective Takt clients</li>
<li>Be accountable for technical components, including planning, prioritization, and delivery</li>
<li>Partner with the recruiting organization to fill the pipeline with exceptional talent, promote referrals, nurture relationships through informal meetings, and serve as an ambassador for Takt at events and conferences</li>
<li>Promote continuous feedback, address growth areas for team members, and celebrate strengths and contributions of your team</li>
<li>Be emblematic of Takt’s values and lead by example, set the right context, and inspire the team to do their best work by fostering an environment that’s open, transparent and respectful of each other</li>
<li>Develop Engineering onboarding processes and documentation practices, ensure new team members have a seamless experience and are productive quickly</li>
</ul>
<p>Skills and Experience:</p>
<ul>
<li>Prior experience in technical leadership and proven ability to build and develop successful teams consisting of varied levels of experience</li>
<li>Technical expertise with Functional Programming in Scala or Haskell</li>
<li>Demonstrated understanding of software design and a deep appreciation for Customer Success</li>
<li>Excellent interpersonal, written, presentation and closing skills</li>
<li>Prior success with hiring top technical talent and contributing to recruiting processes</li>
<li>Experience running projects and understanding of agile development methodologies</li>
<li>Select teams may have domain specific requirements (e.g. Machine Learning, Infrastructure, Security)</li>
<li>Experience managing team members in a distributed environment is a plus</li>
<li>Both mid-large company and startup experience would be a plus</li>
</ul>
<p>About Takt:</p>
<p>Takt distills complex customer data into uniquely tailored experiences; we orchestrate physical and digital exchanges into one seamless journey. Our business is building lasting, trusted relationships between people and brands—and making it look easy.</p>
<p>We're already reaching millions of people a day, and we're just getting started. Our founding leadership is equal parts business, design, and engineering—because we believe differing perspectives + passionate discourse achieve the greatest outcomes. We are collectively talented, but also humble. We give our whole selves. We love learning new things.</p>
<p>Takt is committed to inclusion and diversity and is an equal opportunity employer. All applicants will receive consideration without regard to race, color, religion, gender, gender identity, sexual orientation, national origin, disability, or veteran status.</p>
<p>If you're up for the challenge of a lifetime, we're looking for outstanding talent to join our team.</p>
<p>Get information on <a href="https://functionaljobs.com/jobs/9039-engineering-manager-at-takt">how to apply</a> for this position.</p>Thu, 28 Sep 2017 20:09:15 +0000Gabriel Gonzalez: Type-driven strictnesstag:blogger.com,1999:blog-1777990983847811806.post-921143709736397249
http://www.haskellforall.com/2017/09/type-driven-strictness.html
<html xmlns="http://www.w3.org/1999/xhtml"><head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <meta content="text/css" http-equiv="Content-Style-Type"/> <meta content="pandoc" name="generator"/> <title></title> <style type="text/css">code{white-space: pre;}</style> <style type="text/css">div.sourceCode { overflow-x: auto; } table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode { margin: 0; padding: 0; vertical-align: baseline; border: none; } table.sourceCode { width: 100%; line-height: 100%; } td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; } td.sourceCode { padding-left: 5px; } code > span.kw { color: #007020; font-weight: bold; } /* Keyword */ code > span.dt { color: #902000; } /* DataType */ code > span.dv { color: #40a070; } /* DecVal */ code > span.bn { color: #40a070; } /* BaseN */ code > span.fl { color: #40a070; } /* Float */ code > span.ch { color: #4070a0; } /* Char */ code > span.st { color: #4070a0; } /* String */ code > span.co { color: #60a0b0; font-style: italic; } /* Comment */ code > span.ot { color: #007020; } /* Other */ code > span.al { color: #ff0000; font-weight: bold; } /* Alert */ code > span.fu { color: #06287e; } /* Function */ code > span.er { color: #ff0000; font-weight: bold; } /* Error */ code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ code > span.cn { color: #880000; } /* Constant */ code > span.sc { color: #4070a0; } /* SpecialChar */ code > span.vs { color: #4070a0; } /* VerbatimString */ code > span.ss { color: #bb6688; } /* SpecialString */ code > span.im { } /* Import */ code > span.va { color: #19177c; } /* Variable */ code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ code > span.op { color: #666666; } /* Operator */ code > span.bu { } /* BuiltIn */ code > span.ex { } /* Extension */ code > span.pp { color: #bc7a00; } /* Preprocessor */ code > span.at { color: #7d9029; } /* Attribute */ code > span.do { color: #ba2121; font-style: italic; } /* Documentation */ code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ </style></head><body><p>I was recently trying to optimize Dhall's performance because the interpreter was performing poorly on some simple examples.</p><p>For example, consider this tiny expression that prints 3000 exclamation marks:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="dt">Natural</span><span class="fu">/</span>fold <span class="fu">+</span><span class="dv">3000</span> <span class="dt">Text</span> (λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>) <span class="st">""</span></code></pre></div><p>The above Dhall expression takes over 14 seconds to evaluate to normal form, which is not acceptable:</p><pre><code>$ bench 'dhall <<< './exclaim'<br />benchmarking dhall <<< ./exclaim<br />time 14.42 s (14.23 s .. 14.57 s)<br /> 1.000 R² (1.000 R² .. 1.000 R²)<br />mean 14.62 s (14.54 s .. 14.66 s)<br />std dev 70.22 ms (0.0 s .. 77.84 ms)<br />variance introduced by outliers: 19% (moderately inflated)</code></pre><h2 id="strict">Strict</h2><p>The performance suffers because Dhall is lazy in the accumulator, meaning that the accumulator builds up a gigantic expression like this:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">(λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>)<br />( (λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>)<br /> ( (λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>)<br /> ( (λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>)<br /> ( (λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>)<br /> <span class="fu">...</span><br /> {<span class="dt">Repeat</span> this nesting <span class="dv">3000</span> times}<br /> <span class="fu">...</span><br /> <span class="st">""</span><br /> )<br /> )<br /> )<br />)</code></pre></div><p>... and then attempts to normalize the entire expression, which takes a long time and wastes a lot of memory.</p><p>The reason for this lazy behavior is the following code for evaluating <code>Natural/fold</code> in the Dhall interpreter:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">normalize (<span class="dt">App</span> (<span class="dt">App</span> (<span class="dt">App</span> (<span class="dt">App</span> <span class="dt">NaturalFold</span> (<span class="dt">NaturalLit</span> n0)) _) succ') zero)<br /> <span class="fu">=</span> normalize (go n0)<br /> <span class="kw">where</span><br /> go <span class="fu">!</span><span class="dv">0</span> <span class="fu">=</span> zero<br /> go <span class="fu">!</span>n <span class="fu">=</span> <span class="dt">App</span> succ' (go (n <span class="fu">-</span> <span class="dv">1</span>))</code></pre></div><p>You can read that as saying:</p><ul><li>Given an expression of the form <code>Natural/Fold n succ zero</code></li><li>Wrap the value <code>zero</code> in <code>n</code> calls of the function <code>succ</code><ul><li>i.e. <code>succ (succ (succ (... {n times} ... (succ zero) ...)))</code></li></ul></li><li>Then normalize that</li></ul><p>A smarter approach would be to keep the accumulator strict, which means that we evaluate as we go instead of deferring all evaluation to the end. For example, the accumulator starts off as just the empty string:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="st">""</span></code></pre></div><p>... then after one iteration of the loop we get the following accumulator:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">(λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>) <span class="st">""</span></code></pre></div><p>... and if we evaluate that accumulator immediately we get:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="st">"!"</span></code></pre></div><p>Then the next iteration of the loop produces the following accumulator:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">(λ(x <span class="fu">:</span> <span class="dt">Text</span>) <span class="ot">→</span> x <span class="fu">++</span> <span class="st">"!"</span>) <span class="st">"!"</span></code></pre></div><p>... which we can again immediately evaluate to get:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="st">"!!"</span></code></pre></div><p>This is significantly more efficient than leaving the expression unevaluated.</p><p>We can easily implement such a strict loop by making the following change to the interpreter:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">normalize (<span class="dt">App</span> (<span class="dt">App</span> (<span class="dt">App</span> (<span class="dt">App</span> <span class="dt">NaturalFold</span> (<span class="dt">NaturalLit</span> n0)) _) succ') zero)<br /> <span class="fu">=</span> go n0<br /> <span class="kw">where</span><br /> go <span class="fu">!</span><span class="dv">0</span> <span class="fu">=</span> normalize zero<br /> go <span class="fu">!</span>n <span class="fu">=</span> normalize (<span class="dt">App</span> succ' (go (n <span class="fu">-</span> <span class="dv">1</span>)))</code></pre></div><p>The difference here is that we still build up a chain of <code>n</code> calls to <code>succ</code> but now we normalize our expression in between each call to the <code>succ</code> function instead of waiting until the end to <code>normalize</code>.</p><p>Once we do this runtime improves dramatically, going down from 15 seconds to 90 milliseconds:</p><pre><code>$ bench 'dhall <<< './example'<br />benchmarking dhall <<< ./example<br />time 88.92 ms (87.14 ms .. 90.74 ms)<br /> 0.999 R² (0.999 R² .. 1.000 R²)<br />mean 86.06 ms (84.98 ms .. 87.15 ms)<br />std dev 1.734 ms (1.086 ms .. 2.504 ms)</code></pre><p>... or in other words about 30 microseconds per element. We could still do more to optimize this but at least we're now in the right ballpark for an interpreter. For reference, Python is 4x faster on my machine for the following equivalent program:</p><div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="bu">print</span> (<span class="st">"!"</span> <span class="op">*</span> <span class="dv">3000</span>)</code></pre></div><pre><code>$ bench 'python exclaim.py'<br />benchmarking python exclaim.py<br />time 24.55 ms (24.09 ms .. 25.02 ms)<br /> 0.998 R² (0.996 R² .. 1.000 R²)<br />mean 24.53 ms (24.16 ms .. 24.88 ms)<br />std dev 798.4 μs (559.8 μs .. 1.087 ms)</code></pre><p>However, these results don't necessarily imply that a strict accumulator is always better.</p><h2 id="lazy">Lazy</h2><p>Sometimes laziness is more efficient, though. Consider this program:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="dt">List</span><span class="fu">/</span>build<br /><span class="dt">Integer</span><br />( λ(list <span class="fu">:</span> <span class="dt">Type</span>)<br /><span class="ot">→</span> λ(cons <span class="fu">:</span> <span class="dt">Integer</span> <span class="ot">→</span> list <span class="ot">→</span> list)<br /><span class="ot">→</span> <span class="dt">Natural</span><span class="fu">/</span>fold <span class="fu">+</span><span class="dv">6000</span> list (cons <span class="dv">1</span>)<br />)</code></pre></div><p>The above example uses <code>Natural/fold</code> to build a list of 6000 <code>1</code>s.</p><p>In this case the accumulator of the fold is a list that grows by one element after each step of the fold. We don't want to normalize the list on each iteration because that would lead to quadratic time complexity. Instead we prefer to defer normalization to the end of the loop so that we get linear time complexity.</p><p>We can measure the difference pretty easily. A strict loop takes over 6 seconds to complete:</p><pre><code>bench 'dhall <<< ./ones'<br />benchmarking dhall <<< ./ones<br />time 6.625 s (6.175 s .. 7.067 s)<br /> 0.999 R² (0.998 R² .. 1.000 R²)<br />mean 6.656 s (6.551 s .. 6.719 s)<br />std dev 95.98 ms (0.0 s .. 108.3 ms)<br />variance introduced by outliers: 19% (moderately inflated)</code></pre><p>... whereas a lazy loop completes in about 180 milliseconds:</p><pre><code>$ bench 'dhall <<< ./g'<br />benchmarking dhall <<< ./g<br />time 182.5 ms (175.1 ms .. 191.3 ms)<br /> 0.998 R² (0.995 R² .. 1.000 R²)<br />mean 177.5 ms (172.1 ms .. 180.8 ms)<br />std dev 5.589 ms (2.674 ms .. 8.273 ms)<br />variance introduced by outliers: 12% (moderately inflated)</code></pre><p>Moreover, the difference in performance will only worsen with larger list sizes due to the difference in time complexity.</p><p>Also, in case you were wondering, Python is about 7x faster:</p><div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="bu">print</span> ([<span class="dv">1</span>] <span class="op">*</span> <span class="dv">6000</span>)</code></pre></div><pre><code>$ bench 'python ones.py'<br />benchmarking python ones.py<br />time 25.36 ms (24.75 ms .. 25.92 ms)<br /> 0.998 R² (0.996 R² .. 0.999 R²)<br />mean 25.64 ms (25.16 ms .. 26.03 ms)<br />std dev 917.8 μs (685.7 μs .. 1.348 ms)</code></pre><h1 id="why-not-both">Why not both?</h1><p>This poses a conundrum because we'd like to efficiently support both of these use cases. How can we know when to be lazy or strict?</p><p>We can use Dhall's type system to guide whether or not we keep the accumulator strict. We already have access to the type of the accumulator for our loop, so we can define a function that tells us if our accumulator type is compact or not:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">compact ::</span> <span class="dt">Expr</span> s a <span class="ot">-></span> <span class="dt">Bool</span><br />compact <span class="dt">Bool</span> <span class="fu">=</span> <span class="dt">True</span><br />compact <span class="dt">Natural</span> <span class="fu">=</span> <span class="dt">True</span><br />compact <span class="dt">Integer</span> <span class="fu">=</span> <span class="dt">True</span><br />compact <span class="dt">Double</span> <span class="fu">=</span> <span class="dt">True</span><br />compact <span class="dt">Text</span> <span class="fu">=</span> <span class="dt">True</span><br />compact (<span class="dt">App</span> <span class="dt">List</span> _) <span class="fu">=</span> <span class="dt">False</span><br />compact (<span class="dt">App</span> <span class="dt">Optional</span> t) <span class="fu">=</span> compact t<br />compact (<span class="dt">Record</span> kvs) <span class="fu">=</span> all compact kvs<br />compact (<span class="dt">Union</span> kvs) <span class="fu">=</span> all compact kvs<br />compact _ <span class="fu">=</span> <span class="dt">False</span></code></pre></div><p>You can read this function as saying:</p><ul><li>primitive types are compact</li><li>lists are not compact</li><li>optional types are compact if the corresponding non-optional type is compact</li><li>a record type is compact if all the field types are compact</li><li>a union type is compact if all the alternative types are compact</li></ul><p>Now, all we need to do is modify our fold logic to use this <code>compact</code> function to decide whether or not we use a strict or lazy loop:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">normalize (<span class="dt">App</span> (<span class="dt">App</span> (<span class="dt">App</span> (<span class="dt">App</span> <span class="dt">NaturalFold</span> (<span class="dt">NaturalLit</span> n0)) t) succ') zero) <span class="fu">=</span><br /> <span class="kw">if</span> compact (normalize t) <span class="kw">then</span> strict <span class="kw">else</span> lazy<br /> <span class="kw">where</span><br /> strict <span class="fu">=</span> strictLoop n0<br /> lazy <span class="fu">=</span> normalize ( lazyLoop n0)<br /><br /> strictLoop <span class="fu">!</span><span class="dv">0</span> <span class="fu">=</span> normalize zero<br /> strictLoop <span class="fu">!</span>n <span class="fu">=</span> normalize (<span class="dt">App</span> succ' (strictLoop (n <span class="fu">-</span> <span class="dv">1</span>)))<br /><br /> lazyLoop <span class="fu">!</span><span class="dv">0</span> <span class="fu">=</span> zero<br /> lazyLoop <span class="fu">!</span>n <span class="fu">=</span> <span class="dt">App</span> succ' (lazyLoop (n <span class="fu">-</span> <span class="dv">1</span>))</code></pre></div><p>Now we get the best of both worlds and our interpreter gives excellent performance in both of the above examples.</p><h2 id="fizzbuzz">Fizzbuzz</h2><p>Here's a bonus example for people who got this far!</p><p>The original Dhall expression that motivated this post was an attempt to implement <a href="https://rosettacode.org/wiki/FizzBuzz">FizzBuzz</a> in Dhall. The program I ended up writing was:</p><div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"> <span class="kw">let</span> pred <span class="fu">=</span><br /> λ(n <span class="fu">:</span> <span class="dt">Natural</span>)<br /> <span class="ot">→</span> <span class="kw">let</span> start <span class="fu">=</span> { next <span class="fu">=</span> <span class="fu">+</span><span class="dv">0</span>, prev <span class="fu">=</span> <span class="fu">+</span><span class="dv">0</span> }<br /> <br /> <span class="kw">in</span> <span class="kw">let</span> step <span class="fu">=</span><br /> λ(state <span class="fu">:</span> { next <span class="fu">:</span> <span class="dt">Natural</span>, prev <span class="fu">:</span> <span class="dt">Natural</span> })<br /> <span class="ot">→</span> { next <span class="fu">=</span> state<span class="fu">.</span>next <span class="fu">+</span> <span class="fu">+</span><span class="dv">1</span>, prev <span class="fu">=</span> state<span class="fu">.</span>next }<br /> <br /> <span class="kw">in</span> <span class="kw">let</span> result <span class="fu">=</span><br /> <span class="dt">Natural</span><span class="fu">/</span>fold<br /> n<br /> { next <span class="fu">:</span> <span class="dt">Natural</span>, prev <span class="fu">:</span> <span class="dt">Natural</span> }<br /> step<br /> start<br /> <br /> <span class="kw">in</span> result<span class="fu">.</span>prev<br /><br /><span class="kw">in</span> <span class="kw">let</span> not <span class="fu">=</span> λ(b <span class="fu">:</span> <span class="dt">Bool</span>) <span class="ot">→</span> b <span class="fu">==</span> <span class="dt">False</span><br /><br /><span class="kw">in</span> <span class="kw">let</span> succ <span class="fu">=</span><br /> λ ( state<br /> <span class="fu">:</span> { buzz <span class="fu">:</span> <span class="dt">Natural</span>, fizz <span class="fu">:</span> <span class="dt">Natural</span>, index <span class="fu">:</span> <span class="dt">Natural</span>, text <span class="fu">:</span> <span class="dt">Text</span> }<br /> )<br /> <span class="ot">→</span> <span class="kw">let</span> fizzy <span class="fu">=</span> <span class="dt">Natural</span><span class="fu">/</span>isZero state<span class="fu">.</span>fizz<br /> <br /> <span class="kw">in</span> <span class="kw">let</span> buzzy <span class="fu">=</span> <span class="dt">Natural</span><span class="fu">/</span>isZero state<span class="fu">.</span>buzz<br /> <br /> <span class="kw">in</span> <span class="kw">let</span> line <span class="fu">=</span><br /> <span class="kw">if</span> fizzy <span class="fu">&&</span> buzzy <span class="kw">then</span> <span class="st">"FizzBuzz"</span><br /> <br /> <span class="kw">else</span> <span class="kw">if</span> fizzy <span class="fu">&&</span> not buzzy <span class="kw">then</span> <span class="st">"Fizz"</span><br /> <br /> <span class="kw">else</span> <span class="kw">if</span> not fizzy <span class="fu">&&</span> buzzy <span class="kw">then</span> <span class="st">"Buzz"</span><br /> <br /> <span class="kw">else</span> <span class="dt">Integer</span><span class="fu">/</span>show (<span class="dt">Natural</span><span class="fu">/</span>toInteger state<span class="fu">.</span>index)<br /> <br /> <span class="kw">in</span> { buzz <span class="fu">=</span> pred (<span class="kw">if</span> buzzy <span class="kw">then</span> <span class="fu">+</span><span class="dv">5</span> <span class="kw">else</span> state<span class="fu">.</span>buzz)<br /> , fizz <span class="fu">=</span> pred (<span class="kw">if</span> fizzy <span class="kw">then</span> <span class="fu">+</span><span class="dv">3</span> <span class="kw">else</span> state<span class="fu">.</span>fizz)<br /> , index <span class="fu">=</span> state<span class="fu">.</span>index <span class="fu">+</span> <span class="fu">+</span><span class="dv">1</span><br /> , text <span class="fu">=</span> state<span class="fu">.</span>text <span class="fu">++</span> line <span class="fu">++</span> <span class="st">"\n"</span><br /> }<br /><br /><span class="kw">in</span> <span class="kw">let</span> zero <span class="fu">=</span> { buzz <span class="fu">=</span> <span class="fu">+</span><span class="dv">5</span>, fizz <span class="fu">=</span> <span class="fu">+</span><span class="dv">3</span>, index <span class="fu">=</span> <span class="fu">+</span><span class="dv">0</span>, text <span class="fu">=</span> <span class="st">""</span> }<br /><br /><span class="kw">in</span> <span class="kw">let</span> fizzBuzz <span class="fu">=</span><br /> λ(n <span class="fu">:</span> <span class="dt">Natural</span>)<br /> <span class="ot">→</span> <span class="kw">let</span> result <span class="fu">=</span><br /> <span class="dt">Natural</span><span class="fu">/</span>fold<br /> n<br /> { buzz <span class="fu">:</span> <span class="dt">Natural</span><br /> , fizz <span class="fu">:</span> <span class="dt">Natural</span><br /> , index <span class="fu">:</span> <span class="dt">Natural</span><br /> , text <span class="fu">:</span> <span class="dt">Text</span><br /> }<br /> succ<br /> zero<br /> <br /> <span class="kw">in</span> result<span class="fu">.</span>text<br /><br /><span class="kw">in</span> fizzBuzz</code></pre></div><p>However, this program runs incredibly slowly, taking over 7 seconds just to compute 20 elements:</p><pre><code>bench 'dhall <<< "./fizzbuzz +20"'<br />benchmarking dhall <<< "./fizzbuzz +20"<br />time 7.450 s (7.194 s .. 7.962 s)<br /> 0.999 R² (NaN R² .. 1.000 R²)<br />mean 7.643 s (7.512 s .. 7.739 s)<br />std dev 145.0 ms (0.0 s .. 165.6 ms)<br />variance introduced by outliers: 19% (moderately inflated)</code></pre><p>However, if you use a strict fold then the program takes half a second to go through 10,000 elements:</p><pre><code>$ bench 'dhall <<< "./fizzbuzz +10000"'<br />benchmarking dhall <<< "./fizzbuzz +10000"<br />time 591.5 ms (567.3 ms .. NaN s)<br /> 1.000 R² (0.999 R² .. 1.000 R²)<br />mean 583.4 ms (574.0 ms .. 588.8 ms)<br />std dev 8.418 ms (0.0 s .. 9.301 ms)<br />variance introduced by outliers: 19% (moderately inflated)</code></pre><h2 id="conclusion">Conclusion</h2><p>Many people associate dynamic languages with interpreters, but Dhall is an example of a statically typed interpreter. Dhall's evaluator is not sophisticated at all but can still take advantage of static type information to achieve comparable performance with Python (which is a significantly more mature interpreter). This makes me wonder if the next generation of interpreters will be statically typed in order to enable better optimizations.</p></body></html>Wed, 27 Sep 2017 13:17:29 +0000noreply@blogger.com (Gabriel Gonzalez)Tweag I/O: Immutability and unboxing in array programminghttp://www.tweag.io/posts/2017-09-27-array-package.html
http://www.tweag.io/posts/2017-09-27-array-package.html
<div>Manuel M T Chakravarty</div><p><em>This is the third post in a series about array programming in Haskell — you might be interested in the <a href="http://www.tweag.io/posts/2017-08-09-array-programming-in-haskell.html">first</a> and <a href="http://www.tweag.io/posts/2017-08-31-hmatrix.html">second</a>, too.</em></p>
<p>In the <a href="http://www.tweag.io/posts/2017-08-31-hmatrix.html">previous post</a> of this series, we discussed commonly used vector and matrix routines, which are available in highly-optimised implementations in most programming languages. However, often we need to implement our own custom array algorithms. To this end, the <a href="https://www.haskell.org/onlinereport/haskell2010/">Haskell standard (Haskell 2010)</a> already comes with a simple <a href="https://www.haskell.org/onlinereport/haskell2010/haskellch14.html#x22-20100014">array API</a> in the form of the <code>Data.Array</code> standard module. These arrays are <em>immutable</em>, <em>boxed</em>, and <em>non-strict</em>. This allows for the elegant, high-level description of many array algorithms, but it is suboptimal for compute-intensive applications as boxing and non-strictness, especially in combination with the reliance on association lists for array construction, lead to code that is <a href="https://link.springer.com/chapter/10.1007/978-3-540-44833-4_2">very difficult for the Haskell compiler to optimise</a>, resulting in rather limited performance.</p>
<h2>Strictness</h2>
<p>The extra expressiveness granted by non-strictness is nicely displayed by the following example, courtesy of the classic <a href="https://www.haskell.org/tutorial/">A Gentle Introduction to Haskell</a>. The <code>wavefront</code> function defines an <em>n</em>x<em>n</em> array whose leftmost column and topmost row are populated with 1s (by the first two list comprehensions below). All other array elements are defined as the sum of the three elements to their left, top, and top-left (by the third list comprehension in the definition).</p>
<pre><code>wavefront :: Int -> Array (Int,Int) Int
wavefront n = arr
where
arr = array ((1,1),(n,n))
([((1,j), 1) | j <- [1..n]] ++
[((i,1), 1) | i <- [2..n]] ++
[((i,j), arr!(i,j-1) + arr!(i-1,j-1) + arr!(i-1,j))
| i <- [2..n], j <- [2..n]])
</code></pre>
<p>The <code>!</code> infix operator implements array indexing:</p>
<pre><code>(!) :: Ix i => Array i e -> i -> e
</code></pre>
<p>Moreover, the function <code>array</code> constructs an array from an association list mapping indexes to values:</p>
<pre><code>array :: Ix i => (i, i) -> [(i, e)] -> Array i e
</code></pre>
<p>The elegance of <code>wavefront</code> is in the recursive definition of the array <code>arr</code>. In the expression <code>arr!(i,j-1) + arr!(i-1,j-1) + arr!(i-1,j)</code>, we access the elements to the left, top, and top-left of the current one by appropriate indexing of the very array that we are currently in the process of defining. Such a recursive dependency is only valid for a non-strict data structure.</p>
<h2>Boxing</h2>
<p>Unfortunately, the expressiveness of non-strict arrays comes at a price, especially if the array elements are simple numbers. Instead of being able to store those numeric elements in-place in the array, non-strict arrays require a <em>boxed</em> representation, where the elements are pointers to heap objects containing the numeric values. This additional indirection requires extra memory and drastically reduces the efficiency of array access, especially in tight loops. The layout difference between an unboxed (left) and a boxed (right) representation is illustrated below.</p>
<p><img src="http://blog.tweag.io/../img/posts/unboxed-vs-boxed.png" alt="Boxed versus unboxed array representation" style="" title="Boxed versus unboxed array representation" /></p>
<p>While both strict and non-strict data structures admit boxed representations, non-strict structures typically require boxing. To provide an alternative to the standard non-strict arrays, the <a href="https://hackage.haskell.org/package/array"><code>array</code> package</a> provides strict, unboxed arrays of type <code>Data.Array.Unboxed.UArray i e</code>. By way of overloading via the type class <a href="https://hackage.haskell.org/package/array-0.5.2.0/docs/Data-Array-IArray.html#t:IArray"><code>Data.Array.IArray.IArray</code></a>, they provide the same API as the standard non-strict, boxed arrays. However, the element type is restricted to basic types that can be stored unboxed, such as integral and floating-point numeric types.</p>
<p>Unfortunately, array construction based on association lists, such as the <a href="https://hackage.haskell.org/package/array-0.5.2.0/docs/Data-Array-IArray.html#v:array"><code>array</code></a> function, still severely limits the performance of immutable <code>UArray</code>s. Nevertheless, at least, array access by way of <a href="https://hackage.haskell.org/package/array-0.5.2.0/docs/Data-Array-IArray.html#v:-33-"><code>(!)</code></a> is efficient for unboxed arrays.</p>
<h2>Immutability</h2>
<p>While immutable arrays —i.e., arrays that cannot directly be in-place updated— are semantically simpler, it turns out that indexed-based array construction is drastically more efficient for mutable arrays. Hence, computationally demanding Haskell array code typically adopts a two-phase array life cycle: (1) arrays are allocated as mutable arrays and initialised using in-place array update; once initialised, (2) they are <em>frozen</em> by making them immutable.</p>
<p>This usage pattern is supported by the interface provided by the <a href="https://hackage.haskell.org/package/array-0.5.2.0/docs/Data-Array-MArray.html#t:MArray"><code>Data.Array.MArray.MArray</code></a> class and we use it in the following example function <code>generate</code> to initialise an array of <code>Double</code>s with an index-based generator function <code>gen</code>:</p>
<pre><code>generate :: Ix i => (i, i) -> (i -> Double) -> UArray i Double
generate bnds gen
= runSTUArray $ do
arr <- newArray_ bnds
mapM_ (\i -> writeArray arr i (gen i)) (range bnds)
return arr
</code></pre>
<p>Mutable arrays come in various flavours that are, in particularly, distinguished by the monad in which the array operations take place. Usually, this is either <code>IO</code> or <code>ST</code>, and the <code>array</code> package provides both boxed and unboxed variants for both monads. We have the boxed <code>IOArray</code> and the unboxed <code>IOUArray</code> as well as the boxed <code>STArray</code> and the unboxed <code>STUArray</code>. The above definition of <code>generate</code> uses <code>STUArray</code> to initialise the array, and then, freezes it into a <code>UArray</code>, which is returned. The choice of <code>STUArray</code> is implicit in the use of <a href="https://hackage.haskell.org/package/array-0.5.2.0/docs/Data-Array-ST.html#v:runSTUArray"><code>runSTUArray</code></a>, which executes the code in the state transformer monad <code>ST</code> and freezes the <code>STUArray</code> into a <code>UArray</code>.</p>
<p>The function <code>newArray_</code> provides a fresh <em>uninitialised</em> array:</p>
<pre><code>newArray_ :: (MArray a e m, Ix i) => (i, i) -> m (a i e)
</code></pre>
<p>We can read and write a mutable array with</p>
<pre><code>readArray :: (MArray a e m, Ix i) => a i e -> i -> m e
writeArray :: (MArray a e m, Ix i) => a i e -> i -> e -> m ()
</code></pre>
<p>In the definition of <code>generate</code>, we use <code>writeArray</code> once on each index in the range <code>range bnds</code> of the mutable array to initialise the value at index <code>i</code> with the value of the generator function at that index, <code>gen i</code>.</p>
<p>Generally, we can freeze a mutable array, obtaining an immutable array, with</p>
<pre><code>freeze :: (Ix i, MArray a e m, IArray b e) => a i e -> m (b i e)
</code></pre>
<p>There is also the unsafe variant <code>unsafeFreeze</code> that avoids copying the array during freezing, but puts the onus on the programmer to ensure that the mutable argument is subsequently not updated anymore. In the code for <code>generate</code> above, we indirectly use <code>unsafeFreeze</code> by way of <code>runSTUArray</code>. As <code>runSTUArray</code> makes it impossible to use the mutable array after freezing, this encapsulated use of <code>unsafeFreeze</code> is always safe.</p>
<p>An expression, such as, <code>generate (1,100) ((^2) . fromIntegral)</code> produces an unboxed array with the first one hundred square numbers. Internally, it is based on the initialisation of a mutable array, but this is completely abstracted over by the definition of <code>generate</code>. While there is no inbuilt need to ever freeze a mutable array, good functional programming style requires to keep mutable code as localised as possible and to avoid passing mutable data structures around.</p>
<h2>Summary</h2>
<p>Well written code based on unboxed arrays and using the discussed pattern to create arrays by initialising a mutable version, which is subsequently frozen, can achieve performance comparable to low-level C code. In fact, the collection-oriented high-performance array frameworks that we will discuss in subsequent blog posts work exactly in this manner.</p>Wed, 27 Sep 2017 00:00:00 +0000Holden Karau: StrataConf NY 2017tag:blogger.com,1999:blog-23427281.post-5032943064865109998
http://feedproxy.google.com/~r/holdenkarau/iYtm/~3/1XUIuk31q0M/strataconf-ny-2017.html
<span style="">If you're in New York for Strata this week come join me on Wednesday for a book signing at 10:55 ( https://conferences.oreilly.com/strata/strata-ny/public/content/author-signings ), talk at 2:55 ( https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/60802 ), and office hours at 4:35 ( https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/63328 ) :)</span><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?a=1XUIuk31q0M:ZG9C3TwAaTw:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?d=yIl2AUoC8zA" border="0" /></a> <a href="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?a=1XUIuk31q0M:ZG9C3TwAaTw:63t7Ie-LG7Y"><img src="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?d=63t7Ie-LG7Y" border="0" /></a> <a href="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?a=1XUIuk31q0M:ZG9C3TwAaTw:V_sGLiPBpWU"><img src="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?i=1XUIuk31q0M:ZG9C3TwAaTw:V_sGLiPBpWU" border="0" /></a> <a href="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?a=1XUIuk31q0M:ZG9C3TwAaTw:7Q72WNTAKBA"><img src="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?d=7Q72WNTAKBA" border="0" /></a> <a href="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?a=1XUIuk31q0M:ZG9C3TwAaTw:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?d=qj6IDK7rITs" border="0" /></a> <a href="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?a=1XUIuk31q0M:ZG9C3TwAaTw:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/~ff/holdenkarau/iYtm?i=1XUIuk31q0M:ZG9C3TwAaTw:gIN9vFwOqvQ" border="0" /></a>
</div><img src="http://feeds.feedburner.com/~r/holdenkarau/iYtm/~4/1XUIuk31q0M" alt="" height="1" width="1" />Mon, 25 Sep 2017 19:35:26 +0000noreply@blogger.com (Holden Karau)Functional Jobs: Haskell or Scala engineer at Courex (Full-time)urn:uuid:aba8a1fe-3817-668e-7bf6-e56bbe0c139e
https://functionaljobs.com/jobs/9036-haskell-or-scala-engineer-at-courex
<h2>What we do</h2>
<p>Courex, a subsidiary of Keppel T&T, is an 8 year old ecommerce logistics company driven by technology. We help our customers manage their supply chain so they can focus on selling. We do the following</p>
<ul>
<li>last mile delivery</li>
<li>warehousing</li>
<li>omnichannel integration</li>
</ul>
<p>Our operations is driven by technology. Some interesting stuff</p>
<ul>
<li>We run a hybrid crowd-sourced(uber style) + fixed fleet model.</li>
<li>We built an automated parcel dimension measurement machine using Kinect</li>
<li>We have autonomous robots coming in late 2017 to pick and sort parcels</li>
</ul>
<p>Experience a different sort of scale. Not bits and bytes, but parcels, machines and people. Your work affects the real world in a huge traditional industry.</p>
<p>As part of the Keppel group, your work will reach the supply chain across Southeast Asia and China. Help us digitise the supply chain.</p>
<h2>What are we looking for</h2>
<p>We have openings in 2 teams. The inventory management product, which is written in Haskell, and the inventory synchronisation product which is written in Scala. Getting the inventory right is crucial in the supply chain, and functional programming gives us the confidence to do that.</p>
<p>The inventory management product manages how inventory flows in and out of the various warehouses in the region. Whereas the inventory synchronisation team synchronises the state of the inventory to various places such as Amazon, Lazada and Shopee. </p>
<p>We are looking for people interested in functional programming. It doesn't matter if you don't have working experience in them. Qualifications are not necessary. We like people who are practical and prolific. We are expanding to Southeast Asia. Our HQ is in Singapore, but you can work from one of Malaysia, Indonesia or Vietnam.</p>
<p>Get information on <a href="https://functionaljobs.com/jobs/9036-haskell-or-scala-engineer-at-courex">how to apply</a> for this position.</p>Mon, 25 Sep 2017 07:34:48 +0000Christopher Allen: Alternatives to Typed Holes for talking to your compilerhttp://bitemyapp.com//posts/2017-09-23-please-stop-using-typed-holes.html
http://bitemyapp.com//posts/2017-09-23-please-stop-using-typed-holes.html
<div class="info">
</div>
<div class="post">
<p>Rejected title: Type Praxis</p>
<p>I frequently see people recommend that others use typed holes. I think people are more apt to recommend typed holes than the alternatives because it’s a bespoke feature intended to enable discovering the type of a sub-expression more easily. Which is fair enough, except it doesn’t really have a good use-case! I will demonstrate in this post why.</p>
<p>I frequently find myself relying on GHC Haskell’s features in order to off-load brain effort. The idea behind typed holes is that if you have an incomplete expression and aren’t sure what type the remaining part should be, you can ask the compiler! Lets reuse the example from the Haskell Wiki: https://wiki.haskell.org/GHC/Typed_holes</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">pleaseShow ::</span> <span class="dt">Show</span> a <span class="ot">=></span> <span class="dt">Bool</span> <span class="ot">-></span> a <span class="ot">-></span> <span class="dt">Maybe</span> <span class="dt">String</span>
pleaseShow <span class="dt">False</span> _ <span class="fu">=</span> <span class="dt">Nothing</span>
pleaseShow <span class="dt">True</span> a <span class="fu">=</span> <span class="dt">Just</span> (show _a)</code></pre></div>
<p>The idea here is that we aren’t sure what should go at the end of the final line and we’re using <code>_a</code> to ask GHC what the type of <code>_a</code> should be. You get a type error that tries to describe the typed hole as follows:</p>
<pre><code> • Found hole: _a :: a0
Where: ‘a0’ is an ambiguous type variable
Or perhaps ‘_a’ is mis-spelled, or not in scope
• In the first argument of ‘show’, namely ‘_a’
In the first argument of ‘Just’, namely ‘(show _a)’
In the expression: Just (show _a)
• Relevant bindings include
a :: a
pleaseShow :: Bool -> a -> Maybe String</code></pre>
<p>Okay so here’s the problem. There’s a <code>Show</code> constraint on <code>a</code> but the typed hole message doesn’t bother saying so:</p>
<pre><code> • Found hole: _a :: a0</code></pre>
<p>This represents sort of a problem. Typeclass constraints aren’t always as syntactically obvious as they are from the declaration of <code>pleaseShow</code> here:</p>
<pre><code>pleaseShow :: Show a => Bool -> a -> Maybe String</code></pre>
<p>Sometimes they arise from other sub-expressions in your code and aren’t manifest in the type of your declaration!</p>
<p>You can’t productively point new people to typed holes because they’ll get extremely confused about type variables that have no constraints. If they’re reading good learning material, they’ll know that means they can’t actually do anything with something that is parametrically polymorphic. Even that framing aside, they just won’t know what terms are available to them for anything polymorphic.</p>
<p>Then we come to the expert. The expert is more likely to be working with code leveraging typeclasses and polymorphism and therefore…typed holes is of less help to them. If they’re aware of what typeclass constraints are attached to a type variable, fine, but the compiler is still forcing the programmer to juggle more context in their head than is really necessary.</p>
<h2 id="in-which-i-offer-a-better-alternative">In which I offer a better alternative</h2>
<pre><code>pleaseShow :: Show a => Bool -> a -> Maybe String
pleaseShow False _ = Nothing
pleaseShow True a =
let x :: z
x = a
in Just (show undefined)</code></pre>
<p>This time we get an error that mentions where the original type came from along with the relevant typeclass constraints:</p>
<pre><code> • Couldn't match expected type ‘z’ with actual type ‘a’
‘a’ is a rigid type variable bound by
the type signature for:
pleaseShow :: forall a. Show a => Bool -> a -> Maybe String
‘z’ is a rigid type variable bound by
the type signature for:
x :: forall z. z</code></pre>
<p>Keep in mind, this isn’t perfect! It’s not strictly the same as typed holes either as it’s more about contradicting the compiler about what type <code>a</code> had in order to discover what its type is. However, at least this way, we get a more complete picture of what the type of <code>a</code> is. Also note how I used undefined in order to ignore the parts of my code I wasn’t interested in getting errors about. This isn’t a perfect fit here as it results in GHC wanting to know which type it’s meant to expect from <code>undefined</code>, but in more typical circumstances, it works great for positing hypotheticals without bothering to write the actual code.</p>
<p>We’re about to do something more gnarly looking in the next section, the tl;dr is this:</p>
<h3 id="tldr">TL;DR</h3>
<p>Use let expressions, <code>undefined</code>, impossible types and the like instead of typed holes. And don’t recommend Typed Holes to new people, they’re more confusing than helpful and the facilities of typed holes don’t scale well to more complicated contexts anyway.</p>
<h2 id="tackling-slightly-more-complicated-situations">Tackling slightly more complicated situations</h2>
<hr />
<em>Warning: If you haven’t worked through about 2/3s of the <a href="http://haskellbook.com">Haskell Book</a> or possess the equivalent practice and knowledge, you are unlikely to grok this section.</em>
<hr />
<p>Sometimes you want to be able to posit something or lay down types for sub-expressions in a situation where you have a polymorphic type arising from a typeclass instance or function declaration. In those situations, knowing how to combine ScopedTypeVariables, InstanceSigs, and let expressions can be very valuable!</p>
<p>What if we’re stumped on something like this?</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">doubleBubble ::</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b)
doubleBubble f ffa <span class="fu">=</span>
undefined</code></pre></div>
<p>So we try to start by assigning a type to a sub-expression:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">doubleBubble ::</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b)
doubleBubble f ffa <span class="fu">=</span>
<span class="kw">let</span><span class="ot"> x ::</span> z
x <span class="fu">=</span> f
<span class="kw">in</span> undefined</code></pre></div>
<p>And get the following type error:</p>
<pre><code> • Couldn't match expected type ‘z’
with actual type ‘f1 (f2 (a -> b))’</code></pre>
<p>Fair enough, what if we try to make the types agree?</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">doubleBubble ::</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b)
doubleBubble f ffa <span class="fu">=</span>
<span class="kw">let</span><span class="ot"> x ::</span> f1 (f2 (a <span class="ot">-></span> b))
x <span class="fu">=</span> f
<span class="kw">in</span> undefined</code></pre></div>
<p>We get a type error?!</p>
<pre><code> • Couldn't match type ‘f1’ with ‘f4’
‘f1’ is a rigid type variable bound by
the type signature for:
doubleBubble :: forall (f1 :: * -> *) (f2 :: * -> *) a b.
(Applicative f1, Applicative f2) =>
f1 (f2 (a -> b)) -> f1 (f2 a) -> f1 (f2 b)
‘f4’ is a rigid type variable bound by
the type signature for:
x :: forall (f4 :: * -> *) (f5 :: * -> *) a1 b1. f4 (f5 (a1 -> b1))</code></pre>
<p>The issue is that types usually only last the scope of a single type signature denoted by <code>::</code>, so the variables <code>f1</code>, <code>a</code>, <code>b</code>, and the like can only be referenced in our declaration. That kinda sucks, how do we keep referring to the same type variables under our declaration? <code>ScopedTypeVariables</code>!</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">{-# LANGUAGE ScopedTypeVariables #-}</span>
<span class="ot">doubleBubble ::</span> forall f1 f2 a b
<span class="fu">.</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b)
doubleBubble f ffa <span class="fu">=</span>
<span class="kw">let</span><span class="ot"> x ::</span> f1 (f2 (a <span class="ot">-></span> b))
x <span class="fu">=</span> f
<span class="kw">in</span> undefined</code></pre></div>
<p>This now type-checks because we used <code>forall</code> to tell GHC that we wanted those variables to be lexically scoped! Now we’re really cooking with gas. Lets follow a chain of experiments and how they change our type errors:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">doubleBubble ::</span> forall f1 f2 a b
<span class="fu">.</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b)
doubleBubble f ffa <span class="fu">=</span>
<span class="kw">let</span><span class="ot"> x ::</span> z
x <span class="fu">=</span> fmap (<span class="fu"><*></span>) f
<span class="kw">in</span> undefined</code></pre></div>
<pre><code> • Couldn't match expected type ‘z’
with actual type ‘f1 (f2 a -> f2 b)’</code></pre>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">doubleBubble ::</span> forall f1 f2 a b
<span class="fu">.</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b)
doubleBubble f ffa <span class="fu">=</span>
<span class="kw">let</span><span class="ot"> x ::</span> z
x <span class="fu">=</span> (fmap (<span class="fu"><*></span>) f) <span class="fu"><*></span> ffa
<span class="kw">in</span> undefined</code></pre></div>
<pre><code> • Couldn't match expected type ‘z’ with actual type ‘f1 (f2 b)’</code></pre>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="co">-- this typechecks.</span>
<span class="ot">doubleBubble ::</span> forall f1 f2 a b
<span class="fu">.</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b) <span class="co">-- <---</span>
doubleBubble f ffa <span class="fu">=</span> <span class="co">------ hmm</span>
<span class="kw">let</span><span class="ot"> x ::</span> f1 (f2 b) <span class="co">-- <--------</span>
x <span class="fu">=</span> (fmap (<span class="fu"><*></span>) f) <span class="fu"><*></span> ffa
<span class="kw">in</span> undefined</code></pre></div>
<p>And now we’re done:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">doubleBubble ::</span> forall f1 f2 a b
<span class="fu">.</span> ( <span class="dt">Applicative</span> f1
, <span class="dt">Applicative</span> f2 )
<span class="ot">=></span> f1 (f2 (a <span class="ot">-></span> b))
<span class="ot">-></span> f1 (f2 a)
<span class="ot">-></span> f1 (f2 b) <span class="co">-- <---</span>
doubleBubble f ffa <span class="fu">=</span> <span class="co">------ hmm</span>
<span class="kw">let</span><span class="ot"> x ::</span> f1 (f2 b) <span class="co">-- <--------</span>
x <span class="fu">=</span> (fmap (<span class="fu"><*></span>) f) <span class="fu"><*></span> ffa
<span class="kw">in</span> x</code></pre></div>
<p>The intuition here is that we have to applicatively (monoidal functor, remember?) combine the <code>* -> *</code> kinded structure twice,</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"> f1 (f2 (a <span class="ot">-></span> b))
<span class="co">-- <> <></span>
<span class="ot">-></span> f1 (f2 a)</code></pre></div>
<p>Once for <code>f1</code> of the function and <code>f1</code> of the value, once for <code>f2</code> of the function and <code>f2</code> of the value.</p>
<pre><code>Prelude> :t (\f a -> f <*> a)
(\f a -> f <*> a) :: Applicative f => f (a -> b) -> f a -> f b
Prelude> :t (\f a -> (fmap (<*>) f) <*> a)
(\f a -> (fmap (<*>) f) <*> a)
:: (Applicative f, Applicative f1) =>
f1 (f (a -> b)) -> f1 (f a) -> f1 (f b)</code></pre>
<p>The following doesn’t fit because we end up triggering the <code>Reader</code> (function type) Applicative:</p>
<pre><code>Prelude> :t (\f a -> ((<*>) f) <*> a)
(\f a -> ((<*>) f) <*> a)
:: (a1 -> a -> b) -> ((a1 -> a) -> a1) -> (a1 -> a) -> b</code></pre>
<p>Rewriting the working solution a little:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">apply <span class="fu">=</span> (<span class="fu"><*></span>)
doubleAp f a <span class="fu">=</span> apply (fmap apply f) a</code></pre></div>
<pre><code>Prelude> let apply = (<*>)
Prelude> let doubleAp f a = apply (fmap apply f) a
Prelude> :t doubleAp
doubleAp
:: (Applicative f1, Applicative f) =>
f1 (f (a -> b)) -> f1 (f a) -> f1 (f b)</code></pre>
<p>Then breaking down:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">doubleAp f a <span class="fu">=</span> apply (fmap apply f) a
<span class="co">-- [1] [2] [3]</span></code></pre></div>
<ol style="">
<li>This <code>apply</code> grafts in the pre-lifted <code>apply</code>, cf.</li>
</ol>
<pre><code>Prelude> import Data.Void
Prelude> let v :: Void; v = undefined
Prelude> let doubleAp f a = v (fmap apply f) a
<interactive>:104:20: error:
• Couldn't match expected type ‘f1 (f a -> f b) -> t1 -> t’
with actual type ‘Void’</code></pre>
<ol start="2" style="">
<li>This <code>fmap</code> lifts a regular <code>apply</code> into a type that can graft together two values embedded in <code>f</code> such that the type is: <code>f a -> f b</code>, cf.</li>
</ol>
<pre><code>Prelude> let doubleAp f a = apply (v apply f) a
<interactive>:105:27: error:
• Couldn't match expected type ‘(f0 (a0 -> b0) -> f0 a0 -> f0 b0)
-> t -> f (a -> b)’
with actual type ‘Void’</code></pre>
<ol start="3" style="">
<li>This is the <code>apply</code> lifted by <code>fmap</code>, transformed from:</li>
</ol>
<p><code>(f0 (a0 -> b0)</code> into <code>f0 a0 -> f0 b0</code></p>
<p>(The void error here is less useful)</p>
<p>Kicking in the contradiction we get for <code>a</code> if we replace it with the <code>Void</code> typed <code>v</code> variable:</p>
<pre><code>Prelude> let doubleAp f a = apply (fmap apply f) v
<interactive>:107:41: error:
• Couldn't match expected type ‘f1 (f a)’ with actual type ‘Void’
• In the second argument of ‘apply’, namely ‘v’
In the expression: apply (fmap apply f) v</code></pre>
<p>Not bad eh? I find it’s better to teach people these techniques than to point them to typed holes, but reasonable minds disagree. Even when a learner is relatively early in the learning process, these techniques can be made approachable/digestible.</p>
<p>That’s all folks. Below is just a demonstration of the missing-constraint problem with an example from the Haskell Wiki.</p>
<h2 id="re-demonstration-of-the-missing-constraint-problem-using-the-haskell-wikis-example">Re-demonstration of the missing constraint problem using the Haskell Wiki’s example</h2>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">module</span> <span class="dt">FreeMonad</span> <span class="kw">where</span>
<span class="kw">data</span> <span class="dt">Free</span> f a
<span class="fu">=</span> <span class="dt">Pure</span> a
<span class="fu">|</span> <span class="dt">Free</span> (f (<span class="dt">Free</span> f a))
<span class="co">-- These are just to shut the compiler up, we</span>
<span class="co">-- are not concerned with these right now.</span>
<span class="kw">instance</span> <span class="dt">Functor</span> f <span class="ot">=></span> <span class="dt">Functor</span> (<span class="dt">Free</span> f) <span class="kw">where</span>
fmap <span class="fu">=</span> undefined
<span class="kw">instance</span> <span class="dt">Functor</span> f <span class="ot">=></span> <span class="dt">Applicative</span> (<span class="dt">Free</span> f) <span class="kw">where</span>
pure <span class="fu">=</span> undefined
(<span class="fu"><*></span>) <span class="fu">=</span> undefined
<span class="co">-- Okay, we do care about the Monad though.</span>
<span class="kw">instance</span> <span class="dt">Functor</span> f <span class="ot">=></span> <span class="dt">Monad</span> (<span class="dt">Free</span> f) <span class="kw">where</span>
return a <span class="fu">=</span> <span class="dt">Pure</span> a
<span class="dt">Pure</span> a <span class="fu">>>=</span> f <span class="fu">=</span> f a
<span class="dt">Free</span> f <span class="fu">>>=</span> g <span class="fu">=</span> <span class="dt">Free</span> _a</code></pre></div>
<pre><code>code/FreeMonad.hs:20:23: error:
• Found hole: _a :: f (Free f b)
Where: ‘f’ is a rigid type variable bound by
the instance declaration at code/FreeMonad.hs:17:10
‘b’ is a rigid type variable bound by
the type signature for:
(>>=) :: forall a b. Free f a -> (a -> Free f b) -> Free f b
at code/FreeMonad.hs:19:10
Or perhaps ‘_a’ is mis-spelled, or not in scope
• In the first argument of ‘Free’, namely ‘_a’
In the expression: Free _a
In an equation for ‘>>=’: (Free f) >>= g = Free _a
• Relevant bindings include
g :: a -> Free f b (bound at code/FreeMonad.hs:20:14)
f :: f (Free f a) (bound at code/FreeMonad.hs:20:8)
(>>=) :: Free f a -> (a -> Free f b) -> Free f b
(bound at code/FreeMonad.hs:19:3)
Failed, modules loaded: none.</code></pre>
<p>^^ Look ma, no <code>Functor</code>.</p>
<pre><code>_a :: f (Free f b)</code></pre>
<pre><code>instance Functor f => Monad (Free f) where</code></pre>
<p>Not consistently, but I’m more likely to get better type errors when I create contradictions manually via let expressions than I am using typed holes.</p>
</div>
<div class="blurb">
<p>
I know this site is a bit of a disaster zone, but if you like my writing or think you could learn something useful from me, please <a href="http://haskellbook.com/">take a look at the Haskell book</a> I've been writing. There's a free sample available too!
</p>
</div>
<div class="footer">
<p>
Posted on September 23, 2017
</p>
</div>Sat, 23 Sep 2017 00:00:00 +0000Tweag I/O: GHC compiler plugins in the wild:<br/> typing Javahttp://www.tweag.io/posts/2017-09-22-inline-java-ghc-plugin.html
http://www.tweag.io/posts/2017-09-22-inline-java-ghc-plugin.html
<div>Facundo Domínguez, Mathieu Boespflug</div><p><a href="http://blog.tweag.io/2017-09-15-inline-java-tutorial.html">Previously</a>, we discussed how to
<em>use</em> <a href="https://www.stackage.org/package/inline-java">inline-java</a> to call any Java function
from Haskell. The reverse is also possible, though that will be
a topic for a future post. In this post, we'll peek underneath the
hood to talk a little about <em>how</em> inline-java does its deed.</p>
<p>You might find it an interesting read for at least the following
reason: since the latest v0.7 release of inline-java, it's an example
use of a recent feature of GHC called compiler plugins. These allow
you to introspect and transform types and the abstract syntax tree
before handing them off to later stages of the compiler pipeline. We
use this to good effect in order to check that argument and return
types on the Java side line up with those on the Haskell side (and
<em>vice versa</em>).</p>
<h1>Calling Java</h1>
<p><code>inline-java</code> makes it possible to invoke code written in Java using
a Haskell language feature known
as
<a href="https://scholar.google.com/citations?view_op=view_citation&citation_for_view=jjWDm9wAAAAJ:2osOgNQ5qMEC">quasiquotes</a>.</p>
<pre><code class="language-Haskell">{-# LANGUAGE QuasiQuotes #-}
import Language.Java (withJVM)
import Language.Java.Inline
main :: IO ()
main = withJVM [] $ do
let x = 1.5 :: Double
y <- [java| { System.out.println($x);
return $x + 1;
} |]
print (y :: Double)
</code></pre>
<p>The function <code>withJVM</code> starts an instance of the Java Virtual Machine (JVM),
and the <code>java</code> quasiquotation executes the Java code passed to it as a block
of statements.</p>
<p>In this example, the Haskell value <code>x</code> of type <code>Double</code> is coerced into
a Java value of primitive type <code>double</code>, which is then used whenever the
antiquoted variable <code>$x</code> appears inside the quasiquotation.
When the quasiquotation finishes executing, the Java value resulting
from evaluating <code>$x + 1</code> is coerced back to a Haskell value of type
<code>Double</code>.</p>
<p>GHC doesn't parse or generate any Java. Neither does <code>inline-java</code>.
So how can this program possibly work? The answer is that <code>inline-java</code>
feeds the quasiquotation to the <code>javac</code> compiler, which generates some
bytecode that is stored in the object file of the module. At runtime,
<code>inline-java</code> arranges for the bytecode to be handed to the JVM using
the <a href="https://www.stackage.org/package/jni">jni</a> package.
Finally, <code>inline-java</code> makes use of the
<a href="https://www.stackage.org/package/jvm"><code>jvm</code></a> package
to have the bytecode executed.</p>
<h1>Type safety</h1>
<p>A notable characteristic of this approach is that we know at compile
time if types are correct. We know that Java won't return an object if
on the Haskell side we expect a double, because the Java side knows
it's on the hook for handing us a double. <code>javac</code> will raise a compile
time error if the Java code doesn't do that. Even if the Haskell side
expected an object, say of type <code>java.util.List</code>, the Java
quasiquotation can't return an object of type <code>java.lang.String</code>
either. And conversely for arguments, Java and Haskell need to agree
on the type of arguments, or a compile-time error ensues.</p>
<p>Given that no one compiler analyses both languages, how can
type-checking work across language boundaries? Fortunately, both
compilers can be put to cooperate on the task. First, GHC
infers the types of the antiquoted variables and the return type which
is expected of the quasiquotation. Then, these types are translated to
Java types. The translation is conducted by a machinery of type classes
living in the <a href="https://github.com/tweag/inline-java/jvm">jvm</a> package.
The details of this process are not important at this point. What
matters is that it enables us to translate types across languages. For
instance,</p>
<table>
<thead>
<tr>
<th style="text-align: left;">Haskell type</th>
<th style="text-align: right;">Java type</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">Double</td>
<td style="text-align: right;">double</td>
</tr>
<tr>
<td style="text-align: left;">[Double]</td>
<td style="text-align: right;">double[]</td>
</tr>
<tr>
<td style="text-align: left;">ByteString</td>
<td style="text-align: right;">byte[]</td>
</tr>
<tr>
<td style="text-align: left;">Text</td>
<td style="text-align: right;">java.lang.String</td>
</tr>
</tbody>
</table>
<p>The translated types are passed to <code>javac</code> together with the rest of the
quasiquoted Java code. In our running example this would be</p>
<pre><code class="language-Java">double fresh_name(double $x) {
System.out.println($x);
return $x + 1;
}
</code></pre>
<p>Finally, the <code>javac</code> compiler type checks the quasiquotation. Type
mismatches will be discovered and reported at this stage.</p>
<p>It turns out that the first step is by far the most intricate.
Specifically, for inline-java to query the types that GHC inferred for
the antiquoted variables, and also query the type of the entire
quasiquotation.</p>
<h1>Looking for the types</h1>
<p>At first, it appears as if determining these types is trivial. There is
a Template Haskell primitive called
<a href="https://www.stackage.org/haddock/lts-9.0/template-haskell-2.11.1.0/Language-Haskell-TH.html#v:reify">reify</a>.</p>
<pre><code class="language-Haskell">reify :: Name -> Q Info
data Info =
...
| VarI Name Type (Maybe Dec)
...
</code></pre>
<p>Given an antiquoted variable <code>$x</code>, we ought to be able to use <code>reify 'x</code>
to determine its Haskell type. Well, except that this doesn't quite
work, because type checking is not finished when <code>reify</code> gets evaluated. From
there, we went down a rabbit hole of trying to propose patches to
Template Haskell to reliably get our hands on the inferred types.
If you want to follow the intricacies of our journey, here are the
related GHC issues for your amusement:
<a href="https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Reify">initial discussion</a>,
<a href="https://ghc.haskell.org/trac/ghc/ticket/12777">12777</a>,
<a href="https://ghc.haskell.org/trac/ghc/ticket/12778">12778</a>,
<a href="https://ghc.haskell.org/trac/ghc/ticket/13608">13608</a>.</p>
<p>After many discussions with Simon Peyton Jones, and some deal of
creative hacking, we could kind of get the inferred types for antiquoted
variables, but only for as long as the java quasiquotation didn't appear
inside Template Haskell brackets (<code>[| ... |]</code>). Moreover, we made no
progress getting the expected type of the quasiquotation.
Every idea we came up with required difficult compromises in the design.
In the meantime, we had to choose between checking the type of the
values returned by quasiquotations at runtime or using unsafe coercions,
neither of which is an attractive option.</p>
<p>Eventually, we learnt that Template Haskell was not the only way to
query the output of the type checker.</p>
<h1>Enter GHC Core plugins</h1>
<p>The GHC compiler uses an explicitly typed intermediate language known as
Core. All type applications of terms in Core are explicit, making it
possible to learn the types inferred at the type checking phase
by inspecting Core terms.
In order to get our hands on Core terms, we can use
<a href="https://downloads.haskell.org/~ghc/8.0.2/docs/html/users_guide/extending_ghc.html#core-plugins-in-more-detail">Core plugins</a>.
We could think of a Core plugin as a set of Core-to-Core passes that we
can
ask GHC to add to the compilation pipeline. The passes can be inserted
anywhere in the Core pipeline, and in particular, they can be inserted
right after desugaring, the phase which generates Core from the abstract
syntax tree of a Haskell program.</p>
<p>Quasiquotations disappear from the abstract syntax tree when Template
Haskell is executed. This happens well before the plugin passes.
In order to enable the plugin to find the location of the
quasiquotations, the quasiquoter can insert some artificial
function call as a beacon or marker. In <code>inline-java</code>, our example program
looks something as follows after Template Haskell runs.</p>
<pre><code class="language-Haskell">main :: IO ()
main = withJVM [] $ do
let x = 1.5 :: Double
y <- qqMarker
"{ System.out.println($x); return $x + 1; }"
x
print (y :: Double)
qqMarker :: forall args r. String -> args -> IO r
qqMarker = error "inline-java: The Plugin is not enabled."
</code></pre>
<p>The GHC Plugin is supposed to replace the call to <code>qqMarker</code> with an
appropriate call to the generated Java method. The all-important
point, however, is that the calls to <code>qqMarker</code> are annotated with the
types we want to determine in Core.</p>
<pre><code class="language-Haskell">main :: IO ()
main = ...
qqMarker
@ Double
@ Double
"{ System.out.println($x); return $x + 1; }"
...
</code></pre>
<p>The type parameters provide us with the type of the antiquoted
variable and the expected type of the quasiquotation. From here, the
plugin has all the information it needs to generate the Java code to
feed to <code>javac</code>. In addition, the plugin can inject the generated bytecode
in the object file of the module, and it arranges for this bytecode to
be located at runtime so it can be loaded in the JVM.</p>
<p>Now the user needs to remember to tell GHC to use the plugin by passing it
the option <code>-fplugin=Language.Java.Inline.Plugin</code>. But this is only until
Template Haskell learns
<a href="https://phabricator.haskell.org/D3821">the ability to tell GHC which plugins to use</a>.</p>
<h1>Summary</h1>
<p>By using a GHC plugin, we have simplified <code>inline-java</code> from a
complicated spaghetti which sprung from attempting to use
Template Haskell's <code>reify</code> and didn't fully addressed the type lookup
problem in a robust way. Now we have a straight forward story which
starts by
introducing the <code>qqMarker</code> beacons, attaches the Java bytecode in the
plugin phase and ends by loading it at runtime into the JVM.</p>
<p>Writing a compiler plugin is similar to writing Template Haskell code.
Both approaches need to manipulate abstract syntax trees. The plugin
approach can be regarded as more coupled with a particular version of
the compiler, since it relies on the internal Core language. However,
Core changes relatively little over the years, and anyway, a pass that
looks for some markers is hardly going to change a lot even if Core
did change.</p>
<p>Many thanks to Simon Peyton Jones for his patience to walk with us over
our attempts to <em>fix</em> Template Haskell. Without this dialog with the
compiler implementors, it would have been difficult for us to explore as
much of the design space as we needed to.</p>Fri, 22 Sep 2017 00:00:00 +0000Brent Yorgey: New baby, and Haskell Alphabethttp://byorgey.wordpress.com/?p=2063
https://byorgey.wordpress.com/2017/09/21/new-baby-and-haskell-alphabet/
<p>My wife and I just had a baby!</p>
<div style="text-align: center;">
<div class="figure">
<img src="https://byorgey.files.wordpress.com/2017/09/2017-09-1809-24-57.jpg?w=640" /><p></p>
</div>
</div>
<p>If you missed seeing me at ICFP, this is why.</p>
<p>In honor of my son’s birth (he will need to learn the alphabet and Haskell soon)—and at the instigation of <a href="http://very.science/">Kenny Foner</a>—I revived the <a href="http://ozark.hendrix.edu/~yorgey/haskell-alphabet/">Haskell Alphabet</a> by converting it to modern Hakyll and updating some of the broken or outdated links. Some of it is a bit outdated (I wrote it seven years ago), but it’s still a fun little piece of Haskell history. Enjoy!</p><br /> <a href="http://feeds.wordpress.com/1.0/gocomments/byorgey.wordpress.com/2063/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/byorgey.wordpress.com/2063/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=byorgey.wordpress.com&blog=1152889&post=2063&subd=byorgey&ref=&feed=1" alt="" height="1" border="0" width="1" />Thu, 21 Sep 2017 18:46:46 +0000Neil Mitchell: Shake 0.16 - revised rule definitionstag:blogger.com,1999:blog-7094652.post-7788926387030631401
http://neilmitchell.blogspot.com/2017/09/shake-016-revised-rule-definitions.html
<p><em>Summary: I've just released shake v0.16. A lot has changed, but it's probably only visible if you have defined your own rules or oracles.</em></p><p>Shake-0.16 is now out, 8 months since the last release, and with a lot of improvements. For full details read the <a href="https://github.com/ndmitchell/shake/blob/master/CHANGES.txt">changelog</a>, but in this post I'm going to go through a few of the things that might have the biggest impact on users.</p><p><strong>Rule types redefined</strong></p><p>Since the first version of Shake there has been a <code>Rule key value</code> type class defining all rule types - for instance the file rule type has <code>key</code> of filename and <code>value</code> of modification time. With version 0.16 the type class is gone, rules are harder to write, but offer higher performance and more customisation. For people using the builtin rule types, you'll see those advantages, and in the future see additional features that weren't previously possible. For people defining custom rule types, those will require rewriting - <a href="https://hackage.haskell.org/package/shake-0.16/docs/Development-Shake-Rule.html">read the docs</a> and if things get tough, ask <a href="http://stackoverflow.com/questions/tagged/shake-build-system">on StackOverflow</a>.</p><p>The one place many users might encounter the changes are that oracle rules now require a <code>type instance</code> defining between the <code>key</code> and <code>value</code> types. For example, if defining an oracle for the <code>CompilerVersion</code> given the <code>CompilerName</code>, you would have to add:</p><pre><code>type instance RuleResult CompilerName = CompilerVersion<br /></code></pre><p>As a result of this <code>type instance</code> the previously problematic <code>askOracle</code> can now infer the result type, removing possible sources of error and simplifying callers.</p><p>The redefining of rule types represents most of the work in this release.</p><p><strong>Add <code>cmd_</code></strong></p><p>The <code>cmd_</code> function is not much code, but I suspect will turn out to be remarkably useful. The <code>cmd</code> function in Shake is variadic (can take multiple arguments) and polymorphic in the return type (you can run it in multiple monads with multiple results). However, because of the overloading, if you didn't use the result of <code>cmd</code> it couldn't be resolved, leading to ugly code such as <code>() <- cmd args</code>. With <code>cmd_</code> the result is constrained to be <code>m ()</code>, so <code>cmd_ args</code> can be used.</p><p><strong>Rework Skip/Rebuild</strong></p><p>Since the beginning Shake has tried to mirror the <code>make</code> command line flags. In terms of flags to selectively control rebuilding, <code>make</code> is based entirely on ordered comparison of timestamps, and flags such as <code>--assume-new</code> don't make a lot of sense for Shake. In this release Shake stops trying to pretend to be <code>make</code>, removing the old flags (that never worked properly) and adding <code>--skip</code> (don't build something even if it is otherwise required) and <code>--build</code> (build something regardless). Both these flags can take <a href="https://hackage.haskell.org/package/shake/docs/Development-Shake.html#t:FilePattern">file patterns</a>, e.g, <code>--build=**/*.o</code> to rebuild all object files. I don't think these flags are finished with, but it's certainly less of a mess than before.</p>Wed, 20 Sep 2017 19:53:00 +0000noreply@blogger.com (Neil Mitchell)Mark Jason Dominus: Gompertz' law for wooden utility polestag:,2017:/tech/gompertz
https://blog.plover.com/tech/gompertz.html
<p><a href="https://en.wikipedia.org/wiki/Gompertz%E2%80%93Makeham_law_of_mortality">Gompertz'
law</a>
says that the human death rate increases exponentially with age. That
is, if your chance of dying during this year is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24x%24" />, then your
chance of dying during next year is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24cx%24" /> for some constant <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%3e1%24" />.
The death rate doubles every 8 years, so the constant <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%24" /> is
empirically around <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%242%5e%7b1%2f8%7d%20%5capprox%201%2e09%24" />. This is of course
mathematically incoherent, since it predicts that sufficiently old
people will have a mortality rate greater than 100%. But a number of
things are both true and mathematically incoherent, and this is one of
them. (<a href="https://en.wikipedia.org/wiki/Zipf%27s_law">Zipf's law</a> is another.)</p>
<p>The Gravity and Levity blog has
<a href="https://gravityandlevity.wordpress.com/2009/07/08/your-body-wasnt-built-to-last-a-lesson-from-human-mortality-rates/">a superb article about this</a>
from 2009 that reasons backwards from Gompertz' law to rule out
certain theories of mortality, such as the theory that death is due to
the random whims of a fickle god. (If death were entirely random, and
if you had a 50% chance of making it to age 70, then you would have a
25% chance of living to 140, and a 12.5% chance of living to 210,
which we know is not the case.)</p>
<p>Gravity and Levity says:</p>
<blockquote>
<p>Surprisingly enough, the Gompertz law holds across a large number of
countries, time periods, and even different species.</p>
</blockquote>
<p>To this list I will add wooden utility poles.</p>
<p>A couple of weeks ago Toph asked me why there were so many old rusty
staples embedded in the utility poles near our house, and this is easy
to explain: people staple up their yard sale posters and lost-cat
flyers, and then the posters and flyers go away and leave behind the
staples. (I once went out with a pliers and extracted a few dozen
staples from one pole; it was very satisfying but ultimately
ineffective.) If new flyer is stapled up each week, that is 52
staples per year, and 1040 in twenty years. If we agree that 20 years
is the absolute minimum plausible lifetime of a pole, we should not be
surprised if typical poles have hundreds or thousands of staples each.</p>
<p>But this morning I got to wondering what <em>is</em> the expected lifetime of
a wooden utility pole? I guessed it was probably in the range of 40
to 70 years. And happily, because of the Wonders of the Internet, I
could look it up right then and there, on the way to the trolley stop,
and spend my commute time reading about it.</p>
<p>It was not hard to find <a href="http://quanta-technology.com/sites/default/files/doc-files/Wood%20Pole%20Management%20Practices%20-%202012.pdf">an authoritative sounding and widely-cited
2012
study</a>
by electric utility consultants <a href="http://quanta-technology.com/">Quanta
Technology</a>.</p>
<p><strong>Summary</strong>: Most poles die because of fungal rot, so pole lifetime
varies widely depending on the local climate. An unmaintained pole
will last 50–60 years in a cold or dry climate and 30-40 years in a
hot wet climate. Well-maintained poles will last around twice as
long.</p>
<p>Anyway, Gompertz' law holds for wooden utility poles also. According
to the study:</p>
<blockquote>
<p>Failure and breakdown rates for wood poles are thought to increase
exponentially with deterioration and advancing time in service.</p>
</blockquote>
<p>The Quanta study presents this chart, taken from the (then
forthcoming) 2012 book <a href="https://www.crcpress.com/Aging-Power-Delivery-Infrastructures-Second-Edition/Willis-Schrieber/p/book/9781138072985"><em>Aging Power Delivery
Infrastructures</em></a>:</p>
<p align="center"><img src="https://pic.blog.plover.com/tech/gompertz/gompertz-poles.png" /></p>
<p>The solid line is the pole failure rate for a particular unnamed
utility company in a median climate. The failure rate with increasing
age clearly increases exponentially, as Gompertz' law dictates,
doubling every 12½ years or so: Around 1 in 200 poles fails at age 50,
around 1 in 100 of the remaining poles fails at age 62.5, and around 1
in 50 of the remaining poles fails at age 75.</p>
<p>(The dashed and dotted lines represent poles that are removed from
service for other reasons.)</p>
<p>From Gompertz' law itself and a minimum of data, we can extrapolate
the maximum human lifespan. The death rate for 65-year-old women is
around 1%, and since it doubles every 8 years or so, we find that 50%
of women are dead by age 88, and all but the most outlying outliers
are dead by age 120. And indeed, the human longevity record is
currently attributed to Jeanne Calment, who died in 1997 at the age of
122½.</p>
<p>Similarly we can extrapolate the maximum service time for a wooden
utility pole. Half of them make it to 90 years, but if you have a
large installed base of 110-year-old poles you will be replacing about
one-seventh of them every year and it might make more sense to rip
them all out at once and start over. At a rate of one yard sale per
week, a 110-year-old pole will have accumulated 5,720 staples.</p>
<p>The Quanta study does not address deterioration of utility poles due
to the accumulation of rusty staples.</p>Wed, 20 Sep 2017 18:41:00 +0000mjd@plover.com (Mark Dominus)Dominic Orchard: Scrap Your Reprinterhttp://dorchard.wordpress.com/?p=381
https://dorchard.wordpress.com/2017/09/20/scrap-your-reprinter/
<p>Back in 2013, <a href="http://www.cl.cam.ac.uk/~acr31/">Andrew Rice</a> and I were doing some initial groundwork on how to build tools to help scientists write better code (e.g., with the help of refactoring tools and verification tools). We talked to a lot of scientists who wrote Fortran almost exclusively, so we started creating infrastructure for building tools to work on Fortran. This was the kernel of the <a href="http://camfort.github.io">CamFort</a> project for which we got an EPSRC grant in 2015 (which is ongoing). The CamFort tool now has a couple of fairly well developed specification/verification features, and a few refactoring features. Early on, I started building everything in Haskell using the brilliant <a href="http://hackage.haskell.org/package/uniplate">uniplate</a> library, based on the Scrap Your Boilerplate <a href="https://dorchard.wordpress.com/2017/09/20/scrap-your-reprinter/#syr1">[1]</a> work. This helped us to get the tool off the ground quickly by utilising the power of datatype generic programming. Fairly quickly we hit upon an interesting problem with building refactoring tools: how do you output source code for a refactored AST whilst preserving all the original comments and white space? It is not enough just to pretty print the AST, unless your AST contains <b>all</b> the comments and layout information. Building a parser to capture all this information is extremely hard, and we use a parser generator which limits flexibility (but is really useful for a large grammar). Another approach is to output patch/edit information for the original source code, calculated from the AST.</p>
<p>In the end, I came up with a datatype generic algorithm which I call the <em>reprinter. </em>The reprinter takes the original source code and an updated AST (which contains location information) and maps them into a new piece of source code. Here is an illustration which I’ll briefly explain:</p>
<p><img src="https://dorchard.files.wordpress.com/2017/09/reprinter.png?w=1100" alt="reprinter" class="alignnone size-full wp-image-835" /></p>
<p>Some source text (arithmetic code in prefix notation here) is parsed into an AST. The AST contains the “spans” of each syntactic fragment: the start position and end position in the original source code (for simplicity in this illustration, just the column number is represented). Some transformation/refactoring is applied next. In this case, the transformation rewrites redundant additions of 0, which happens in the node coming from source locations 10 to 16. The refactored node is marked in red. The reprinting then runs, stitching together the original source code with the updated source tree. A pretty printer is used to generate code for any new nodes, but all the original source text for the other nodes is preserved. The cool thing about this algorithm is that it is datatype generic: it works for any datatype, with some modest side conditions about storing source spans. The implementation uses the Scrap Your Zipper <a href="https://dorchard.wordpress.com/2017/09/20/scrap-your-reprinter/#syr2">[2]</a> library to do a context-dependent generic traversal of a datatype. In essence, the algorithm is similar to what one might do if you were to spit out edit information from an AST, then apply this to a piece of source text. But, the algorithm does this generically, and in a single simultaneous pass of the AST and the input source text.</p>
<p>I’ve always thought it was a cute and useful algorithm, which combined some cool techniques from functional programming. As with all the “Scrap Your X” libraries it saves huge amounts of time and messing around, especially when your AST representation keeps changing (which it did/does for us). The algorithm is really useful anywhere you need to update human-written source code in a layout-preserving way; for example, IDEs and refactoring tools but also in interactive theorem provers and program synthesis tools, where you need to synthesise source text into some existing human-written code. This is one of the ways it is used in CamFort, where specifications are synthesised from code analysis data and then inserted as comments into user code.</p>
<p>This summer, I was fortunate enough to have the resources to hire several interns. One of the interns, <a href="https://github.com/harry-clarke">Harry Clarke</a>, (amongst other things) worked with me to tidy up the code for the reprinter, add some better interfaces, make it usable <a href="http://hackage.haskell.org/package/reprinter">as a library for others</a>, and write it all up. He presented the work at <a href="http://iflconference.org/cfp.html">IFL 2017</a> and <a href="https://www.cs.kent.ac.uk/people/staff/dao7/publ/reprinter2017.pdf">the pre-proceedings version of the paper is available</a>. We are working on a post-proceedings version for December, so any comments gratefully appreciated.</p>
<hr />
<p><a name="syr1"></a>[1] Lämmel, Ralf, and Simon Peyton Jones. Scrap your boilerplate: a practical design pattern for generic programming. Vol. 38. No. 3. ACM, 2003.</p>
<p><a name="syr2"></a>[2] Adams, Michael D. “Scrap your zippers: a generic zipper for heterogeneous types.” <i>Proceedings of the 6th ACM SIGPLAN workshop on Generic programming</i>. ACM, 2010.</p><br /> <a href="http://feeds.wordpress.com/1.0/gocomments/dorchard.wordpress.com/381/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/dorchard.wordpress.com/381/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=dorchard.wordpress.com&blog=10180011&post=381&subd=dorchard&ref=&feed=1" alt="" height="1" border="0" width="1" />Wed, 20 Sep 2017 15:20:57 +0000Wolfgang Jeltsch: Registration for Haskell in Leipzig 2017 is openhttp://jeltsch.wordpress.com/?p=689
https://jeltsch.wordpress.com/2017/09/20/registration-for-haskell-in-leipzig-2017-is-open/
<p><a href="https://hal2017.softbase.org/" title="HaL 2017">Haskell in Leipzig 2017</a> opened its gates for everyone interested in Haskell and generally functional programming. Expect a great day of talks, tutorials, and a performance with a focus on FRP, followed by a Hackathon. <a href="http://nfa.imn.htwk-leipzig.de/HAL2017/" title="HaL 2017 practical info">Register</a> early and get your ticket at a reduced rate. Looking forward to meeting you in Leipzig.<span id="more-689"></span></p>
<h2 id="about">About</h2>
<p>Haskell is a modern functional programming language that allows rapid development of robust and correct software. It is renowned for its expressive type system, its unique approaches to concurrency and parallelism, and its excellent refactoring capabilities. Haskell is both the playing field of cutting-edge programming language research and a reliable base for commercial software development.</p>
<p>The workshop series Haskell in Leipzig (HaL), now in its 12th year, brings together Haskell developers, Haskell researchers, Haskell enthusiasts, and Haskell beginners to listen to talks, take part in tutorials, join in interesting conversations, and hack together. To support the latter, HaL will include a one-day hackathon this year. The workshop will have a focus on functional reactive programming (FRP) this time, while continuing to be open to all aspects of Haskell. As in the previous year, the workshop will be in English.</p>
<h2 id="invited-speaker">Invited Speaker</h2>
<ul>
<li><a href="http://www.cs.nott.ac.uk/~psxip1/" title="Ivan Perez">Ivan Perez</a>, University of Nottingham, UK</li>
</ul>
<h2 id="invited-performer">Invited Performer</h2>
<ul>
<li><a href="https://github.com/lennart" title="Lennart Melzer">Lennart Melzer</a>, Robert-Schumann-Hochschule Düsseldorf, Germany</li>
</ul>
<h2 id="registration">Registration</h2>
<p>Registration information is available on <a href="http://nfa.imn.htwk-leipzig.de/HAL2017/" title="HaL 2017 practical info">the web page of the local organizers</a>.</p>
<h2 id="program-committee">Program Committee</h2>
<ul>
<li>Edward Amsden, Plow Technologies, USA</li>
<li>Heinrich Apfelmus, Germany</li>
<li>Jurriaan Hage, Utrecht University, The Netherlands</li>
<li>Petra Hofstedt, BTU Cottbus-Senftenberg, Germany</li>
<li>Wolfgang Jeltsch, Tallinn University of Technology, Estonia (chair)</li>
<li>Andres Löh, Well-Typed LLP, Germany</li>
<li>Keiko Nakata, SAP SE, Germany</li>
<li>Henrik Nilsson, University of Nottingham, UK</li>
<li>Ertuğrul Söylemez, Intelego GmbH, Germany</li>
<li>Henning Thielemann, Germany</li>
<li>Niki Vazou, University of Maryland, USA</li>
<li>Johannes Waldmann, HTWK Leipzig, Germany</li>
</ul><br /> Tagged: <a href="https://jeltsch.wordpress.com/tag/conference/">conference</a>, <a href="https://jeltsch.wordpress.com/tag/frp/">FRP</a>, <a href="https://jeltsch.wordpress.com/tag/functional-programming/">functional programming</a>, <a href="https://jeltsch.wordpress.com/tag/haskell/">Haskell</a> <a href="http://feeds.wordpress.com/1.0/gocomments/jeltsch.wordpress.com/689/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/jeltsch.wordpress.com/689/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=jeltsch.wordpress.com&blog=33918393&post=689&subd=jeltsch&ref=&feed=1" alt="" height="1" border="0" width="1" />Wed, 20 Sep 2017 14:56:13 +0000Douglas M. Auclair (geophf): August 2017 1HaskellADay 1Liners Problems and Solutionstag:blogger.com,1999:blog-4650294074444534066.post-6678497687175036583
http://logicaltypes.blogspot.com/2017/09/august-2017-1haskelladay-1liners.html
<ul style="line-height: 18px; margin: 0.5em 0px; padding: 0px 2.5em;"><li>August 1st, 2017:</li>f :: (Maybe a, b) -> Maybe (a, b) Define points-free. <li>August 1st, 2017:<br />Given f above and f a and f b are mutually exclusive in Maybe monad, define<br />g :: Maybe (a, b) -> Maybe (a, b) -> (Maybe a, b)<br />points free</li><li>August 1st, 2017:<br />Now, let's define the dual of f<br />f' :: Maybe (a, b) -> (Maybe a, b)<br />points free</li></ul>Tue, 19 Sep 2017 15:47:21 +0000noreply@blogger.com (geophf)Well-Typed.Com: Visualizing lazy evaluationhttp://www.well-typed.com/blog/2017/09/visualize-cbn
http://www.well-typed.com/blog/2017/09/visualize-cbn
<script>
function addScript(filename)
{
var scriptBlock=document.createElement('script')
scriptBlock.setAttribute("type","text/javascript")
scriptBlock.setAttribute("src", filename)
document.getElementsByTagName("head")[0].appendChild(scriptBlock)
}
addScript("../../../aux/files/foldrl-animated/length-naive.js");
addScript("../../../aux/files/foldrl-animated/length-acc.js");
addScript("../../../aux/files/foldrl-animated/length-acc-strict.js");
addScript("../../../aux/files/foldrl-animated/mapM_Maybe.js");
</script>
<p>Haskell and other call-by-need languages use <em>lazy evaluation</em> as their default evaluation strategy. For beginners and advanced programmers alike this can sometimes be confusing. At Well-Typed a core part of our business is teaching Haskell, which we do through public courses (such as the upcoming Skills Matter courses <a href="https://skillsmatter.com/courses/464-well-typed-fast-track-to-haskell">Fast Track to Haskell</a>, <a href="https://skillsmatter.com/courses/542-well-typed-s-guide-to-haskell-performance-and-optimization">Haskell Performance and Optimization</a> and <a href="https://skillsmatter.com/courses/504-well-typed-s-guide-to-the-haskell-type-system">The Haskell Type System</a>), private in-house courses targeted at specific client needs, and of course through writing blog posts.</p>
<p>In order to help us design these courses, we developed a tool called <code>visualize-cbn</code>. It is a simple interpreter for a mini Haskell-like language which outputs the state of the program at every step in a human readable format. It can also generate a HTML/JavaScript version with “Previous” and “Next” buttons to step through a program. We released the tool as open source <a href="https://github.com/well-typed/visualize-cbn">to github</a> and <a href="http://hackage.haskell.org/package/visualize-cbn">Hackage</a>, in the hope that it will be useful to others.</p>
<p>The <a href="https://github.com/well-typed/visualize-cbn/blob/master/README.md"><code>README.md</code></a> file in the repo explains how to run the tool. In this blog post we will illustrate how one might take advantage of it. We will revisit the infamous triple of functions <code>foldr</code>, <code>foldl</code>, <code>foldl'</code>, and show how they behave. As a slightly more advanced example, we will also study the memory behaviour of <code>mapM</code> in the <code>Maybe</code> monad. Hopefully, this show-rather-than-tell blog post might help some people understand these functions better.</p>
<p>Throughout this section we will use this definition of <code>enumFromTo</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">enumFromTo n m <span class="fu">=</span>
<span class="kw">if</span> n <span class="fu"><=</span> m <span class="kw">then</span> n <span class="fu">:</span> enumFromTo (n <span class="fu">+</span> <span class="dv">1</span>) m
<span class="kw">else</span> []</code></pre></div>
<p>so that, say, <code>[1..3]</code> corresponds to <code>(enumFromTo 1 3)</code>.</p>
<h2 id="foldrfoldrlfoldl"><code>foldr</code>/<code>foldrl</code>/<code>foldl'</code></h2>
<p>In this section we will examine the difference between these three functions. We will not study these functions directly, however, but study a slightly simpler variant in the form of three definitions of <code>length</code> on lists. For a more in-depth discussion of this triple of functions, see <a href="http://www.well-typed.com/blog/2014/04/fixing-foldl/">our earlier blog post on this topic</a>.</p>
<h3 id="foldr"><code>foldr</code></h3>
<p>Consider the naive definition of <code>length</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">length xs <span class="fu">=</span>
<span class="kw">case</span> xs <span class="kw">of</span>
[] <span class="ot">-></span> <span class="dv">0</span>
(x<span class="fu">:</span>xs') <span class="ot">-></span> <span class="dv">1</span> <span class="fu">+</span> length xs'</code></pre></div>
<p>This corresponds to defining <code>length = foldr (\x n -> 1 + n) 0</code>.</p>
<p>Let’s consider what happens when we compute <code>length [1..3]</code>; you can click on <em>Prev</em> and <em>Next</em> to step through the execution:</p>
<blockquote>
<p><a style="cursor: pointer;">Prev</a> <a style="cursor: pointer;">Next</a> (step <span id="lengthNaive_step">Step</span>, <span id="lengthNaive_status">Status</span>)</p>
<table cellpadding="5" width="100%" style="border-collapse: collapse;" border="1">
<tbody><tr>
<td>
<div style="font-family: monospace;" id="lengthNaive_term">
Term
</div>
</td>
</tr>
<tr>
<td>
<div style="font-family: monospace;" id="lengthNaive_heap">
Heap
</div>
</td>
</tr>
</tbody></table>
</blockquote>
<p>When you step through this, notice what is going on:</p>
<ul>
<li>We first apply the definition of <code>length</code>.</li>
<li>Then in step 1, <code>length</code> needs to do a case analysis, which forces us apply <code>enumFromTo</code>, and evaluate it until we have a top-level <code>Cons</code> constructor (in step 4)</li>
<li>At that point we can execute the pattern match and the process continues.</li>
<li>When we evaluate <code>enumFromTo (add 1 1) 3</code> in step 6, note that the expression <code>add 1 1</code> is only evaluated once, although it is used twice; this sharing of computation is what makes Haskell a true <em>lazy</em> (call-by-need) language (as opposed to a call-by-name language).</li>
<li>In these animations these shared expressions are shown separately below the expression; you can think of this as the “heap”, and accordingly the animation also shows when these expressions are garbage collected (e.g. step 14).</li>
</ul>
<p>Note as you step through that we have a build up of calls to <code>add</code> which are only resolved until the very end. This is the source of the memory leak in this definition of <code>length</code> (corresponding to <code>foldr</code>).</p>
<h3 id="foldl"><code>foldl</code></h3>
<p>In a <code>foldl</code>-style definition of <code>length</code>, we introduce an accumulator:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">length acc xs <span class="fu">=</span>
<span class="kw">case</span> xs <span class="kw">of</span>
[] <span class="ot">-></span> acc
x<span class="fu">:</span>xs' <span class="ot">-></span> length (<span class="dv">1</span> <span class="fu">+</span> acc) xs'</code></pre></div>
<p>This corresponds to defining <code>length = foldl (\n x -> 1 + n) 0</code>.</p>
<p>Unlike the previous definition, this is tail-recursive; however, it still suffers from a memory leak due to Haskell’s extremely lazy nature. You will see why when you step through the execution:</p>
<blockquote>
<p><a style="cursor: pointer;">Prev</a> <a style="cursor: pointer;">Next</a> (step <span id="lengthAcc_step">Step</span>, <span id="lengthAcc_status">Status</span>)</p>
<table cellpadding="5" width="100%" style="border-collapse: collapse;" border="1">
<tbody><tr>
<td>
<div style="font-family: monospace;" id="lengthAcc_term">
Term
</div>
</td>
</tr>
<tr>
<td>
<div style="font-family: monospace;" id="lengthAcc_heap">
Heap
</div>
</td>
</tr>
</tbody></table>
</blockquote>
<p>Notice how we are still building up a chain of additions, except they are now in the accumulator instead. This chain is only resolved (and garbage collected) at the very end (step 26).</p>
<h3 id="foldl-1"><code>foldl'</code></h3>
<p>In the final definition, we make sure to evaluate the accumulator as we go:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell">length acc xs <span class="fu">=</span>
<span class="kw">case</span> xs <span class="kw">of</span>
[] <span class="ot">-></span> acc
x<span class="fu">:</span>xs' <span class="ot">-></span> <span class="kw">let</span> acc' <span class="fu">=</span> add <span class="dv">1</span> acc <span class="kw">in</span> seq acc' (length acc' xs')</code></pre></div>
<p>This corresponds to defining <code>length = foldl' (\n x -> 1 + n) 0</code>.</p>
<p>If you step through this note that we evaluate the accumulator immediately at each step, and moreover that garbage collection can now happen as we compute as well:</p>
<blockquote>
<p><a style="cursor: pointer;">Prev</a> <a style="cursor: pointer;">Next</a> (step <span id="lengthAccStrict_step">Step</span>, <span id="lengthAccStrict_status">Status</span>)</p>
<table cellpadding="5" width="100%" style="border-collapse: collapse;" border="1">
<tbody><tr>
<td>
<div style="font-family: monospace;" id="lengthAccStrict_term">
Term
</div>
</td>
</tr>
<tr>
<td>
<div style="font-family: monospace;" id="lengthAccStrict_heap">
Heap
</div>
</td>
</tr>
</tbody></table>
</blockquote>
<h2 id="mapm-over-the-maybe-monad"><code>mapM</code> over the <code>Maybe</code> monad</h2>
<p>As a final example of a slightly more advanced nature, try predicting what will happen when we run this in <code>ghci</code>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">case</span> mapM return [<span class="dv">1</span><span class="fu">..</span>] <span class="kw">of</span> <span class="dt">Just</span> (x<span class="fu">:</span>_) <span class="ot">-></span> x</code></pre></div>
<p>If you try it out and the result is not what you expected, perhaps stepping through the following evaluation of <code>mapM return [1..3]</code> to weak-head normal form (whnf: when there is a constructor at the top-level) will help you understand:</p>
<blockquote>
<p><a style="cursor: pointer;">Prev</a> <a style="cursor: pointer;">Next</a> (step <span id="mapM_step">Step</span>, <span id="mapM_status">Status</span>)</p>
<table cellpadding="5" width="100%" style="border-collapse: collapse;" border="1">
<tbody><tr>
<td>
<div style="font-family: monospace;" id="mapM_term">
Term
</div>
</td>
</tr>
<tr>
<td>
<div style="font-family: monospace;" id="mapM_heap">
Heap
</div>
</td>
</tr>
</tbody></table>
</blockquote>
<p>Note that this expression reduces to whnf only after the entire list has been evaluated, and moreover that this requires an <code>O(n)</code> nested pattern matches. The take-away point from this example is that <code>mapM</code> should not be applied to long lists in most monads, as this will result in a memory leak.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Laziness can be tricky to understand sometimes, and being able to go through the evaluation of a program step by step can be very helpful. The <code>visualize-cbn</code> tool can be used to generate HTML/JavaScript files that can be used to visualize this evaluation as shown on this blog post; alternatively, it can also write the evaluation trace to the console. The source files (the various definitions of <code>length</code>) can be found in the <a href="https://github.com/well-typed/visualize-cbn">repo</a>. Feedback and pull requests are of course always welcome :)</p>Mon, 18 Sep 2017 15:45:16 +0000Neil Mitchell: Existential Serialisationtag:blogger.com,1999:blog-7094652.post-4289862185865729339
http://neilmitchell.blogspot.com/2017/09/existential-serialisation.html
<p><em>Summary: Using static pointers you can perform binary serialisation of existentials.</em></p><p>Many moons ago <a href="https://stackoverflow.com/questions/8101067/binary-instance-for-an-existential/45984547">I asked</a> how to write a <code>Binary</code> instance for a type including an existential, such as:</p><pre><code>data Foo = forall a . (Typeable a, Binary a) => Foo a<br /></code></pre><p>Here we have a constructor <code>Foo</code> which contains a value. We don't statically know the type of the contained value, but we do know it has the type classes <code>Typeable</code> (so we can at runtime switch on its type) and <code>Binary</code> (so we can serialise it). But how can we deserialise it? We can store the relevant <code>TypeRep</code> when serialising, but when deserialising there is no mechanism to map from <code>TypeRep</code> to a <code>Binary</code> instance.</p><p>In <a href="http://shakebuild.com">Shake</a>, I needed to serialise existentials, as described in <a href="http://ndmitchell.com/downloads/paper-shake_before_building-10_sep_2012.pdf">the S4.1 of the original paper</a>. My solution was to build a global mapping table, storing pairs of <code>TypeRep</code> and <code>Binary</code> instances for the types I knew were relevant. This solution works, but cannot deserialise anything that has not already been added to the global table, which required certain functions to live in weird places to ensure that they were called before deserialisation. Effective, but ugly.</p><p>Recently <a href="https://stackoverflow.com/users/1942289/abhiroop-sarkar">Abhiroop Sarkar</a> suggested using the relatively new <a href="https://downloads.haskell.org/~ghc/7.10.1/docs/html/users_guide/static-pointers.html">static pointers extension</a>. This extension lets you turn top-level bindings with no arguments into a <code>StaticPtr</code> which can then be serialised/deserialsed, even between different instances of a process. To take advantage of this feature, we can redefine <code>Foo</code> as:</p><pre><code>data Foo = forall a . (StaticFoo a, Binary a) => Foo a<br /><br />class StaticFoo a where<br /> staticFoo :: a -> StaticPtr (Get Foo)<br /></code></pre><p>The approach is to switch from serialising the <code>TypeRep</code> (from which we try to look up <code>Get Foo</code>), to serialising the <code>Get Foo</code> directly. We can write a <code>Binary Foo</code> instance by defining <code>put</code>:</p><pre><code>put :: Foo -> Put<br />put (Foo x) = do<br /> put $ staticKey $ staticFoo x<br /> put x<br /></code></pre><p>Here we simply grab a <code>StaticPtr (Get Foo)</code> which can deserialise the object, then use <code>staticKey</code> to turn it into something that can be serialised itself. Next, we write out the payload. To reverse this process we define <code>get</code>:</p><pre><code>get :: Get Foo<br />get = do<br /> ptr <- get<br /> case unsafePerformIO (unsafeLookupStaticPtr ptr) of<br /> Just value -> deRefStaticPtr value :: Get Foo<br /> Nothing -> error "Binary Foo: unknown static pointer"<br /></code></pre><p>We first <code>get</code> the <code>staticKey</code>, use <code>unsafeLookupStaticPtr</code> to turn it into a <code>StaticPtr (Get Foo)</code> followed by <code>deRefStaticPtr</code> to turn it into a <code>Get Foo</code>. The <code>unsafe</code> prefix on these functions is justified - bugs while developing this code resulted in segfaults.</p><p>The final piece of the puzzle is defining <code>StaticFoo</code> instances for the various types we might want to serialise. As an example for <code>String</code>:</p><pre><code>instance StaticFoo String where<br /> staticFoo _ = static (Foo <$> (get :: Get String))<br /></code></pre><p>We perform the <code>get</code>, wrap a <code>Foo</code> around it, and then turn it into a <code>StaticPtr</code>. All other types follow the same pattern, replacing <code>String</code> with <code>Int</code> (for example). The expression passed to <code>static</code> must have no free variables, including type variables, so we cannot define an instance for <code>a</code>, or even an instance for <code>[a]</code> - it must be <code>[Char]</code> and <code>[Int]</code> separately.</p><p>A complete code sample and test case is available <a href="https://gist.github.com/ndmitchell/a4f2edcedd2d4398efea4755b5d2408f">here</a>.</p><p>This approach works, and importantly allows extra constraints on the existential. The two disadvantages are: 1) that <code>static</code> isn't very flexible or easy to abstract over, resulting in a lot of <code>StaticFoo</code> boilerplate; 2) the static pointer is not guaranteed to be valid if the program changes in any way.</p><p>Will Shake be moving over to this approach? No. The next version of Shake has undergone an extensive rewrite, and in the process, moved away from needing this feature. A problem I had for 8 years has been solved, just as I no longer need the solution!</p>Sun, 17 Sep 2017 11:46:00 +0000noreply@blogger.com (Neil Mitchell)Dimitri Sabadie: State of luminancetag:blogger.com,1999:blog-8976038770606708499.post-3262371064455503487
http://phaazon.blogspot.com/2017/09/state-of-luminance.html
<p>I’ve been a bit surprised for a few days now. Some rustaceans posted two links about my <a href="https://crates.io/crates/spectra">spectra</a> and <a href="https://crates.io/crates/luminance">luminance</a> crates on reddit – links <a href="https://www.reddit.com/r/rust/comments/6z58n3/spectra_a_demoscene_engine_in_rust">here</a> and <a href="https://www.reddit.com/r/rust_gamedev/comments/6zmtb4/luminance_typesafe_typelevel_and_stateless_rust">here</a>. I didn’t really expect that: the code is public, on Github, but I don’t communicate about them <em>that much</em>.</p><p>However, I saw interested people, and I think it’s the right time to write a blog post about some design choices. I’ll start off with luminance and I’ll speak about spectra in another blog post because I truly think spectra starts to become very interesting (I have my own shading language bundled up within, which is basically GLSL on steroids).</p><h1 id="luminance-and-how-it-copes-with-fast-and-as-stateless-as-possible-graphics">luminance and how it copes with fast and as-stateless-as-possible graphics</h1><h2 id="origins">Origins</h2><p>luminance is a library that I wrote, historically, in Haskell. You can find the package <a href="https://hackage.haskell.org/package/luminance">here</a> if you’re interested – careful though, it’s dying and the Rust version has completely drifted path with it. Nevertheless, when I ported the library to Rust, I imported the same “way of programming” that I had·have in Haskell – besides the allocation scheme; remember, I’m a demoscener, I <strong>do care a lot about performance, CPU usage, cache friendliness and runtime size</strong>. So the Rust luminance crate was made to be hybrid: it has the cool functional side that I imported from my Haskell codebase, and the runtime performance I wanted that I had when I wrote my two first 64k in C++11. I had to remove and work around some features that only Haskell could provide, such as higher-kinded types, type families, functional dependencies, GADTs and a few other things such as existential quantification (trait objects saved me here, even though I don’t use them that much in luminance now).</p><p>I have to admit, I dithered a lot about the scope of luminance — both in Haskell and Rust. At first, I thought that it’d be great to have a <em>“base”</em> crate, hosting common and abstracted code, and <em>“backend”</em> crates, implementing the abstract interface. That would enable me to have several backends – OpenGL, Vulkan, Metal, a software implementation, something for Amiigaaaaaaa, etc. Though, time has passed, and now, I think it’s:</p><ul class="incremental"><li>Overkill.</li><li>A waste of framework.</li></ul><p>The idea is that if you need to be very low-level on the graphics stack of your application, you’re likely to know what you are doing. And then, your needs will be very precise and well-defined. You might want very specific piece of code to be available, related to a very specific technology. That’s the reason why abstracting over very low-level code is not a good path to me: you need to expose as most as posssible the low-level interface. That’s the goal of luminance: exposing OpenGL’s interface in a stateless, bindless and typesafe way, with no or as minimal as possible runtime overhead.</p><p>More reading <a href="http://phaazon.blogspot.fr/2016/08/luminance-designs.html">here</a>.</p><h2 id="today">Today</h2><p>Today, luminance is almost stable – it still receives massive architecture redesign from time to time, but it’ll hit the <code>1.0.0</code> release soon. As discussed with <a href="https://github.com/kvark">kvark</a> lately, luminance is not about the same scope as <a href="https://crates.io/crates/gfx">gfx</a>’s one. The goals of luminance are:</p><ul class="incremental"><li>To be a typesafe, stateless and bindless OpenGL framework.</li><li>To provide a friendly experience and expose as much as possible all of the OpenGL features.</li><li>To be very lightweight (the target is to be able to use it without <code>std</code> nor <code>core</code>).</li></ul><p>To achieve that, luminance is written with several aspects in mind:</p><ul class="incremental"><li>Allocation must be explicitely stated by the user: we must avoid as much as possible to allocate things in luminance since it might become both a bottleneck and an issue to the lightweight aspect.</li><li>Performance is a first priority; safety comes second. If you have a feature that can be either exclusively performant or safe, it must then be performant. Most of the current code is, for our joy, both performant and safe. However, some invariants are left around the place and you might shoot your own feet. This is an issue and some reworking must be done (along with tagging some functions and traits <code>unsafe</code>).</li><li>No concept of backends will ever end up in luminance. If it’s decided to switch to <a href="https://www.khronos.org/vulkan">Vulkan</a>, the whole luminance API will and <strong>must</strong> be impacted, so that people can use <a href="https://www.khronos.org/vulkan">Vulkan</a> the best possible way.</li><li>A bit like the first point, the code must be written in a way that the generated binary is as small as possible. Generics are not forbidden – they’re actually recommended – but things like crate dependencies are likely to be forbidden (exception for the <code>gl</code> dependency, of course).</li><li>Windowing <strong>must not be addressed by luminance</strong>. This is crucial. As a demoscener, if I want to write a 64k with luminance, I must be able to use a library over X11 or the Windows API to setup the OpenGL context myself, set the OpenGL pointers myself, etc. This is not the typical usecase – who cares besides demosceners?! – but it’s still a good advantage since you end up with loose coupling for free.</li></ul><h2 id="the-new-luminance">The new luminance</h2><p>luminance has received more attention lately, and I think it’s a good thing to talk about how to use it. I’ll add examples on github and its <a href="https://docs.rs/luminance">docs.rs</a> online documentation.</p><p>I’m going to do that like a tutorial. It’s easier to read and you can test the code in the same time. Let’s render a triangle!</p><blockquote><p>Note: keep in mind that you need a nightly compiler to compile luminance.</p></blockquote><h3 id="getting-your-feet-wet">Getting your feet wet</h3><p>I’ll do everything from scratch with you. I’ll work in <code>/tmp</code>:</p><pre><code>$ cd /tmp</code></pre><p>First thing first, let’s setup a <code>lumitest</code> Rust binary project:</p><pre><code>$ cargo init --bin lumitest<br /> Created binary (application) project<br />$ cd lumitest</code></pre><p>Let’s edit our <code>Cargo.toml</code> to use luminance. We’ll need two crates:</p><ul class="incremental"><li>The <a href="https://crates.io/crates/luminance">luminance</a> crate.</li><li>A way to open the OpenGL context; we’ll use GLFW, so the <a href="https://crates.io/crates/luminance-glfw">luminance-glfw</a> crate.</li></ul><p>At the time of writing, corresponding versions are <a href="https://crates.io/crates/luminance/0.23.0">luminance-0.23.0</a> and <a href="https://crates.io/crates/luminance-glfw/0.3.2">luminance-glfw-0.3.2</a>.</p><p>Have the following <code>[dependencies]</code> section</p><pre class="toml"><code>[dependencies]<br />luminance = "0.23.0"<br />luminance-glfw = "0.3.2"</code></pre><pre><code>$ cargo check</code></pre><p>Everything should be fine at this point. Now, let’s step in in writing some code.</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">extern</span> <span class="kw">crate</span> luminance;<br /><span class="kw">extern</span> <span class="kw">crate</span> luminance_glfw;<br /><br /><span class="kw">use</span> luminance_glfw::<span class="op">{</span>Device, WindowDim, WindowOpt<span class="op">}</span>;<br /><br /><span class="kw">const</span> SCREEN_WIDTH: <span class="dt">u32</span> = <span class="dv">960</span>;<br /><span class="kw">const</span> SCREEN_HEIGHT: <span class="dt">u32</span> = <span class="dv">540</span>;<br /><br /><span class="kw">fn</span> main() <span class="op">{</span><br /> <span class="kw">let</span> rdev = Device::new(WindowDim::Windowed(SCREEN_WIDTH, SCREEN_HEIGHT), <span class="st">"lumitest"</span>, WindowOpt::default());<br /><span class="op">}</span></code></pre></div><p>The <code>main</code> function creates a <a href="https://docs.rs/luminance-glfw/0.3.2/luminance_glfw/struct.Device.html">Device</a> that is responsible in holding the windowing stuff for us.</p><p>Let’s go on:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">match</span> rdev <span class="op">{</span><br /> <span class="cn">Err</span>(e) => <span class="op">{</span><br /> <span class="pp">eprintln!</span>(<span class="st">"{:#?}"</span>, e);<br /> ::std::process::exit(<span class="dv">1</span>);<br /> <span class="op">}</span><br /><br /> <span class="cn">Ok</span>(<span class="kw">mut</span> dev) => <span class="op">{</span><br /> <span class="pp">println!</span>(<span class="st">"let’s go!"</span>);<br /> <span class="op">}</span><br /><span class="op">}</span></code></pre></div><p>This block will catch any <code>Device</code> errors and will print them to <code>stderr</code> if there’s any.</p><p>Let’s write the main loop:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="ot">'app</span>: <span class="kw">loop</span> <span class="op">{</span><br /> <span class="kw">for</span> (_, ev) <span class="kw">in</span> dev.events() <span class="op">{</span> <span class="co">// the pair is an interface mistake; it’ll be removed</span><br /> <span class="kw">match</span> ev <span class="op">{</span><br /> WindowEvent::Close | WindowEvent::Key(Key::Escape, _, Action::Release, _) => <span class="kw">break</span> <span class="ot">'app</span>,<br /> _ => ()<br /> <span class="op">}</span><br /> <span class="op">}</span><br /><span class="op">}</span></code></pre></div><p>This loop runs forever and will exit if you hit the escape key or quit the application.</p><h3 id="setting-up-the-resources">Setting up the resources</h3><p>Now, the most interesting thing: rendering the actual triangle! You will need a few things:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">type</span> Position = <span class="op">[</span><span class="dt">f32</span>; <span class="dv">2</span><span class="op">]</span>;<br /><span class="kw">type</span> RGB = <span class="op">[</span><span class="dt">f32</span>; <span class="dv">3</span><span class="op">]</span>;<br /><span class="kw">type</span> Vertex = (Position, RGB);<br /><br /><span class="kw">const</span> TRIANGLE_VERTS: <span class="op">[</span>Vertex; <span class="dv">3</span><span class="op">]</span> = <span class="op">[</span><br /> (<span class="op">[</span>-<span class="dv">0.5</span>, -<span class="dv">0.5</span><span class="op">]</span>, <span class="op">[</span><span class="dv">0.8</span>, <span class="dv">0.5</span>, <span class="dv">0.5</span><span class="op">]</span>), <span class="co">// red bottom leftmost</span><br /> (<span class="op">[</span>-<span class="dv">0.</span>, <span class="dv">0.5</span><span class="op">]</span>, <span class="op">[</span><span class="dv">0.5</span>, <span class="dv">0.8</span>, <span class="dv">0.5</span><span class="op">]</span>), <span class="co">// green top</span><br /> (<span class="op">[</span><span class="dv">0.5</span>, -<span class="dv">0.5</span><span class="op">]</span>, <span class="op">[</span><span class="dv">0.5</span>, <span class="dv">0.5</span>, <span class="dv">0.8</span><span class="op">]</span>) <span class="co">// blue bottom rightmost</span><br /><span class="op">]</span>;</code></pre></div><p><code>Position</code>, <code>Color</code> and <code>Vertex</code> define what a vertex is. In our case, we use a 2D position and a RGB color.</p><blockquote><p>You have a lot of choices here to define the type of your vertices. In theory, you can choose any type you want. However, it must implement the <a href="https://docs.rs/luminance/0.23.0/luminance/vertex/trait.Vertex.html"><code>Vertex</code></a> trait. Have a look at the implementors that already exist for a faster start off!</p></blockquote><blockquote><p><strong>Important</strong>: do not confuse between <code>[f32; 2]</code> and <code>(f32, f32)</code>. The former is a single 2D vertex component. The latter is two 1D components. It’ll make a huge difference when writing shaders.</p></blockquote><p>The <code>TRIANGLE_VERTS</code> is a constant array with three vertices defined in it: the three vertices of our triangle. Let’s pass those vertices to the GPU with the <a href="https://docs.rs/luminance/0.23.0/luminance/tess/struct.Tess.html"><code>Tess</code></a> type:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="co">// at the top location</span><br /><span class="kw">use</span> luminance::tess::<span class="op">{</span>Mode, Tess, TessVertices<span class="op">}</span>;<br /><br /><span class="co">// just above the main loop</span><br /><span class="kw">let</span> triangle = Tess::new(Mode::Triangle, TessVertices::Fill(&TRIANGLE_VERTS), <span class="cn">None</span>);</code></pre></div><p>This will pass the <code>TRIANGLE_VERTS</code> vertices to the GPU. You’re given back a <code>triangle</code> object. The <a href="https://docs.rs/luminance/0.23.0/luminance/tess/enum.Mode.html"><code>Mode</code></a> is a hint object that states how vertices must be connected to each other. <code>TessVertices</code> lets you slice your vertices – this is typically enjoyable when you use a mapped buffer that contains a dynamic number of vertices.</p><p>We’ll need a <em>shader</em> to render that triangle. First, we’ll place its source code in <code>data</code>:</p><pre><code>$ mkdir data</code></pre><p>Paste this in <code>data/vs.glsl</code>:</p><div class="sourceCode"><pre class="sourceCode glsl"><code class="sourceCode glsl"><span class="kw">layout</span> (<span class="dt">location</span> = <span class="dv">0</span>) <span class="dt">in</span> <span class="dt">vec2</span> co;<br /><span class="kw">layout</span> (<span class="dt">location</span> = <span class="dv">1</span>) <span class="dt">in</span> <span class="dt">vec3</span> color;<br /><br /><span class="dt">out</span> <span class="dt">vec3</span> v_color;<br /><br /><span class="dt">void</span> <span class="fu">main</span>() {<br /> <span class="bu">gl_Position</span> = <span class="dt">vec4</span>(co, <span class="dv">0</span>., <span class="dv">1</span>.);<br /> v_color = color;<br />}</code></pre></div><p>Paste this in <code>data/fs.glsl</code>:</p><div class="sourceCode"><pre class="sourceCode glsl"><code class="sourceCode glsl"><span class="dt">in</span> <span class="dt">vec3</span> v_color;<br /><br /><span class="dt">out</span> <span class="dt">vec4</span> frag;<br /><br /><span class="dt">void</span> <span class="fu">main</span>() {<br /> frag = <span class="dt">vec4</span>(v_color, <span class="dv">1</span>.);<br />}</code></pre></div><p>And add this to your <code>main.rs</code>:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">const</span> SHADER_VS: &<span class="dt">str</span> = <span class="pp">include_str!</span>(<span class="st">"../data/vs.glsl"</span>);<br /><span class="kw">const</span> SHADER_FS: &<span class="dt">str</span> = <span class="pp">include_str!</span>(<span class="st">"../data/fs.glsl"</span>);</code></pre></div><blockquote><p>Note: this is not a typical workflow. If you’re interested in shaders, have a look at how I do it in <a href="https://crates.io/crates/spectra">spectra</a>. That is, hot reloading it via SPSL (Spectra Shading Language), which enables to write GLSL modules and compose them in a single file but just writing functions. The functional programming style!</p></blockquote><p>Same thing as for the tessellation, we need to pass the source to the GPU’s compiler to end up with a shader object:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="co">// add this at the top of your main.rs</span><br /><span class="kw">use</span> luminance::shader::program::Program;<br /><br /><span class="co">// below declaring triangle</span><br /><span class="kw">let</span> (shader, warnings) = Program::<Vertex, (), ()>::from_strings(<span class="cn">None</span>, SHADER_VS, <span class="cn">None</span>, SHADER_FS).unwrap();<br /><br /><span class="kw">for</span> warning <span class="kw">in</span> &warnings <span class="op">{</span><br /> <span class="pp">eprintln!</span>(<span class="st">"{:#?}"</span>, warning);<br /><span class="op">}</span></code></pre></div><p>Finally, we need to tell luminance in which framebuffer we want to make the render. It’s simple: to the default framebuffer, which ends up to be… your screen’s back buffer! This is done this way with luminance:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">use</span> luminance::framebuffer::Framebuffer;<br /><br /><span class="kw">let</span> screen = Framebuffer::default(<span class="op">[</span>SCREEN_WIDTH, SCREEN_HEIGHT<span class="op">]</span>);</code></pre></div><p>And we’re done for the resources. Let’s step in the actual render now.</p><h3 id="the-actual-render-the-pipeline">The actual render: the pipeline</h3><p>luminance’s approach to render is somewhat not very intuitive yet very simple and very efficient: the render pipeline is explicitly defined by the programmer in Rust, on the fly. That means that you must express the actual state the GPU must have for the whole pipeline. Because of the nature of the pipeline, which is an AST (Abstract Syntax Tree), you can batch sub-parts of the pipeline (we call such parts <em>nodes</em>) and you end up with minimal GPU state switches. The theory is as following:</p><ul class="incremental"><li>At top most, you have the <a href="https://docs.rs/luminance/0.23.0/luminance/pipeline/fn.pipeline.html"><code>pipeline</code></a> function that introduces the concept of <em>shading things to a framebuffer</em>.</li><li>Nested, you find the concept of a <em>shader gate</em>. That is, an object linked to its parent (pipeline) and that gives you the concept of <em>shading things with a shader</em>. <ul class="incremental"><li>Nested, you find the concept of <em>rendering things</em>. That is, can set on such nodes GPU states, such as whether you want a depth test, blending, etc.</li><li>Nested, you find the concept of a <em>tessellation gate</em>, enabling you to render actual <code>Tess</code> objects.</li></ul></li></ul><p>That deep nesting enables you to batch your objects on a very fine granularity. Also, notice that the functions are not about slices of <code>Tess</code> or hashmaps of <code>Program</code>. The allocation scheme is completely ignorant about how the data is traversed, which is good: you decide. If you need to borrow things on the fly in a shading gate, you can.</p><p>Let’s get things started:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">use</span> luminance::pipeline::<span class="op">{</span>entry, pipeline<span class="op">}</span>;<br /><br />entry(|_| <span class="op">{</span><br /> pipeline(&screen, <span class="op">[</span><span class="dv">0.</span>, <span class="dv">0.</span>, <span class="dv">0.</span>, <span class="dv">1.</span><span class="op">]</span>, |shd_gate| <span class="op">{</span><br /> shd_gate.shade(&shader, |rdr_gate, _| <span class="op">{</span><br /> rdr_gate.render(<span class="cn">None</span>, <span class="cn">true</span>, |tess_gate| <span class="op">{</span><br /> <span class="kw">let</span> t = &triangle;<br /> tess_gate.render(t.into());<br /> <span class="op">}</span>);<br /> <span class="op">}</span>);<br /> <span class="op">}</span>);<br /><span class="op">}</span>);</code></pre></div><p>We just need a final thing now: since we render to the back buffer of the screen, if we want to see anything appear, we need to <em>swap the buffer chain</em> so that the back buffer become the front buffer and the front buffer become the back buffer. This is done by wrapping our render code in the <a href="https://docs.rs/luminance-glfw/0.3.2/luminance_glfw/struct.Device.html#method.draw"><code>Device::draw</code></a> function:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust">dev.draw(|| <span class="op">{</span><br /> entry(|_| <span class="op">{</span><br /> pipeline(&screen, <span class="op">[</span><span class="dv">0.</span>, <span class="dv">0.</span>, <span class="dv">0.</span>, <span class="dv">1.</span><span class="op">]</span>, |shd_gate| <span class="op">{</span><br /> shd_gate.shade(&shader, |rdr_gate, _| <span class="op">{</span><br /> rdr_gate.render(<span class="cn">None</span>, <span class="cn">true</span>, |tess_gate| <span class="op">{</span><br /> <span class="kw">let</span> t = &triangle;<br /> tess_gate.render(t.into());<br /> <span class="op">}</span>);<br /> <span class="op">}</span>);<br /> <span class="op">}</span>);<br /> <span class="op">}</span>);<br /><span class="op">}</span>);</code></pre></div><p>You should see this:</p><div class="figure"><img src="https://i.imgur.com/x3jC9qm.png" /> </div><p>As you can see, the code is pretty straightforward. Let’s get deeper, and let’s kick some time in!</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">use</span> std::time::Instant;<br /><br /><span class="co">// before the main loop</span><br /><span class="kw">let</span> t_start = Instant::now();<br /><span class="co">// in your main loop</span><br /><span class="kw">let</span> t_dur = t_start.elapsed();<br /><span class="kw">let</span> t = (t_dur.as_secs() <span class="kw">as</span> <span class="dt">f64</span> + t_dur.subsec_nanos() <span class="kw">as</span> <span class="dt">f64</span> * <span class="dv">1</span>e-<span class="dv">9</span>) <span class="kw">as</span> <span class="dt">f32</span>;</code></pre></div><p>We have the time. Now, we need to pass it down to the GPU (i.e. the shader). luminance handles that kind of things with two concepts:</p><ul class="incremental"><li>Uniforms.</li><li>Buffers.</li></ul><p>Uniforms are a good match when you want to send data to a specific shader, like a value that customizes the behavior of a shading algorithm.</p><p>Because buffers are shared, you can use buffers to share data between shader, leveraging the need to pass the data to all shader by hand – you only pass the index to the buffer that contains the data.</p><p>We won’t conver the buffers for this time.</p><p>Because of type safety, luminance requires you to state which types the uniforms the shader contains are. We only need the time, so let’s get this done:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="co">// you need to alter this import</span><br /><span class="kw">use</span> luminance::shader::program::<span class="op">{</span>Program, ProgramError, Uniform, UniformBuilder, UniformInterface, UniformWarning<span class="op">}</span>;<br /><br /><span class="kw">struct</span> TimeUniform(Uniform<<span class="dt">f32</span>>);<br /><br /><span class="kw">impl</span> UniformInterface <span class="kw">for</span> TimeUniform <span class="op">{</span><br /> <span class="kw">fn</span> uniform_interface(builder: UniformBuilder) -> <span class="dt">Result</span><(<span class="kw">Self</span>, <span class="dt">Vec</span><UniformWarning>), ProgramError> <span class="op">{</span><br /> <span class="co">// this will fail if the "t" variable is not used in the shader</span><br /> <span class="co">//let t = builder.ask("t").map_err(ProgramError::UniformWarning)?;</span><br /><br /> <span class="co">// I rather like this one: we just forward up the warning and use the special unbound uniform</span><br /> <span class="kw">match</span> builder.ask(<span class="st">"t"</span>) <span class="op">{</span><br /> <span class="cn">Ok</span>(t) => <span class="cn">Ok</span>((TimeUniform(t), <span class="dt">Vec</span>::new())),<br /> <span class="cn">Err</span>(e) => <span class="cn">Ok</span>((TimeUniform(builder.unbound()), <span class="pp">vec!</span><span class="op">[</span>e<span class="op">]</span>))<br /> <span class="op">}</span><br /> <span class="op">}</span><br /><span class="op">}</span></code></pre></div><p>The <a href="https://docs.rs/luminance/0.23.0/luminance/shader/program/struct.UniformBuilder.html#method.unbound"><code>UniformBuilder::unbound</code></a> is a simple function that gives you any uniform you want: the resulting uniform object will just do nothing when you pass values in. It’s a way to say <em>“— Okay, I don’t use that in the shader yet, but don’t fail, it’s not really an error”.</em> Handy.</p><p>And now, all the magic: how do we access that uniform value? It’s simple: via types! Have you noticed the type of our <code>Program</code>? For the record:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">let</span> (shader, warnings) = Program::<Vertex, (), ()>::from_strings(<span class="cn">None</span>, SHADER_VS, <span class="cn">None</span>, SHADER_FS).unwrap();</code></pre></div><p>See the type is parametered with three type variables:</p><ul class="incremental"><li>The first one – here <code>Vertex</code>, our own type – is for the <em>input</em> of the shader program.</li><li>The second one is for the <em>output</em> of the shader program. It’s currently not used at all by luminance by is reserved, as it will be used later for enforcing even further type safety.</li><li>The third and latter is for the <em>uniform interface</em>.</li></ul><p>You guessed it: we need to change the third parameter from <code>()</code> to <code>TimeUniform</code>:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust"><span class="kw">let</span> (shader, warnings) = Program::<Vertex, (), TimeUniform>::from_strings(<span class="cn">None</span>, SHADER_VS, <span class="cn">None</span>, SHADER_FS).unwrap();</code></pre></div><p>And <em>that’s all</em>. Whenever you <code>shade</code> with a <code>ShaderGate</code>, the type of the shader object is being inspected, and you’re handed with the uniform interface:</p><div class="sourceCode"><pre class="sourceCode rust"><code class="sourceCode rust">shd_gate.shade(&shader, |rdr_gate, uniforms| <span class="op">{</span><br /> uniforms.<span class="dv">0.</span>update(t);<br /><br /> rdr_gate.render(<span class="cn">None</span>, <span class="cn">true</span>, |tess_gate| <span class="op">{</span><br /> <span class="kw">let</span> t = &triangle;<br /> tess_gate.render(t.into());<br /> <span class="op">}</span>);<br /><span class="op">}</span>);</code></pre></div><p>Now, change your fragment shader to this:</p><div class="sourceCode"><pre class="sourceCode glsl"><code class="sourceCode glsl"><span class="dt">in</span> <span class="dt">vec3</span> v_color;<br /><br /><span class="dt">out</span> <span class="dt">vec4</span> frag;<br /><br /><span class="kw">uniform</span> <span class="dt">float</span> t;<br /><br /><span class="dt">void</span> <span class="fu">main</span>() {<br /> frag = <span class="dt">vec4</span>(v_color * <span class="dt">vec3</span>(<span class="bu">cos</span>(t * .<span class="fu">25</span>), <span class="bu">sin</span>(t + <span class="dv">1</span>.), <span class="bu">cos</span>(<span class="fl">1.25</span> * t)), <span class="dv">1</span>.);<br />}</code></pre></div><p>And enjoy the result! Here’s the <a href="https://gist.github.com/phaazon/268d5c0285c6c7cba90d5cea1b99db76">gist</a> that contains the whole <code>main.rs</code>.</p>Thu, 14 Sep 2017 15:58:40 +0000noreply@blogger.com (Dimitri Sabadie)Tom Schrijvers: PPDP & LOPSTR 2017: Call for Participationtag:blogger.com,1999:blog-5844006452378085451.post-7169360913034785527
http://tomschrijvers.blogspot.com/2017/09/ppdp-lopstr-2017-call-for-participation.html
<div style="text-align: left;" dir="ltr">***********************************************<br /><br /> CALL FOR PARTICIPATION<br /><br /><br />PPDP 2017<br /><br /> 19th International Symposium on<br /> Principles and Practice of Declarative Programming<br /> Namur, Belgium, October 9-11<br /><br /> http://complogic.cs.mcgill.ca/ppdp2017<br /><br />co-located with<br /><br /><br />LOPSTR 2017<br /><br /> 27th International Symposium on<br /> Logic-Based Program Synthesis and Transformation<br /> Namur, Belgium, October 10-12<br /><br /> https://www.sci.unich.it/lopstr17/<br /><br />***********************************************<br /><br />Registration is now open:<br /><br />https://events.info.unamur.be/ppdp-lopstr-2017/<br /><br />** Early registration deadline: September 15, 2017 **<br /><br /><br />INVITED TALKS:<br /><br />Marieke Huisman (Universiteit Twente)<br />A Verification Technique for Deterministic Parallel Programs<br />(joint PPDP/LOPSTR speaker)<br /><br />Sumit Gulwani (Microsoft)<br />Programming by Examples: Applications, Algorithms, and Ambiguity Resolution<br />(joint PPDP/LOPSTR speaker)<br /><br />Serge Abiteboul (INRIA)<br />Ethical issues in data management<br />(PPDP)<br /><br />Grigore Rosu (University of Illinois at Urbana-Champaign)<br />K: A Logic-Based Framework for Program Transformation and Analysis<br />(LOPSTR)<br /><br />Please consult the conferences' webpages for a list of accepted papers.<br /><br />Hope to see you in Namur !</div>Tue, 12 Sep 2017 07:27:07 +0000noreply@blogger.com (Tom Schrijvers)Joachim Breitner: Less parentheseshttp://www.joachim-breitner.de/blog/730-Less_parentheses
http://www.joachim-breitner.de/blog/730-Less_parentheses
<p>Yesterday, at the <a href="http://conf.researchr.org/track/hiw-2017/hiw-2017">Haskell Implementers Workshop 2017</a> in Oxford, I gave a lightning talk titled ”syntactic musings”, where I presented three possibly useful syntactic features that one might want to add to a language like Haskell.</p>
<p>The talked caused quite some heated discussions, and since the Internet likes heated discussion, I will happily share these ideas with you</p>
<h3 id="context-aka.-sections">Context aka. Sections</h3>
<p>This is probably the most relevant of the three proposals. Consider a bunch of related functions, say <code>analyseExpr</code> and <code>analyseAlt</code>, like these:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">analyseExpr ::</span> <span class="dt">Expr</span> <span class="ot">-></span> <span class="dt">Expr</span>
analyseExpr (<span class="dt">Var</span> v) <span class="fu">=</span> change v
analyseExpr (<span class="dt">App</span> e1 e2) <span class="fu">=</span>
<span class="dt">App</span> (analyseExpr e1) (analyseExpr e2)
analyseExpr (<span class="dt">Lam</span> v e) <span class="fu">=</span> <span class="dt">Lam</span> v (analyseExpr flag e)
analyseExpr (<span class="dt">Case</span> scrut alts) <span class="fu">=</span>
<span class="dt">Case</span> (analyseExpr scrut) (analyseAlt <span class="fu"><$></span> alts)
<span class="ot">analyseAlt ::</span> <span class="dt">Alt</span> <span class="ot">-></span> <span class="dt">Alt</span>
analyseAlt (dc, pats, e) <span class="fu">=</span> (dc, pats, analyseExpr e)</code></pre>
<p>You have written them, but now you notice that you need to make them configurable, e.g. to do different things in the <code>Var</code> case. You thus add a parameter to all these functions, and hence an argument to every call:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">type</span> <span class="dt">Flag</span> <span class="fu">=</span> <span class="dt">Bool</span>
<span class="ot">analyseExpr ::</span> <span class="dt">Flag</span> <span class="ot">-></span> <span class="dt">Expr</span> <span class="ot">-></span> <span class="dt">Expr</span>
analyseExpr flag (<span class="dt">Var</span> v) <span class="fu">=</span> <span class="kw">if</span> flag <span class="kw">then</span> change1 v <span class="kw">else</span> change2 v
analyseExpr flag (<span class="dt">App</span> e1 e2) <span class="fu">=</span>
<span class="dt">App</span> (analyseExpr flag e1) (analyseExpr flag e2)
analyseExpr flag (<span class="dt">Lam</span> v e) <span class="fu">=</span> <span class="dt">Lam</span> v (analyseExpr (not flag) e)
analyseExpr flag (<span class="dt">Case</span> scrut alts) <span class="fu">=</span>
<span class="dt">Case</span> (analyseExpr flag scrut) (analyseAlt flag <span class="fu"><$></span> alts)
<span class="ot">analyseAlt ::</span> <span class="dt">Flag</span> <span class="ot">-></span> <span class="dt">Alt</span> <span class="ot">-></span> <span class="dt">Alt</span>
analyseAlt flag (dc, pats, e) <span class="fu">=</span> (dc, pats, analyseExpr flag e)</code></pre>
<p>I find this code problematic. The intention was: “<code>flag</code> is a parameter that an external caller can use to change the behaviour of this code, but when reading and reasoning about this code, <code>flag</code> should be considered constant.”</p>
<p>But this intention is neither easily visible nor enforced. And in fact, in the above code, <code>flag</code> does “change”, as <code>analyseExpr</code> passes something else in the <code>Lam</code> case. The idiom is indistinguishable from the <em>environment idiom</em>, where a <em>locally changing</em> environment (such as “variables in scope”) is passed around.</p>
<p>So we are facing exactly the same problem as when reasoning about a loop in an imperative program with mutable variables. And we (pure functional programmers) should know better: We cherish immutability! We want to bind our variables once and have them scope over everything we need to scope over!</p>
<p>The solution I’d like to see in Haskell is common in other languages (Gallina, Idris, Agda, Isar), and this is what it would look like here:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">type</span> <span class="dt">Flag</span> <span class="fu">=</span> <span class="dt">Bool</span>
section (<span class="ot">flag ::</span> <span class="dt">Flag</span>) <span class="kw">where</span>
<span class="ot"> analyseExpr ::</span> <span class="dt">Expr</span> <span class="ot">-></span> <span class="dt">Expr</span>
analyseExpr (<span class="dt">Var</span> v) <span class="fu">=</span> <span class="kw">if</span> flag <span class="kw">then</span> change1 v <span class="kw">else</span> change2v
analyseExpr (<span class="dt">App</span> e1 e2) <span class="fu">=</span>
<span class="dt">App</span> (analyseExpr e1) (analyseExpr e2)
analyseExpr (<span class="dt">Lam</span> v e) <span class="fu">=</span> <span class="dt">Lam</span> v (analyseExpr e)
analyseExpr (<span class="dt">Case</span> scrut alts) <span class="fu">=</span>
<span class="dt">Case</span> (analyseExpr scrut) (analyseAlt <span class="fu"><$></span> alts)
<span class="ot"> analyseAlt ::</span> <span class="dt">Alt</span> <span class="ot">-></span> <span class="dt">Alt</span>
analyseAlt (dc, pats, e) <span class="fu">=</span> (dc, pats, analyseExpr e)</code></pre>
<p>Now the intention is clear: Within a clearly marked block, <code>flag</code> is fixed and when reasoning about this code I do not have to worry that it might change. Either <em>all</em> variables will be passed to <code>change1</code>, or <em>all</em> to <code>change2</code>. An important distinction!</p>
<p>Therefore, inside the <code>section</code>, the type of <code>analyseExpr</code> does not mention <code>Flag</code>, whereas outside its type is <code>Flag -> Expr -> Expr</code>. This is a bit unusual, but not completely: You see precisely the same effect in a <code>class</code> declaration, where the type signature of the methods do not mention the class constraint, but outside the declaration they do.</p>
<p>Note that idioms like implicit parameters or the <code>Reader</code> monad do not give the guarantee that the parameter is (locally) constant.</p>
<p>More details can be found in the <a href="https://github.com/ghc-proposals/ghc-proposals/pull/40">GHC proposal</a> that I prepared, and I invite you to raise concern or voice support there.</p>
<p>Curiously, this problem must have bothered me for longer than I remember: I discovered that seven years ago, I wrote a Template Haskell based implementation of this idea in the <a href="http://hackage.haskell.org/package/seal-module"><code>seal-module</code></a> package!</p>
<h3 id="less-parentheses-1-bulleted-argument-lists">Less parentheses 1: Bulleted argument lists</h3>
<p>The next two proposals are all about removing parentheses. I believe that Haskell’s tendency to express complex code with no or few parentheses is one of its big strengths, as it makes it easier to visualy parse programs. A common idiom is to use the <code>$</code> operator to separate a function from a complex argument without parentheses, but it does not help when there are multiple complex arguments.</p>
<p>For that case I propose to steal an idea from the surprisingly successful markup language markdown, and use bulleted lists to indicate multiple arguments:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">foo ::</span> <span class="dt">Baz</span>
foo <span class="fu">=</span> bracket
• some complicated code
that is evaluated first
• other complicated code for later
• even more complicated code</code></pre>
<p>I find this very easy to visually parse and navigate.</p>
<p>It is actually possible to do this now, if one defines <code>(•) = id</code> with <code>infixl 0 •</code>. A dedicated syntax extension (<code>-XArgumentBullets</code>) is preferable:</p>
<ul>
<li>It only really adds readability if the bullets are nicely vertically aligned, which the compiler should enforce.</li>
<li>I would like to use <code>$</code> inside these complex arguments, and multiple operators of precedence 0 do not mix. (<code>infixl -1 •</code> would help).</li>
<li>It should be possible to nest these, and distinguish different nesting levers based on their indentation.</li>
</ul>
<h3 id="less-parentheses-1-whitespace-precedence">Less parentheses 1: Whitespace precedence</h3>
<p>The final proposal is the most daring. I am convinced that it improves readability and should be considered when creating a new language. As for Haskell, I am at the moment not proposing this as a language extension (but could be convinced to do so if there is enough positive feedback).</p>
<p>Consider this definition of append:</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">(++) ::</span> [a] <span class="ot">-></span> [a] <span class="ot">-></span> [a]
[] <span class="fu">++</span> ys <span class="fu">=</span> ys
(x<span class="fu">:</span>xs) <span class="fu">++</span> ys <span class="fu">=</span> x <span class="fu">:</span> (xs<span class="fu">++</span>ys)</code></pre>
<p>Imagine you were explaining the last line to someone orally. How would you speak it? One common way to do so is to not read the parentheses out aloud, but rather speak parenthesised expression more quickly and add pauses otherwise.</p>
<p>We can do the same in syntax!</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">(++) ::</span> [a] <span class="ot">-></span> [a] <span class="ot">-></span> [a]
[] <span class="fu">++</span> ys <span class="fu">=</span> ys
x<span class="fu">:</span>xs <span class="fu">++</span> ys <span class="fu">=</span> x <span class="fu">:</span> xs<span class="fu">++</span>ys</code></pre>
<p>The rule is simple: <em>A sequence of tokens without any space is implicitly parenthesised.</em></p>
<p>The reaction I got in Oxford was horror and disgust. And that is understandable – we are very used to ignore spacing when parsing expressions (unless it is indentation, of course. Then we are no longer horrified, as our non-Haskell colleagues are when they see our code).</p>
<p>But I am convinced that once you let the rule sink in, you will have no problem parsing such code with ease, and soon even with greater ease than the parenthesised version. It is a very natural thing to look at the general structure, identify “compact chunks of characters”, mentally group them, and then go and separately parse the internals of the chunks and how the chunks relate to each other. More natural than first scanning everything for <code>(</code> and <code>)</code>, matching them up, building a mental tree, and then digging deeper.</p>
<p>Incidentally, there was a non-programmer present during my presentation, and while she did not openly contradict the dismissive groan of the audience, I later learned that she found this variant quite obvious to understand and more easily to read than the parenthesised code.</p>
<p>Some FAQs about this:</p>
<ul>
<li><em>What about an operator with space on one side but not on the other?</em> I’d simply forbid that, and hence enforce readable code.</li>
<li><em>Do operator sections still require parenthesis?</em> Yes, I’d say so.</li>
<li>Does this overrule operator precedence? Yes! <code>a * b+c == a * (b+c)</code>.</li>
<li><em>What is a token</em>? Good question, and I am not not decided. In particular: Is a parenthesised expression a single token? If so, then <code>(Succ a)+b * c</code> parses as <code>((Succ a)+b) * c</code>, otherwise it should probably simply be illegal.</li>
<li><em>Can we extend this so that one space binds tighter than two spaces, and so on?</em> Yes we can, but really, we should not.</li>
<li><em>This is incompatible with Agda’s syntax!</em> Indeed it is, and I really like Agda’s mixfix syntax. Can’t have everything.</li>
<li><em>Has this been done before?</em> I have not seen it in any language, but <a href="http://wall.org/~lewis/2013/10/25/whitespace-precedence.html">Lewis Wall</a> has blogged this idea before.</li>
</ul>
<p>Well, let me know what you think!</p>Sun, 10 Sep 2017 10:10:16 +0000mail@joachim-breitner.de (Joachim Breitner)Mikhail Glushenkov: What's new in Cabal/cabal-install 2.0 — improved new-build, Backpack, foreign libraries and more!http://coldwa.st/e/blog/2017-09-09-Cabal-2-0.html
http://coldwa.st/e/blog/2017-09-09-Cabal-2-0.html
<p>A couple of weeks ago we’ve quietly released versions 2.0 of both Cabal and <code>cabal-install</code> after approximately a year of development. The 2.0 release incorporates more than 1500 commits by <a href="https://gist.github.com/23Skidoo/7109479c43a8de44f8e29fa335e9645c">64 different contributors</a>. This post serves as a formal release announcement and describes what’s new and improved in version 2.0.</p>
<p>There is a number of backwards-incompatible Cabal library API changes in this release that affect packages with Custom setup scripts. Therefore <code>cabal-install</code> will by default use a previous version of Cabal to build setup scripts that don’t <a href="https://www.well-typed.com/blog/2015/07/cabal-setup-deps/">explicitly declare compatibility with Cabal 2.0</a>. The <a href="https://github.com/haskell/cabal/wiki/2.0-migration-guide">2.0 migration guide</a> gives advice for package authors on how to adapt Custom setup scripts to backwards-incompatible changes in this release.</p>
<h2 id="major-new-features">Major new features</h2>
<ul>
<li><p>Much improved <a href="https://www.haskell.org/cabal/users-guide/nix-local-build-overview.html"><code>new-build</code> feature</a> (also known as nix-style local builds), that solves many long-standing problems and is going to become the default mode of operation of <code>cabal-install</code> in version 3.0 (tentative release date: Autumn 2018). Killer features of <code>new-build</code> are reproducible isolated builds with global dependency caching and multi-package projects. For a more extensive introduction to <code>new-build</code>, see <a href="http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/">this blog post by Edward Z. Yang</a>.</p></li>
<li><p>Support for Backpack, a new system for mix-in packages. See <a href="https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst">this article by Edward Z. Yang</a> for an introduction to Backpack and its features.</p></li>
<li><p>Native suport for <a href="https://github.com/haskell/cabal/pull/2540">foreign libraries</a>: Haskell libraries that are intended to be used by non-Haskell code. See <a href="https://www.haskell.org/cabal/users-guide/developing-packages.html#foreign-libraries">this section of the user guide</a> for an introduction to this feature.</p></li>
<li><p>Convenience/internal libraries are now supported (<a href="https://github.com/haskell/cabal/issues/269">#269</a>). An internal library is declared using the stanza <code>library 'libname'</code> and can be only used by other components inside a package.</p></li>
<li><p>Package components can be now built and installed in parallel. This is especially handy when compiling packages with large numbers of independent components (usually those are executables). As a consequence, the <code>Setup.hs</code> command-line interface <a href="https://github.com/ghc-proposals/ghc-proposals/pull/4">now allows to specify the component to be configured</a>.</p></li>
<li><p><a href="https://www.haskell.org/cabal/users-guide/nix-integration.html">Nix package manager integration</a> (<a href="https://github.com/haskell/cabal/issues/3651">#3651</a>).</p></li>
<li><p><a href="https://www.haskell.org/cabal/users-guide/developing-packages.html?highlight=outdated#listing-outdated-dependency-version-bounds">New <code>cabal-install</code> command: <code>outdated</code></a>, for listing outdated version bounds in a <code>.cabal</code> file or a freeze file (<a href="https://github.com/haskell/cabal/issues/4207">#4201</a>). Work on this feature was sponsored by <a href="https://scrive.com/">Scrive AB</a>.</p></li>
<li><p><a href="https://www.haskell.org/cabal/users-guide/installing-packages.html?highlight=reconfigure">New <code>cabal-install</code> command <code>reconfigure</code></a>, which re-runs <code>configure</code> with the most recently used flags (<a href="https://github.com/haskell/cabal/issues/2214">#2214</a>).</p></li>
<li><p>Package repos are now assumed to be <code>hackage-security</code>-enabled by default. If a <code>remote-repo</code> section in <code>~/.cabal/config</code> doesn’t have an explicit <code>secure</code> field, it now defaults to <code>secure: True</code>, unlike in <code>cabal-install</code> 1.24. See <a href="https://www.well-typed.com/blog/2015/04/improving-hackage-security/">this post on the Well-Typed blog</a> for an introduction to <code>hackage-security</code> and what benefits it brings.</p></li>
<li><p>New caret-style version range operator <code>^>=</code> (<a href="https://github.com/haskell/cabal/issues/3705">#3705</a>) that is equivalent to <code>>=</code> intersected with an automatically inferred major upper bound. For example, <code>foo ^>= 1.3.1</code> is equivalent to <code>foo >= 1.3.1 && < 1.4</code>. Besides being a convenient syntax sugar, <code>^>=</code> allows to distinguish “strong” and “weak” upper bounds: <code>foo >= 1.3.1 && < 1.4</code> means “I know for sure that my package doesn’t work with <code>foo-1.4</code>”, while <code>foo ^>= 1.3.1</code> means “I don’t know whether <code>foo-1.4</code>, which is not out yet, will break my package, but I want to be cautious and follow <a href="https://pvp.haskell.org/">PVP</a>”. In the future, this feature will allow to implement automatic version bounds relaxation in a formally sound way (work on this front is progressing on <a href="https://matrix.hackage.haskell.org/"><code>matrix.hackage.haskell.org</code></a>). See <a href="https://www.haskell.org/cabal/users-guide/developing-packages.html?highlight=caret#pkg-field-build-depends">this section of the manual</a> for more information.</p></li>
<li><p>Changed <code>cabal upload</code> to upload a package candidate by default (<a href="https://github.com/haskell/cabal/issues/3419">#3419</a>). Same applies to uploading documentation. Also added a new <code>cabal upload</code> flag <code>--publish</code> for publishing a package on Hackage instead of uploading a candidate (<a href="https://github.com/haskell/cabal/issues/3419">#3419</a>).</p></li>
<li><p><a href="https://www.haskell.org/cabal/users-guide/nix-local-build.html?highlight=allow%20older#cfg-flag---allow-older">Support for <code>--allow-older</code></a> (dual to <code>--allow-newer</code>) (<a href="https://github.com/haskell/cabal/issues/3466">#3466</a>).</p></li>
<li><p><a href="https://www.haskell.org/cabal/users-guide/developing-packages.html?highlight=build%20tool%20depends#pkg-field-build-tool-depends">New <code>build-tool-depends</code> field</a> that replaces <code>build-tools</code> and has a better defined semantics (<a href="https://github.com/haskell/cabal/issues/3708">#3708</a>, <a href="https://github.com/haskell/cabal/issues/1541">#1541</a>). <code>cabal-install</code> will now install required build tools and add them to PATH automatically.</p></li>
<li><p><a href="https://www.haskell.org/cabal/users-guide/developing-packages.html?highlight=autogen%20modules#pkg-field-custom-setup-autogen-modules">New <code>autogen-modules</code> field</a> for automatically generated modules (like <code>Paths_PACKAGENAME</code>) that are not distributed inside the package tarball (<a href="https://github.com/haskell/cabal/issues/3656">#3656</a>).</p></li>
<li><p><a href="https://www.haskell.org/cabal/users-guide/developing-packages.html?highlight=scope#pkg-field-executable-scope">Added a new <code>scope</code> field</a> to the <code>executable</code> stanza (<a href="https://github.com/haskell/cabal/issues/3461">#3461</a>). Executable scope can be either <code>public</code> or <code>private</code>; private executables are those that are expected to be run by other programs rather than users and get installed into <code>$libexecdir/$libexecsubdir</code>. Additionally, <code>$libexecdir</code> now has a subdir structure similar to <code>$lib(sub)dir</code> to allow installing private executables of different packages and package versions alongside one another.</p></li>
<li><p>New <code>--index-state</code> flag for requesting a specific version of the package index (<a href="https://github.com/haskell/cabal/issues/3893">#3893</a>, <a href="https://github.com/haskell/cabal/issues/4115">#4115</a>).</p></li>
<li><p>Added <code>CURRENT_PACKAGE_VERSION</code> CPP constant to <code>cabal_macros.h</code> (<a href="https://github.com/haskell/cabal/issues/4319">#4319</a>).</p></li>
</ul>
<h2 id="minor-improvements-and-bug-fixes">Minor improvements and bug fixes</h2>
<ul>
<li><p>Dropped support for versions of GHC earlier than 6.12 (<a href="https://github.com/haskell/cabal/issues/3111">#3111</a>). Also, GHC compatibility window for the Cabal library has been extended to five years (<a href="https://github.com/haskell/cabal/issues/3838">#3838</a>).</p></li>
<li><p>Added a technical preview version of the ‘cabal doctest’ command (<a href="https://github.com/haskell/cabal/issues/4480">#4480</a>).</p></li>
<li><p>Cabal now invokes GHC with <code>-Wmissing-home-modules</code>, if that flag is supported (added in version 8.2). This means that you’ll get a warning if you forget to list a module in <code>other-modules</code> or <code>exposed-modules</code> (<a href="https://github.com/haskell/cabal/pull/4254">#4254</a>).</p></li>
<li><p>Verbosity <code>-v</code> now takes an extended format which allows specifying exactly what you want to be logged. The format is <code>[silent|normal|verbose|debug] flags</code>, where <code>flags</code> is a space separated list of flags. At the moment, only the flags <code>+callsite</code> and <code>+callstack</code> are supported; these report the call site/stack of a logging output respectively (these are only supported if Cabal is built with GHC 8.0/7.10.2 or greater, respectively).</p></li>
<li><p>The <code>-v/--verbosity</code> option no longer affects GHC verbosity (except in the case of <code>-v0</code>). Use <code>--ghc-options=-v</code> to enable verbose GHC output (<a href="https://github.com/haskell/cabal/issues/3540">#3540</a>, <a href="https://github.com/haskell/cabal/issues/3671">#3671</a>).</p></li>
<li><p>Packages which use internal libraries can result in multiple registrations; thus <code>--gen-pkg-config</code> can now output a directory of registration scripts rather than a single file.</p></li>
<li><p>Changed the default logfile template from <code>.../$pkgid.log</code> to <code>.../$compiler/$libname.log</code> (<a href="https://github.com/haskell/cabal/issues/3807">#3807</a>).</p></li>
<li><p>Macros in ‘cabal_macros.h’ are now <code>#ifndef</code>’d, so that they don’t cause an error if the macro is already defined (<a href="https://github.com/haskell/cabal/issues/3041">#3041</a>).</p></li>
<li><p>Added qualified constraints for setup dependencies. For example, <code>--constraint="setup.bar == 1.0"</code> constrains all setup dependencies on bar, and <code>--constraint="foo:setup.bar == 1.0"</code> constrains foo’s setup dependency on bar (part of <a href="https://github.com/haskell/cabal/issues/3502">#3502</a>).</p></li>
<li><p>Non-qualified constraints, such as –constraint=“bar == 1.0”, now only apply to top-level dependencies. They don’t constrain setup or build-tool dependencies. The new syntax <code>--constraint="any.bar ==1.0"</code> constrains all uses of bar.</p></li>
<li><p>Added a new solver flag, <code>--allow-boot-library-installs</code>, that allows normally non-upgradeable packages like <code>base</code> to be installed or upgraded (<a href="https://github.com/haskell/cabal/issues/4209">#4209</a>). Made the ‘template-haskell’ package non-upgradable again (<a href="https://github.com/haskell/cabal/issues/4185">#4185</a>).</p></li>
<li><p>Fixed password echoing on MinTTY (<a href="https://github.com/haskell/cabal/issues/4128">#4128</a>).</p></li>
<li><p>Added optional solver output visualisation support via the <code>tracetree</code> package (<a href="https://github.com/haskell/cabal/issues/3410">#3410</a>). Mainly intended for debugging.</p></li>
<li><p>New <code>./Setup configure</code> flag <code>--cabal-file</code>, allowing multiple .cabal files in a single directory (<a href="https://github.com/haskell/cabal/issues/3553">#3553</a>). Primarily intended for internal use.</p></li>
<li><p>Removed the <code>--check</code> option from <code>cabal upload</code> (<a href="https://github.com/haskell/cabal/issues/1823">#1823</a>). It was replaced by <a href="https://hackage.haskell.org/upload#candidates">Hackage package candidates</a>.</p></li>
<li><p>Removed the <code>--root-cmd</code> parameter of the ‘install’ command and deprecated <code>cabal install --global</code> (<a href="https://github.com/haskell/cabal/issues/3356">#3356</a>).</p></li>
<li><p>Removed the top-down solver (<a href="https://github.com/haskell/cabal/issues/3598">#3598</a>).</p></li>
<li><p>Cabal no longer supports using a version bound to disambiguate between an internal and external package (<a href="https://github.com/haskell/cabal/issues/4020">#4020</a>). This should not affect many people, as this mode of use already did not work with the dependency solver.</p></li>
<li><p>Miscellaneous minor and/or internal bug fixes and improvements.</p></li>
</ul>
<p>See the full <a href="https://github.com/haskell/cabal/blob/2.0/Cabal/changelog">Cabal 2.0</a> and <a href="https://github.com/haskell/cabal/blob/2.0/cabal-install/changelog"><code>cabal-install</code> 2.0</a> changelogs for the complete list of changes in the 2.0 release.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>Thanks to everyone who contributed code and bug reports. Full list of people who contributed patches to Cabal/<code>cabal-install</code> 2.0 is available <a href="https://gist.github.com/23Skidoo/7109479c43a8de44f8e29fa335e9645c">here</a>.</p>
<h2 id="looking-forward">Looking forward</h2>
<p>We plan to make a new release of Cabal/<code>cabal-install</code> before the end of the year – that is, around December 2017. We want to decouple the Cabal release cycle from the GHC one; that’ll allow us to release a new version of Cabal/<code>cabal-install</code> approximately every six months in the future. Main features that are currently targeted at the 2.2 milestone are:</p>
<ul>
<li><p>Further improvements in <code>new-build</code>, incorporating <a href="https://github.com/haskell/cabal/projects/4">work done by Francesco Gazzetta during HSOC 2017</a>. Currently we are planning to make <code>new-build</code> the default in the 3.0 release (tentative release date: Autumn 2018).</p></li>
<li><p>New Parsec-based parser for <code>.cabal</code> files, joint work by Oleg Grenrus and Duncan Coutts. Faster, less memory hungry, and easier to extend.</p></li>
<li><p><a href="https://github.com/haskell/cabal/issues/4013">A revamped homepage for Cabal</a>, <a href="https://github.com/haskell/cabal/labels/documentation">rewritten user manual</a>, and <a href="https://github.com/haskell/cabal/issues/4637">automated build bots for binary releases</a>. Help in this area would be appreciated!</p></li>
</ul>
<p>We would like to encourage people considering contributing to take a look at <a href="https://github.com/haskell/cabal/issues/">the bug tracker on GitHub</a>, take part in discussions on tickets and pull requests, or submit their own. The bug tracker is reasonably well maintained and it should be relatively clear to new contributors what is in need of attention and which tasks are considered relatively easy. Additionally, <a href="https://github.com/haskell/cabal/wiki/ZuriHac-2017">the list of potential projects from the latest hackathon</a> and the tickets marked <a href="https://github.com/haskell/cabal/issues?q=is%3Aopen+is%3Aissue+label%3A%22meta%3A+easy%22">“easy”</a> and <a href="https://github.com/haskell/cabal/issues?q=is%3Aopen+is%3Aissue+label%3Anewcomer">“newcomer”</a> can be used as a source of ideas for what to work on.</p>
<p>For more in-depth discussion there is also the <a href="https://mail.haskell.org/mailman/listinfo/cabal-devel"><code>cabal-devel</code> mailing list</a> and the <a href="https://wiki.haskell.org/IRC_channel"><code>#hackage</code> IRC channel</a> on FreeNode.</p>Sat, 09 Sep 2017 00:00:00 +0000Functional Jobs: Backend Ruby and Haskell engineer at Health eFilings (Full-time)urn:uuid:b391f677-da0a-89ab-3655-9bc4d973a91d
https://functionaljobs.com/jobs/9034-backend-ruby-and-haskell-engineer-at-health-efilings
<p>Our backend engineering team manages the ingestion and normalization of data sets, from data extraction through to product delivery. We want to work smarter instead of harder, and create domain specific languages, meta-programming etc. where possible.</p>
<p>Our current code base is written in Ruby and Coffee Script, but some new modules are being written in Haskell. You will be on the front lines of creating a Haskell-based infrastructure that is maintainable and can scale to support our needs as we grow.</p>
<p>We currently expect that about 80% of your work will be in Ruby/CoffeeScript, and 20% in Haskell, but that ratio will decrease over time as we move more of our functionality to Haskell. (The faster you can work to migrate functionality to Haskell, the more Haskell you will be doing.)</p>
<p><strong>WHAT WE WILL EXPECT FROM YOU</strong></p>
<p>You will have ownership of an entire module, including responsibility for:</p>
<ul>
<li>Creating new features in a clean and maintainable way</li>
<li>Re-factoring existing code to ensure that we stay agile</li>
<li>Reviewing teammates’ code and providing feedback</li>
<li>Keeping yourself focused and your projects on track</li>
<li>An “I can run through walls” mentality to ensure that goals are met</li>
<li>Answering questions from our implementation team and squashing bugs on a monthly support rotation</li>
</ul>
<p>We are a small team (four engineers), and so it is <em>critical</em> that you be a team player, willing to pitch in and help out your colleagues.</p>
<p><strong>WHAT YOU CAN EXPECT FROM US</strong></p>
<ul>
<li>Autonomy to solve problems in the way you best see fit</li>
<li>A manager who is accountable for ensuring you meet your professional goals</li>
<li>A team who helps each other and always strives to improve</li>
<li>The time to focus on creating the right solution, instead of the easiest one</li>
</ul>
<p><strong>REQUIREMENTS</strong></p>
<ul>
<li>Professional experience as a software engineer</li>
<li>Experience with Haskell and Ruby</li>
<li>A desire for continual self-improvement</li>
<li>An understanding of best practices regarding maintainability and scalability</li>
<li>Must have US work authorization and be located in the US (we cannot sponsor visas at this time)</li>
<li>There are no formal education requirements for this position</li>
</ul>
<p><strong>BONUS POINTS</strong></p>
<ul>
<li>Experience with data scraping and parsing</li>
</ul>
<p><strong>LOCATION</strong></p>
<p>This is expected to be a remote position, although our Madison, Wisconsin office is also available as a work location.</p>
<p>Get information on <a href="https://functionaljobs.com/jobs/9034-backend-ruby-and-haskell-engineer-at-health-efilings">how to apply</a> for this position.</p>Fri, 08 Sep 2017 21:43:31 +0000Dominic Orchard: ICFP / FSCD day 1 – rough noteshttp://dorchard.wordpress.com/?p=393
https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/
<p>(Blog posts for <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/">Day 1</a>, <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/">Day 2</a>, <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/">Day 3</a>, <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/">Day 4</a> (half day))</p>
<p>I decided to take electronic notes at <a href="http://icfp17.sigplan.org/program/program-icfp-2017">ICFP</a> and <a href="http://www.cs.ox.ac.uk/conferences/fscd2017/">FSCD</a> (colocated) this year, and following the example of various people who put their conference notes online (which I’ve found useful), I thought I would attempt the same. However, there is a big caveat: <span style="color: #ff6600;">my notes are going to be partial and may be incorrect;</span> my apologies to the speakers for any mistakes.</p>
<ul>
<li>(ICFP Keynote) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1-keynote">Computational Creativity</a>, Chris Martens</li>
<li>(FSCD Keynote) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#fscd-keynote1">Brzozowski Goes Concurrent</a>, Alexandra Silva</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1-talk1">Faster Coroutine Pipelines</a>, Mike Spivey</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day-talk2">A pretty but not greedy printer (Functional pearl)</a>, Jean-Philippe Bernardy</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1-talk3">Generic Functional Parallel Algorithms: Scan and FFT</a>, Conal Elliott</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1-talk4">A Unified Approach to Solving Seven Programming Problems (Functional Pearl)</a> – William E. Byrd, Michael Ballantyne, Gregory Rosenblatt, Matthew Might</li>
<li>(FSCD) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#fscd-day1pm-talk1">Relating System F and λ2: A Case Study in Coq, Abella and Beluga – </a>Jonas Kaiser, Brigitte Pientka, Gert Smolka</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1pm-talk2">A Framework for Adaptive Differential Privacy</a>, Daniel Winograd-Cort, Andreas Haeberlen, Aaron Roth, Benjamin C. Pierce</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1pm-talk3">Symbolic Conditioning of Arrays in Probabilistic Programs</a>, Praveen Narayanan, Chung-chieh Shan</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1pm-talk4">Abstracting Definitional Interpreters</a>, David Darais, Nicholas Labich, Phúc C. Nguyễn, David Van Horn</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1pm2-talk1">On the Expressive Power of User-Defined Effects: Effect Handlers, Monadic Reflection, Delimited Control</a>, Yannick Forster, Ohad Kammar, Sam Lindley, Matija Pretnar</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1pm2-talk2">Imperative Functional Programs That Explain Their Work</a>, Wilmer Ricciotti, Jan Stolarek, Roly Perera, James Cheney</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/#icfp-day1pm2-talk3">Effect-Driven QuickChecking of Compilers</a>, Jan Midtgaard, Mathias Nygaard Justesen, Patrick Kasting, Flemming Nielson, Hanne Riis Nielson</li>
</ul>
<hr style="height: 3pt;" />
<p>(ICFP keynote #1) <a name="icfp-day1-keynote"></a><strong>Computational Creativity, </strong><em>Chris Martens</em><strong> <a href="https://speakerdeck.com/chrisamaphone/compositional-creativity-icfp-2017">(slides)</a><br />
</strong></p>
<p>In the early days, automated theorem proving by computers was seen as an AI activity. Theorem proving, program synthesis, and AI planning are search procedures but can be viewed as creative procedures.</p>
<p>Linear logic is useful for describing plots (story telling).</p>
<p>There is a difference between what an AI opponent and AI cooperator should do: an opponent need not make intelligible moves, but a cooperator needs to act in a way that is intelligible/understandable to the human player.</p>
<p>Applied Grice’s maxims of conversation to build an “intentional” cooperative player for the Hanabi game. Grice’s maxims are: quantity (not too much or little), quality (tell the truth), relation (be relevant), manner (avoid ambiguity).</p>
<p>Theory of mind: forming mental models of other people’s mental models.</p>
<p>Dynamic epistemic logic, has two indexed modalities:<br />
<img src="https://s0.wp.com/latex.php?latex=%5CBox_a+A&bg=ffffff&fg=000000&s=0" alt="\Box_a A" class="latex" title="\Box_a A" /> (agent ‘a’ believes A)<br />
<img src="https://s0.wp.com/latex.php?latex=%5B%5Calpha%5D+A&bg=ffffff&fg=000000&s=0" alt="[\alpha] A" class="latex" title="[\alpha] A" /> (A holds true after action <img src="https://s0.wp.com/latex.php?latex=%5Calpha&bg=ffffff&fg=000000&s=0" alt="\alpha" class="latex" title="\alpha" />).</p>
<p>Actions are defined inductively:<br />
<img src="https://s0.wp.com/latex.php?latex=%5Cbegin%7Barray%7D%7Brll%7D+%5Calpha%2C+%5Cbeta+%26+%3D+%5C%5C+%26+%5Ctextit%7Bflip%7D+%5C%3B+p+%26+%5Ctext%7B%28change+truth+value%29%7D+%5C%5C+%5Cmid+%26+%3FA+%26+%5Ctext%7B%28precondition%29%7D+%5C%5C+%5Cmid+%26+%5Calpha+%2B+%5Cbeta+%26+%5Ctext%7B%28non+deterministic+choice%29%7D+%5C%5C+%5Cmid+%26+%5Calpha+%3B+%5Cbeta+%C2%A0%26+%5Ctext%7B%28sequence%29%7D+%5C%5C+%5Cmid+%26+%5Calpha%5Ea+%26+%5Ctext%7B%28appearance+to+agent+%27a%27%29%7D+%5C%5C+%5Cmid+%26+%5Calpha%2A+%26+%5Ctext%7B%28public+action%29%7D+%5Cend%7Barray%7D&bg=ffffff&fg=000000&s=0" alt="\begin{array}{rll} \alpha, \beta & = \\ & \textit{flip} \; p & \text{(change truth value)} \\ \mid & ?A & \text{(precondition)} \\ \mid & \alpha + \beta & \text{(non deterministic choice)} \\ \mid & \alpha ; \beta & \text{(sequence)} \\ \mid & \alpha^a & \text{(appearance to agent 'a')} \\ \mid & \alpha* & \text{(public action)} \end{array}" class="latex" title="\begin{array}{rll} \alpha, \beta & = \\ & \textit{flip} \; p & \text{(change truth value)} \\ \mid & ?A & \text{(precondition)} \\ \mid & \alpha + \beta & \text{(non deterministic choice)} \\ \mid & \alpha ; \beta & \text{(sequence)} \\ \mid & \alpha^a & \text{(appearance to agent 'a')} \\ \mid & \alpha* & \text{(public action)} \end{array}" /></p>
<p>Semantics is based on a possible worlds formulation (current actions change future possible worlds). Ostari (Eger & Martens) is a full epistemic logic language that can capture lots of different games with different kinds of knowledge between players.</p>
<p>Key point: <strong>use compositionally </strong>(of various kinds)<strong> as a design principle</strong></p>
<p>Three research challenges:</p>
<ul>
<li>Dynamic logics with functional action languages</li>
<li>Constructive/lambda-calculable DEL</li>
<li>Epistemic session types</li>
</ul>
<hr style="height: 3pt;" />
<p>(FSCD keynote #1)<a name="fscd-keynote1"></a> <strong>Brzozowski Goes Concurrent – </strong><em>Alexandra Silva</em></p>
<p>Ongoing project CoNeCo (Concurrency, Networks, and Coinduction)</p>
<p>SDN – Software Defined Network architectures let you write a program for the network controller. Requires languages for the controller to interact with the underlying switches / routers. Goals of new network PL: raise the level of abstraction beyond the underlying OpenFlow API– make it easier to reason about. Based on Kleene algebra.</p>
<p>NetKat – based on regular expression (Kleene algebra) with tests (KAT) + additional specialized constructions relating to networking. This is compiled to OpenFlow.</p>
<p>Kleene algebra, e.g., <img src="https://s0.wp.com/latex.php?latex=%280+%2B+1%2801%5E%2A0%29%5E%2A1%29%5E%2A&bg=ffffff&fg=000000&s=0" alt="(0 + 1(01^*0)^*1)^*" class="latex" title="(0 + 1(01^*0)^*1)^*" /> – multiples of 3 in binary.</p>
<p>For reasoning about imperative programs, this can be extended with the idea of ‘tests’ (Kozen). Need to capture more of the control flow graph, but there are guards (tests). The solution is to split the underlying alphabet of the Kleene algebra into two: one for actions and one for tests: combine a Kleene algebra and a Boolean algebra (where there is negation), e.g. $\bar{b}$ is the negation of b. Then the three standard control flow constructs of an imperative language are defined:</p>
<p><img src="https://s0.wp.com/latex.php?latex=%5Cbegin%7Barray%7D%7Brl%7D+p+%3B+q+%26+%3D+pq+%5C%5C+%5Ctextbf%7Bif%7D+%5C%3B+b+%5C%3B+%5Ctextbf%7Bthen%7D+%5C%3B+p+%5C%3B+%5Ctextbf%7Belse%7D+%5C%3B+q+%26+%3D+bp+%2B+%5Cbar%7Bb%7Dq+%5C%5C+%5Ctextbf%7Bwhile%7D+%5C%3B+b+%5C%3B+%5Ctextbf%7Bdo%7D+%5C%3B+p+%26+%3D+%28bp%29%5E%2A%5Cbar%7Bb%7D+%5Cend%7Barray%7D&bg=ffffff&fg=000000&s=0" alt="\begin{array}{rl} p ; q & = pq \\ \textbf{if} \; b \; \textbf{then} \; p \; \textbf{else} \; q & = bp + \bar{b}q \\ \textbf{while} \; b \; \textbf{do} \; p & = (bp)^*\bar{b} \end{array}" class="latex" title="\begin{array}{rl} p ; q & = pq \\ \textbf{if} \; b \; \textbf{then} \; p \; \textbf{else} \; q & = bp + \bar{b}q \\ \textbf{while} \; b \; \textbf{do} \; p & = (bp)^*\bar{b} \end{array}" /></p>
<p>Subsumes Hoare logic, i.e.:<br />
<img src="https://s0.wp.com/latex.php?latex=b+%5C%7Bp%5C%7D+c+%5CLeftrightarrow+b%5C%2Cp%5C%2C%5Cbar%7Bc%7D+%3D+0&bg=ffffff&fg=000000&s=0" alt="b \{p\} c \Leftrightarrow b\,p\,\bar{c} = 0" class="latex" title="b \{p\} c \Leftrightarrow b\,p\,\bar{c} = 0" /></p>
<p>Decidable in PSPACE.</p>
<p>What is the minimum number of constructs needed to be added to KAT to deal with packet flow?</p>
<p>A packet is an assignment of constant values n to fields x a packet history is a nonempty sequence of packets. Include assignments (field <- value) and tests on fields (filed = value). e.g.,</p>
<pre style="font-size: 10.5pt; font-family: monospace;">sw = 6; pt = 88; dest <- 10.0.0.1; pt <- 50
</pre>
<p>For all packets incoming on port 88 of switch 6 set the destination IP to 10.0.0.1 and send the packet to port 50.</p>
<p>Can then reason about reachability (can two hosts communicate)? Security (does all untrusted traffic pass through an intrusion detection)? loop detection (packet to be stuck in a forwarding cycle).</p>
<p>Since networks are non-deterministic, this has been adapted to probabilistic NetKAT.</p>
<p>Can we add concurrency to NetKAT (deal with interaction between different components): within this paradigm, can we add concurrency to Kleene algebra? Concurrent Kleene algebra adds a parallel composition || (cf. Hoare 2011). Would like to reason about weak memory model like behaviour (cf. Sewell’s litmus tests). There were developments in concurrent Kleene algebra, but a solid foundations was not yet available.</p>
<p>Have proved a Kleene theorems: equivalence between Kleene algebra and finite automata. In this case, the corresponding automata to a concurrent Kleene algebra is a pomset automata.</p>
<p>Can make Concurrent Kleene algebras into regular subsets of pomsets (useful model of concurrency, cf <a href="http://www.sciencedirect.com/science/article/pii/0304397588901247">Gischer</a>) i.e., <img src="https://s0.wp.com/latex.php?latex=%5B-%5D+%3A+2%5E%7B%5Ctextsf%7BPom%7D_%7B%5CSigma%7D%7D&bg=ffffff&fg=000000&s=0" alt="[-] : 2^{\textsf{Pom}_{\Sigma}}" class="latex" title="[-] : 2^{\textsf{Pom}_{\Sigma}}" />.</p>
<p>Pomset automata- has the idea of splitting into two threads (two paths) that once they reach their accept states, then go onto the next state of the originating state (can be expressed as multiple automata). This reminds me of Milner’s interleaving semantics (for atomic actions) in CCS. Alphabet of the automata is a pomset over some other alphabet of actions.</p>
<p>Showed the (Brzozowski) construction to build an automata from a Concurrent Kleene algebra term, and then how to go back from automata to expressions.</p>
<p>The exchange law captures the interaction of sequential and parallel interaction. Currently working on extending the Kleene theorem in this work to include the exchange law.</p>
<hr style="height: 3pt;" />
<p><a name="icfp-day1-talk1"></a><br />
(ICFP)<strong> Faster Coroutine Pipelines</strong><em>, Mike Spivey</em></p>
<p>(I missed the start and had some trouble catching up).</p>
<p>Coroutine pipelines are a way to structure stream-processing programs. There are libraries for doing this that are based on a term encoding that is then interpreted which suffers serious slow-down when there is process nesting. This paper proposes an alternate approaching using continuation which doesn’t suffer these slow down problems.</p>
<p>The direct style can be mapped to the CPS style which is a monad morphism between two Pipe monads (capturing the coroutine pipelines). To show that the two models are equivalent could be done with logical relations (Mike recommended Relating models of backtracking by Wand and Vaillancourt (2004)).</p>
<p>Termination. Consider a situation:<br />
<img src="https://s0.wp.com/latex.php?latex=p1+%5Cmid%5Cmid+%28%28p2+%5Cmid%5Cmid+p3+%5Cmid%5Cmid+p4%29+%3B+q%29+%5Cmid%5Cmid+p5&bg=ffffff&fg=000000&s=0" alt="p1 \mid\mid ((p2 \mid\mid p3 \mid\mid p4) ; q) \mid\mid p5" class="latex" title="p1 \mid\mid ((p2 \mid\mid p3 \mid\mid p4) ; q) \mid\mid p5" /><br />
where <img src="https://s0.wp.com/latex.php?latex=p3&bg=ffffff&fg=000000&s=0" alt="p3" class="latex" title="p3" /> then terminates.</p>
<p>Need to end up with <img src="https://s0.wp.com/latex.php?latex=p1+%5Cmid%5Cmid+q+%5Cmid%5Cmid+p5&bg=ffffff&fg=000000&s=0" alt="p1 \mid\mid q \mid\mid p5" class="latex" title="p1 \mid\mid q \mid\mid p5" /> not <img src="https://s0.wp.com/latex.php?latex=p1+%5Cmid%5Cmid+%28%28p2+%5Cmid%5Cmid+p4%29+%3B+q%29+%5Cmid%5Cmid+p5&bg=ffffff&fg=000000&s=0" alt="p1 \mid\mid ((p2 \mid\mid p4) ; q) \mid\mid p5" class="latex" title="p1 \mid\mid ((p2 \mid\mid p4) ; q) \mid\mid p5" />. (I wasn’t sure if this was an open question or whether there was a solution in this paper).</p>
<p>Ohad Kammar asked on the ICFP slack whether this relates to<br />
<a href="http://dl.acm.org/citation.cfm?doid=2633357.2633360">Ploeg and Kiselyov’s paper at Haskell 2014 </a>but this didn’t get asked.</p>
<hr style="height: 3pt;" />
<p>(ICFP) <a name="icfp-day-talk2"></a><strong>A pretty but not greedy printer (Functional pearl), </strong><em>Jean-Philippe Bernardy</em></p>
<p>(slides: <a href="https://github.com/jyp/prettiest/blob/master/talk/Outline.org" rel="nofollow">https://github.com/jyp/prettiest/blob/master/talk/Outline.org</a>)</p>
<p>Hughes proposed a pretty printing approach, which was improved upon by Wadler. For example, if you want to pretty print some SExpr but with a maximum column width: when do you spill onto the next line?</p>
<p>Proposes three laws of pretty-printing:<br />
1). Do not print beyond the right margin<br />
2). Do not reveal the structure of the input<br />
3). Use as few lines as possible.</p>
<p>(Hughes breaks 3 and Wadler breaks 2 (I didn’t spot how in the talk)).<br />
These three rules are not compatible with a greedy approach.<br />
This work trades performance for being law-abiding instead.</p>
<p>A small API is provided to write pretty printers (four combinators:<br />
text, flush, (horizontal composition), ).</p>
<p>Phil Wadler asked how you represent <tt>hang</tt> in the library. JP quickly typed up the solution:<br />
<code style="font-size: 10.5pt; font-family: monospace;">hang x y = (x text " " y) (x $$ (text " " y))</code></p>
<p>Phil countered that he could do the same, but JP answered that it didn’t have the expected behaviour. I’m not entirely sure what <code style="font-size: 10.5pt; font-family: monospace;">hang</code> does, but in the Hughes-based library (<a href="https://hackage.haskell.org/package/pretty-1.1.3.5/docs/Text-PrettyPrint.html" rel="nofollow">https://hackage.haskell.org/package/pretty-1.1.3.5/docs/Text-PrettyPrint.html</a>), <code style="font-size: 10.5pt; font-family: monospace;">hang :: Doc -> Int -> Doc -> Doc</code>with behaviour like:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">Prelude Text.PrettyPrint> hang (text "Foo\nBar") 0 (text "Bar")
Foo
Bar Bar
</pre>
<p>(not sure what the extra <tt>Int</tt> argument is for).</p>
<hr style="height: 3pt;" />
<p>(ICFP)<a name="icfp-day1-talk3"></a><strong> Generic Functional Parallel Algorithms: Scan and FFT – </strong><em>Conal Elliott</em></p>
<p>Arrays are the dominant type for parallel programming, but FP uses a variety of data types. Generic programming decomposes data types into fundamental building blocks (sums, products, composition).</p>
<p>Perfect trees can be represented as a n-wise composition of a functor $h$.</p>
<p>Prefix sum (left scan)</p>
<p><img src="https://s0.wp.com/latex.php?latex=b_k+%3D+%5CSigma_%7B1+%5Cleq+i+%5Cleq+k%7D+a_i&bg=ffffff&fg=000000&s=0" alt="b_k = \Sigma_{1 \leq i \leq k} a_i" class="latex" title="b_k = \Sigma_{1 \leq i \leq k} a_i" /><br />
for <img src="https://s0.wp.com/latex.php?latex=k+%3D+1%2C+...%2C+n%2B1&bg=ffffff&fg=000000&s=0" alt="k = 1, ..., n+1" class="latex" title="k = 1, ..., n+1" />, e.g., <tt>scan [1,2,3,4] = [1,3,6,10]</tt><br />
Define more generally as:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">class Functor f => LScan f where
lscan :: Monoid a => f a -> f a x a</pre>
<p>Instances were then given for the components of a datatype generic representation, (constant types, type variable, sum, product, composition) which gives data type generic scan. (Insight: Do left scans on left vectors, not left scans on right vectors! The latter adds a lot of overhead).</p>
<p>Data types were describe numerically, e.g., 2 is a pair, <img src="https://s0.wp.com/latex.php?latex=%5Cstackrel%7B%5Cleftarrow%7D%7B4%7D&bg=ffffff&fg=000000&s=0" alt="\stackrel{\leftarrow}{4}" class="latex" title="\stackrel{\leftarrow}{4}" /> is a 4-vector.</p>
<p>Can do the esame with Discrete Fourier Transform. A nice property is that a<br />
DFT can be factored into separate parts (Johnson 2010, e.g., a 1D DFT of size N = N1*N2 is equivalent to a 2D DFT of size N1xN2– <span style="color: blue;">I couldn’t see this so well on the slide as I was in the overflow room).</span></p>
<hr style="height: 3pt;" />
<p>(ICFP)<a name="icfp-day1-talk4"></a><strong> A Unified Approach to Solving Seven Programming Problems (Functional Pearl) – </strong><em><u>William E. Byrd</u>, Michael Ballantyne, Gregory Rosenblatt, Matthew Might</em></p>
<p>Started off referencing <a href="http://matt.might.net/articles/i-love-you-in-racket/" rel="nofollow">http://matt.might.net/articles/i-love-you-in-racket/</a> – 99 ways to say “I love you” in Racket (all based on different list operations). But what if you wanted more, for some other language? How could you generate this, to say get 1000, or 10000 such examples? The solution put forward here is to using miniKanren, the constraint logic language. The <code style="font-size: 10.5pt; font-family: monospace;">eval</code> function is turned into a relation then can be “run” in reverse, (i.e., <code style="font-size: 10.5pt; font-family: monospace;">run 99 (q) (evalo q '(I love you))</code> gives 99 S-expressions that evaluate to (I love you)). What about for generating quines (programs that evaluate to themselves)? (run 1 (e) (evalo e e)) produced just the <code style="font-size: 10.5pt; font-family: monospace;">0</code> expression and getting three results was just <code style="font-size: 10.5pt; font-family: monospace;">0</code>, <code style="font-size: 10.5pt; font-family: monospace;">#t</code> and <code style="font-size: 10.5pt; font-family: monospace;">#f</code> but with 4 it produced a more interesting quine.</p>
<p>Then, William showed <code style="font-size: 10.5pt; font-family: monospace;">(run n (p q) (eval p q) (evalo q p) (/= q p)</code> generating “twines” (pair of mutually producing programs). This was then applied to a proof checker to turn it into an automated theorem prover! The same technology was used then for program synthesis (although some additional work was needed under the hood to make it fast enough).</p>
<hr style="height: 3pt;" />
<p>(FSCD) <a name="fscd-day1pm-talk1"></a><strong><a>Relating System F and λ2: A Case Study in Coq, Abella and Beluga – </a></strong><em><a href="http://icfp17.sigplan.org/profile/jonaskaiser" class="navigate">Jonas Kaiser</a>, <a href="http://icfp17.sigplan.org/profile/brigittepientka" class="navigate">Brigitte Pientka</a>, <a href="http://icfp17.sigplan.org/profile/gertsmolka" class="navigate">Gert Smolka</a></em></p>
<p>System F (Girard ’72): Two-sorted, types and terms, based on the presentation of System F in Harper’13. Meanwhile, study of CC lead to Pure Type Systems- System F appeared in the Lambda Cube at the corner <img src="https://s0.wp.com/latex.php?latex=%5Clambda2&bg=ffffff&fg=000000&s=0" alt="\lambda2" class="latex" title="\lambda2" />. This work shows the formal relation (that indeed these two systems are equivalent). A proof is partially given in Geuvers ’93, but a full Coq formalisation was given by Kaiser, Tebbi and Smolka at POPL 2017. In this work, the proof is replayed across three tools to provide a useful comparison of the three systems: Coq, Abella and Beluga.</p>
<p>A complication is that the syntax is non-uniform on the System F side: <img src="https://s0.wp.com/latex.php?latex=%5CPi+x+%3A+a+.+b&bg=ffffff&fg=000000&s=0" alt="\Pi x : a . b" class="latex" title="\Pi x : a . b" /> in PTS can correspond to <img src="https://s0.wp.com/latex.php?latex=A+%5Crightarrow+B&bg=ffffff&fg=000000&s=0" alt="A \rightarrow B" class="latex" title="A \rightarrow B" /> and <img src="https://s0.wp.com/latex.php?latex=%5Cforall+X+.+B&bg=ffffff&fg=000000&s=0" alt="\forall X . B" class="latex" title="\forall X . B" />. Have to also keep track of context assumptions. One (of many) “topics of interest” in this comparison is how to manage contexts (which is interesting to me as I am quite interested in CMTT in Beluga).</p>
<p>The approaches were:</p>
<ul>
<li>Coq – <em>first-order de Bruijn indices</em>, parallel substitutions (<code style="font-size: 10.5pt; font-family: monospace;">Autosubst</code> library), and invariants (e.g, a context can be extended with a new term variable). Inductive predicate for typing judgments. Traversal of binders requires context adjustments.</li>
<li>Abella – <em>HOAS, nominal quantification</em> (fresh names handled internally), relation proof search [Abella has two layers: specification and logic]. Contexts represented by lists (as in Coq). A compound inductive predicate is define to relate the different context representations (and to keep them all consistent).</li>
<li>Beluga – <em>HOAS, 1st class contexts (via Contextual Modal Type Theory)</em> and context schemas. Object K (e.g., terms, types, deriviations) are alway paired with a 1-st class context <img src="https://s0.wp.com/latex.php?latex=%5CGamma&bg=ffffff&fg=000000&s=0" alt="\Gamma" class="latex" title="\Gamma" /> that gives it meaning <img src="https://s0.wp.com/latex.php?latex=%5B%5CGamma+%5Cvdash+K%5D&bg=ffffff&fg=000000&s=0" alt="[\Gamma \vdash K]" class="latex" title="[\Gamma \vdash K]" />. There is no concept of free variable., e.g., in Coq <img src="https://s0.wp.com/latex.php?latex=0+%5Cvdash+0_%7Bty%7D+%5Crightarrow+0_%7Bty%7D+%5Ctextsf%7Bty%7D+%5CRightarrow+%5Cbot&bg=ffffff&fg=000000&s=0" alt="0 \vdash 0_{ty} \rightarrow 0_{ty} \textsf{ty} \Rightarrow \bot" class="latex" title="0 \vdash 0_{ty} \rightarrow 0_{ty} \textsf{ty} \Rightarrow \bot" /> is provable but in Belluga <img src="https://s0.wp.com/latex.php?latex=%5Cbullet+%5Cvdash+0&bg=ffffff&fg=000000&s=0" alt="\bullet \vdash 0" class="latex" title="\bullet \vdash 0" /> is <span style="text-decoration: underline;">not even well formed</span> (accessing the first variable 0 (think de Bruijns) in an empty context). Context schemas provide a way to give the structure of the typed contexts (as a dependent record).</li>
</ul>
<hr style="height: 3pt;" />
<p>(ICFP)<strong><a name="icfp-day1pm-talk2"></a><a> A Framework for Adaptive Differential Privacy – </a></strong><span style="color: #000000;"><em><a href="http://icfp17.sigplan.org/profile/danielwinogradcort1" class="navigate">Daniel Winograd-Cort</a>, <a href="http://icfp17.sigplan.org/profile/andreashaeberlen" class="navigate">Andreas Haeberlen</a>, <a href="http://icfp17.sigplan.org/profile/aaronroth" class="navigate">Aaron Roth</a>, <a href="http://icfp17.sigplan.org/profile/benjamincpierce" class="navigate">Benjamin C. Pierce</a></em></span></p>
<p>Associated GitHub: <a href="https://github.com/dwincort/AdaptiveFuzz">https://github.com/dwincort/AdaptiveFuzz</a></p>
<p>They created a new language, Adaptive Fuzz for adaptive differential privacy (DP), uses Fuzz type checking to verify individual pieces in a piecewise static way.</p>
<p>DFuzz is a dependently typed version of Fuzz, can we use this? Privacy cost dependent on feature selection: which could depend on values in the databases: but this is practically useless as we don’t know what is in the database. We can still get static guarantees if it is done piece-wise in two modes: <strong>data</strong> mode (access to sensitive data, uses the Fuzz type checker to provide DP proofs) and <strong>adaptive </strong>mode (computation between pieces). End up with “light” dependent types.</p>
<p>These two modes have the same syntax though (code it one works in the other). Here is an example type for a function that scales one number by the other (multiplication by abs):</p>
<p><img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bscale%7D+%5C%3B+%28c+%3A+%5Cmathbb%7BR%7D%29+%5C%3B+%28n+%3A+%5B%5Ctextsf%7Babs%7D+%5C%3B+c%5D+%5Cmathbb%7BR%7D%29+%3A+%5Cmathbb%7BR%7D&bg=ffffff&fg=000000&s=0" alt="\textsf{scale} \; (c : \mathbb{R}) \; (n : [\textsf{abs} \; c] \mathbb{R}) : \mathbb{R}" class="latex" title="\textsf{scale} \; (c : \mathbb{R}) \; (n : [\textsf{abs} \; c] \mathbb{R}) : \mathbb{R}" /></p>
<p>(scale the second number by the first argument). In adaptive mode the annotations are ignored, in data mode the first argument is forced to be a constant (partial evaluation happens under the hood so we don’t need dependent types really).</p>
<p>General scheme is: analyst (program) writes data mode code, it is partially evaluated and the Fuzz type checker works out a cost, which is the compared against a privacy filter (<span style="color: #0000ff;">where/how is this defined?</span>) Can write a gradient descent algorithm with different levels of randomness provided by different privacy budgets (how much information do we want to reveal at the end, e.g., if budget is infinite then you get not noise in the result). (another example in the paper is feature selection).</p>
<p>This seems like a really interesting mix of a static and semi-static approach (via partial evaluation). I wonder how this relates to fine-grained cost-tracking type systems?</p>
<hr style="height: 4pt; color: #333;" />
<p>(ICFP) – <a name="icfp-day1pm-talk3"></a><b>Symbolic Conditioning of Arrays in Probabilistic Programs –<br />
</b><em>Praveen Narayanan, Chung-chieh Shan</em></p>
<p><em>(</em>tag line:<em> Bayesian inference via program transformation).</em><br />
Say you are benchmarking some iterative code, which you model by <img src="https://s0.wp.com/latex.php?latex=a+n+%2B+b&bg=ffffff&fg=000000&s=0" alt="a n + b" class="latex" title="a n + b" /> for some startup cost <img src="https://s0.wp.com/latex.php?latex=b&bg=ffffff&fg=000000&s=0" alt="b" class="latex" title="b" /> and <img src="https://s0.wp.com/latex.php?latex=n&bg=ffffff&fg=000000&s=0" alt="n" class="latex" title="n" /> iterations and <img src="https://s0.wp.com/latex.php?latex=a&bg=ffffff&fg=000000&s=0" alt="a" class="latex" title="a" /> a cost for the loop. You might then collect some results and do linear regression (best fist line) to see if this model is confirmed. Bayesian linear regression gives you many lines for different kinds of fit.</p>
<ol>
<li>Write a generative model, e.g., some normally distributed <img src="https://s0.wp.com/latex.php?latex=n&bg=ffffff&fg=000000&s=0" alt="n" class="latex" title="n" /> to generate a set of possible lines (normally distributed gradients), to which we maybe then add some additional normally distributed noise for each point. (k measurements)</li>
<li>Observe running times</li>
<li>Infer distribution of lines. Delete those candidates from the generative model that weren’t observed (from step 2), keep the ones that were.</li>
</ol>
<p>Bayesian inference is compositional: you can build it from different components, by treating distributions as programs (e.g., a program <code style="font-size: 10.5pt; font-family: monospace;">normal (a * n_1 + b)</code> is gives us one of the lines in the generative model) and then transforming the program. A program transformation implements the model-theoretic notion of “disintegration” but the existing approach for this is problematic (I wasn’t quite clear on how). A new transformation approach is proposed which provides a smaller output program via some additional combinators which replace an unrolled set of <img src="https://s0.wp.com/latex.php?latex=k&bg=ffffff&fg=000000&s=0" alt="k" class="latex" title="k" /> observations into a replication like combination (<code style="font-size: 10.5pt; font-family: monospace;">plate</code>).</p>
<div>
<hr style="height: 4pt; color: #333;" />
<p>(ICFP) <a name="icfp-day1pm-talk4"></a><strong><a>Abstracting Definitional Interpreters – </a></strong><em><a>David Darais, Nicholas Labich, Phúc C. Nguyễn, David Van Horn</a></em></p>
</div>
<p>Definitional interpreters: programs that take ASTs and run them, giving an executable semantics. Key challenge: a parameterised interpreter that recovers both concrete and abstract semantics.</p>
<p>A concrete interpreter for the lambda calculus: <code style="font-size: 10.5pt; font-family: monospace;">exp -> m (val)</code> where <code style="font-size: 10.5pt; font-family: monospace;">m</code> is a monad which (at least) captures the environment and the store (which is used for representing the variable environment). The definition is written non-recursive “unfixed” (unrolled), which can be fixed (e.g., apply to Y combinator) later to run the interpreter. By leaving it unfixed, we can interpose different behaviour in the recursive cases.</p>
<p>To make the interpreter abstract, abstract the primitive operations and types, e.g., integers are extended with an ‘abstract’ value, i.e., <img src="https://s0.wp.com/latex.php?latex=%5Cmathbb%7BZ%7D+%5Ccup+%5C%7B%5Ctexttt%7B%27N%7D%5C%7D&bg=ffffff&fg=000000&s=0" alt="\mathbb{Z} \cup \{\texttt{'N}\}" class="latex" title="\mathbb{Z} \cup \{\texttt{'N}\}" /> where <code style="font-size: 10.5pt; font-family: monospace;">'N</code> represents the abstract integer. The semantics is then main non-deterministic so that an <code style="font-size: 10.5pt; font-family: monospace;">iszero?</code> predicate on <code style="font-size: 10.5pt; font-family: monospace;">'N</code> splits into two execution traces. But its partial (could run forever).</p>
<p>How to make it total (always terminating). Look for states that it has already seen before, sufficient for termination but unsound for abstraction. If you observe <code style="font-size: 10.5pt; font-family: monospace;">fact 'N</code> then one branch fails (non-terminating as visit previous state) and the <code style="font-size: 10.5pt; font-family: monospace;">1</code> branch succeeds, so the result is just <code style="font-size: 10.5pt; font-family: monospace;">1</code> (not ideal). Instead, look in a “cache” of precomputed values to return something like <code style="font-size: 10.5pt; font-family: monospace;">[[fact 'N]] = {1} U {'N x $[fact 'N]}</code>. Intercept recursion points to evaluate through a cache and to stop when a previously computed value is hit (sounds space intensive?).</p>
<hr style="height: 3pt; color: #333;" />
<p>(ICFP) – <a name="icfp-day1pm2-talk1"></a><strong>On the Expressive Power of User-Defined Effects: Effect Handlers, Monadic Reflection, Delimited Control – </strong><em>Yannick Forster, Ohad Kammar, Sam Lindley, Matija Pretnar</em></p>
<p>Say you want to write a program like:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">toggle = { x <- get!
y <- not! x
put! y
x }</pre>
<p>If we don’t have effects, then we can do explicit state passing, but you have to do a full rewrite. Ideally want only local transformations. (Ohad showed this in three different styles, see below).</p>
<p>Relative expressiveness in language design, compare/contrast: (1 – Eff) algebraic effects and handlers (2 – Mon) monads (3 – Del) delimited control. Did this by extending CBPV in these three directions (and formalised this in Abella) and defining macro translations between every pair of these extended languages. Expressivity is stated as formal translations between calculi. Then this was considering in the typed setting. Interestingly there is no typed macro translation from Eff to Del nor from Eff to Mon.</p>
<p>[There is a large design space which yields lots of different languages, this study is a start. Inexpressivity is brittle: adding other language features can change the result.]</p>
<p>In the Eff version, we give standard state handlers to implement <code style="font-size: 10.5pt; font-family: monospace;">get</code> and <code style="font-size: 10.5pt; font-family: monospace;">put</code>. Its type contains the effect-system like information <code style="font-size: 10.5pt; font-family: monospace;">State = {get : 1 -> bit, put : bit -> 1}</code>where <code style="font-size: 10.5pt; font-family: monospace;">toggle : U_State F bit.</code></p>
<p>In the Mon version, monadic reflection is used with a state monad.<br />
(Q: <code style="font-size: 10.5pt; font-family: monospace;">put</code> was of type <code style="font-size: 10.5pt; font-family: monospace;">U (bit -> F)</code> but I thought the monad would be <code style="font-size: 10.5pt; font-family: monospace;">T = U F</code>?)</p>
<p>(Q: In the translation from Mon -> Eff, how do generate the effect type if the source program only uses <code style="font-size: 10.5pt; font-family: monospace;">put</code> or <code style="font-size: 10.5pt; font-family: monospace;">get</code> ? I guess it just maps to the set of all effect operations).</p>
<p>(Q: the talked showed just simple state effects; are the results shown for any other kind of effect).</p>
<hr style="height: 4pt;" />
<p>(ICFP) – <a name="icfp-day1pm2-talk2"></a><strong>Imperative Functional Programs That Explain Their Work</strong> – <em>Wilmer Ricciotti, Jan Stolarek, Roly Perera, James Cheney</em></p>
<p><a href="https://github.com/jstolarek/slicer" rel="nofollow">https://github.com/jstolarek/slicer</a></p>
<p>Slicing can be used to explain particular (parts of) a program output: which parts of the source code contributed to a value. Extend TML (Transparent ML) to Imperative TML.</p>
<p>Construct backwards and forward slicing as a Galois connection (between two lattices). The lattices are based on the idea of definedness: where ‘holes’ make programs more undefined, e.g., <img src="https://s0.wp.com/latex.php?latex=1+%2B+2&bg=ffffff&fg=000000&s=0" alt="1 + 2" class="latex" title="1 + 2" /> is above <img src="https://s0.wp.com/latex.php?latex=1+%2B+%5CBox&bg=ffffff&fg=000000&s=0" alt="1 + \Box" class="latex" title="1 + \Box" /> and <img src="https://s0.wp.com/latex.php?latex=%5Cbox+%2B+2&bg=ffffff&fg=000000&s=0" alt="\box + 2" class="latex" title="\box + 2" /> (partial expressions). Forward slicing preserves meets in the lattice; backward slicing should be consistent and minimal with respect to forward slicing (<span style="color: #0000ff;">I missed what this meant exactly, but I talked to Janek later to clarify: consistency is <img src="https://s0.wp.com/latex.php?latex=x+%5Cleq+%5Ctextsf%7Bfwd%7D%28%5Ctextsf%7Bbwd%7D%28x%29%29&bg=ffffff&fg=000000&s=0" alt="x \leq \textsf{fwd}(\textsf{bwd}(x))" class="latex" title="x \leq \textsf{fwd}(\textsf{bwd}(x))" /></span>). The idea then is that we have <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bbwd%7D+%3A+values_%5CBox+%5Crightarrow+expr_%5CBox&bg=ffffff&fg=000000&s=0" alt="\textsf{bwd} : values_\Box \rightarrow expr_\Box" class="latex" title="\textsf{bwd} : values_\Box \rightarrow expr_\Box" /> and <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bfwd%7D+%3A+expr_%5CBox+%5Crightarrow+values_%5CBox&bg=ffffff&fg=000000&s=0" alt="\textsf{fwd} : expr_\Box \rightarrow values_\Box" class="latex" title="\textsf{fwd} : expr_\Box \rightarrow values_\Box" /> which form a Galois connection, i.e., <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bbwd%7D%28x%29+%5Cleq+y+%C2%A0%5CLeftrightarrow+x+%5Cleq+%5Ctextsf%7Bfwd%7D%28y%29&bg=ffffff&fg=000000&s=0" alt="\textsf{bwd}(x) \leq y \Leftrightarrow x \leq \textsf{fwd}(y)" class="latex" title="\textsf{bwd}(x) \leq y \Leftrightarrow x \leq \textsf{fwd}(y)" />. [Given one of the slicings we don’t necessarily get the other for free].</p>
<p>Given a small program <code style="font-size: 10.5pt; font-family: monospace;">l1 := 0; (!l1, !l2)</code> a partial version of it <code style="font-size: 10.5pt; font-family: monospace;"><img src="https://s0.wp.com/latex.php?latex=%5CBox&bg=ffffff&fg=000000&s=0" alt="\Box" class="latex" title="\Box" />; (!l1, !l)</code> and an initial store <code style="font-size: 10.5pt; font-family: monospace;">[l1 -> 1, l2 -> 2]</code> should forward slice to <code style="font-size: 10.5pt; font-family: monospace;">(<img src="https://s0.wp.com/latex.php?latex=%5CBox&bg=ffffff&fg=000000&s=0" alt="\Box" class="latex" title="\Box" />, 2)</code> by propagating the “hole” through the store.</p>
<hr style="height: 3pt;" />
<p>(ICFP) <a name="icfp-day1pm2-talk3"></a><strong>Effect-Driven QuickChecking of Compilers – </strong><em>Jan Midtgaard, Mathias Nygaard Justesen, Patrick Kasting, Flemming Nielson, Hanne Riis Nielson</em></p>
<p>They made a generator for writing OCaml programs: useful for automated testing, e.g., comparing two compilers against each other (compile -> run -> diff, any difference will be suspicious).</p>
<p>Generated a program <code style="font-size: 10.5pt; font-family: monospace;">let k = (let i = print_newline () in fun q -> fun i -> "") () in 0</code> which found a bug: different behaviour between the native code generator and the bytecode generator due to effects being delayed (indefinitely). Another example was found (related), due to effects being removed.</p>
<p>Program generation, how to do it:</p>
<ul>
<li>Try: generate arbitrary strings?! Most won’t lex or parse).</li>
<li>Instead, follow the grammar (see Celento et al ’80). Most won’t type check!</li>
<li>Instead, follow the simply-typed lambda calculus (Palka et al. ’11) by bottom-up reading of a typing relation. This will make it through the type checker (I guess you can generalise the type rules).</li>
</ul>
<p>Observed behaviour in OCaml depends on evaluation order (when effects are involved). OCaml bytecode uses right-to-left but sometimes native code backend uses left-to-right (hm!) But the specification is itself a bit loose. Can we avoid generating such programs since they are not well specified for anyway?</p>
<ul>
<li>Instead, following a type-and-effect systems which has one bit for marking pure/effectful (boolean lattice effect system). Extend to a pair of bits, where the second bit is order dependence (extend the effect algebra a bit more). Our goal is then a program which may have effects but may not have order-dependent effects.</li>
</ul>
<p>Have an effect-dependent soundness theorem (that the effect bits correctly anticipate the effects).</p>
<p>Next, there is a type preserving shrinker to reduce huge ugly generated programs into a minimal (much nicer) example. e.g., replace a pure integer expression by a literal <code style="font-size: 10.5pt; font-family: monospace;">0</code> and large literals by small literals, replace beta-redexes by let bindings, etc. Shrunk tests are checked whether they still show a difference.</p>
<p>A question was asked whether the effect-based mechanism could be used to rule-out other allowed difference, e.g., in floating point. Jan answered that floating point isn’t generated anyway, but an interesting idea.</p>
<p>Another question: how do you avoid generating non-terminating programs? Jan explained that since it is STLC (+ extension) only total terms are generated.</p><br /> <a href="http://feeds.wordpress.com/1.0/gocomments/dorchard.wordpress.com/393/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/dorchard.wordpress.com/393/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=dorchard.wordpress.com&blog=10180011&post=393&subd=dorchard&ref=&feed=1" alt="" height="1" border="0" width="1" />Fri, 08 Sep 2017 13:05:35 +0000Dominic Orchard: FSCD day 4 – rough noteshttp://dorchard.wordpress.com/?p=711
https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/
<p>(Blog posts for <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/">Day 1</a>, <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/">Day 2</a>, <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/">Day 3</a>, <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/">Day 4</a> (half day))</p>
<p>I decided to take electronic notes at <a href="http://icfp17.sigplan.org/program/program-icfp-2017">ICFP</a> and <a href="http://www.cs.ox.ac.uk/conferences/fscd2017/">FSCD</a> (colocated) this year, and following the example of various people who put their conference notes online (which I’ve found useful), I thought I would attempt the same. However, there is a big caveat: <span style="color: #ff6600;">my notes are going to be partial and may be incorrect</span>; my apologies to the speakers for any mistakes.</p>
<ul>
<li>(FSCD Keynote #3) <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/#fscd-day4-keynote">Type systems for the relational verification of higher order programs</a>, Marco Gaboardi</li>
<li>(FSCD) <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/#fscd-day4-talk1">Arrays and References in Resource Aware ML</a>, Benjamin Lichtman, Jan Hoffmann</li>
<li>(FSCD) <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/#fscd-day4-talk2">The Complexity of Principal Inhabitation</a>, Andrej Dudenhefner, Jakob Rehof</li>
<li>(FSCD) <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/#fscd-day4-talk3">Types as Resources for Classical Natural Deduction</a>, Delia Kesner, Pierre Vial</li>
</ul>
<hr />
<p><a name="fscd-day4-keynote"></a>(FSCD Keynote #3) Type systems for the relational verification of higher order programs, Marco Gaboardi</p>
<p>Relational properties <img src="https://s0.wp.com/latex.php?latex=R%28X_1%2C+X_2%29+%5CRightarrow+S%28P%28X_1%29%2C+P%28X_2%29%29&bg=ffffff&fg=000000&s=0" alt="R(X_1, X_2) \Rightarrow S(P(X_1), P(X_2))" class="latex" title="R(X_1, X_2) \Rightarrow S(P(X_1), P(X_2))" />. For example, take <img src="https://s0.wp.com/latex.php?latex=R&bg=ffffff&fg=000000&s=0" alt="R" class="latex" title="R" /> and <img src="https://s0.wp.com/latex.php?latex=S&bg=ffffff&fg=000000&s=0" alt="S" class="latex" title="S" /> to be notions of program equivalence (equivalent inputs produce equivalent outputs). Another relation might be in information-flow security where relations <img src="https://s0.wp.com/latex.php?latex=R%2C+S&bg=ffffff&fg=000000&s=0" alt="R, S" class="latex" title="R, S" /> mean “these two programs are low-level equivalent” (low-security).<br />
Another is differential privacy where <img src="https://s0.wp.com/latex.php?latex=R&bg=ffffff&fg=000000&s=0" alt="R" class="latex" title="R" /> means two programs differ in one individual data point and <img src="https://s0.wp.com/latex.php?latex=S%28Y_%21%2C+Y_%40%29+%3D+Pr%5BY_1%5D+%5Cleq+e%5E%5Cepsilon+Pr%5BY_2%5D&bg=ffffff&fg=000000&s=0" alt="S(Y_!, Y_@) = Pr[Y_1] \leq e^\epsilon Pr[Y_2]" class="latex" title="S(Y_!, Y_@) = Pr[Y_1] \leq e^\epsilon Pr[Y_2]" />.</p>
<p>In relational cost analysis, we want to compute the difference in cost (the relative cost) between the two programs (may depend on the input and the underlying relation), e.g., <img src="https://s0.wp.com/latex.php?latex=cost%28P%29-cost%28Q%29+%5Cleq+f%28X_1%2C+X_2%2C+R%2C+S%29&bg=ffffff&fg=000000&s=0" alt="cost(P)-cost(Q) \leq f(X_1, X_2, R, S)" class="latex" title="cost(P)-cost(Q) \leq f(X_1, X_2, R, S)" /> (giving an upper bound computed in terms of the inputs and underlying relations). This is useful for guaranteeing compile optimisations (not worse, or definitely better), ruling out side-channel attacks (i.e., the <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bcost%7D%28e%5Bv_1%2Fx%5D%29+-+%5Ctextsf%7Bcost%7D%28e%5Bv_2%2Fx%5D%29+%3D+0&bg=ffffff&fg=000000&s=0" alt="\textsf{cost}(e[v_1/x]) - \textsf{cost}(e[v_2/x]) = 0" class="latex" title="\textsf{cost}(e[v_1/x]) - \textsf{cost}(e[v_2/x]) = 0" />, such that different inputs do not yield different costs thus information about the inputs is not leaked).</p>
<p>Motivating example 1: find in a list of lists (2D data structure). We want to prove that one implementation of this algorithm is faster than the other (I’ve omitted the code that Marco showed us, which was expressed via a common higher-order function with two different parameter functions to give the two implementations).<br />
Motivating example 2: prove a precise upper bound on the relative cost of insertion sort between two lists (different by <img src="https://s0.wp.com/latex.php?latex=n&bg=ffffff&fg=000000&s=0" alt="n" class="latex" title="n" /> in length).</p>
<p>[Clarkson, Schneider ’08] – formalises the idea of ‘hyperproperties’. Properties are sets of traces, hyperproperties are sets of sets of traces. Relation verification is a set of pair of traces (a 2-property). They show how to reduce verification of hyperproperties into the verification of properties. Using these kind of results requires encodings, they do not reflect the relational nature of the property, and they do not reflect the connection of the relational reasoning to the programs syntax (see the talk from ICFP day 2, on A Relational Logic for Higher-order Programs). Lots of previous work (going back to Abadi’s System R).</p>
<p>Relational typing: <img src="https://s0.wp.com/latex.php?latex=%5CGamma+%5Cvdash+t_1+%5Capprox+t_2+%3A+%5Ctau&bg=ffffff&fg=000000&s=0" alt="\Gamma \vdash t_1 \approx t_2 : \tau" class="latex" title="\Gamma \vdash t_1 \approx t_2 : \tau" />. Talk about relational properties of the input and relational properties of the output. Usually, if we interpret to relations then we want to prove soundness as: <img src="https://s0.wp.com/latex.php?latex=%28%5Cbar%7Bv_1%7D%2C+%5Cbar%7Bv_2%7D%29+%5Cin+%5B%5CGamma%5D+%5CRightarrow+%28%5Bt_1%5B%5Cbar%7Bv_1%7D%2F%5CGamma%5D%5D%2C+t_2%5B%5Cbar%7Bv_2%7D%2FGamma%5D%5D%29+%5Cin+%5B%5Ctau%5D&bg=ffffff&fg=000000&s=0" alt="(\bar{v_1}, \bar{v_2}) \in [\Gamma] \Rightarrow ([t_1[\bar{v_1}/\Gamma]], t_2[\bar{v_2}/Gamma]]) \in [\tau]" class="latex" title="(\bar{v_1}, \bar{v_2}) \in [\Gamma] \Rightarrow ([t_1[\bar{v_1}/\Gamma]], t_2[\bar{v_2}/Gamma]]) \in [\tau]" />.\<br />
How can we reason about the cost? Two approaches from here: (1) lightweight typing (bottom up) extending type system with the necessary constructors, (2) heavyweight typing (top down) use a powerful type system and encode the needed constructions. (In the future, we want to find a balance between the two approaches).</p>
<p>Approach (1) (lightweight) called Relcost a relational refinement type-and-effect system. The idea is to take advantage of structural similarities between programs and inputs as much as possible. There will be two sorts of type (unary and relational). Typing judgments <img src="https://s0.wp.com/latex.php?latex=%5COmega+%5Cvdash%5EU_L+t+%3A+A&bg=ffffff&fg=000000&s=0" alt="\Omega \vdash^U_L t : A" class="latex" title="\Omega \vdash^U_L t : A" /> where <img src="https://s0.wp.com/latex.php?latex=U&bg=ffffff&fg=000000&s=0" alt="U" class="latex" title="U" /> is the upper bound and <img src="https://s0.wp.com/latex.php?latex=L&bg=ffffff&fg=000000&s=0" alt="L" class="latex" title="L" /> is the lower bound execution cost, which are thought of as a unary effect. A relational judgement <img src="https://s0.wp.com/latex.php?latex=%5CGamma+%5Cvdash+t_1+%5Cominus+t_2+%5Cpreceq+D+%3A+%5Ctau&bg=ffffff&fg=000000&s=0" alt="\Gamma \vdash t_1 \ominus t_2 \preceq D : \tau" class="latex" title="\Gamma \vdash t_1 \ominus t_2 \preceq D : \tau" /> where <img src="https://s0.wp.com/latex.php?latex=D&bg=ffffff&fg=000000&s=0" alt="D" class="latex" title="D" /> is the upper bound on the relative cost of the two programs. The types have an annotation function type (like in normal effect systems) <img src="https://s0.wp.com/latex.php?latex=%5Csigma+%5Cxrightarrow%7BU%2C+L%7D+%5Ctau&bg=ffffff&fg=000000&s=0" alt="\sigma \xrightarrow{U, L} \tau" class="latex" title="\sigma \xrightarrow{U, L} \tau" /> and data types for integers and lists have indices to express dependencies.</p>
<p>e.g. <img src="https://s0.wp.com/latex.php?latex=n+%3A+%5Ctextsf%7Bint%7D%2C+f+%3A+%5Ctextsf%7Bint%7D+%5Cxrightarrow%7Bk%2C+k%7D+%5Ctextsf%7Bint%7D+%5Cvdash%5E%7Bk%2B1%7D_1+%5Ctextsf%7Bif%7D+%5C%2C+n+%5Cleq+0+%5Ctextsf%7Bthen%7D+f+%5C%3B+n+%5Ctextsf%7Belse%7D+%5C%3B+1+%3A+%5Ctextsf%7Bint%7D&bg=ffffff&fg=000000&s=0" alt="n : \textsf{int}, f : \textsf{int} \xrightarrow{k, k} \textsf{int} \vdash^{k+1}_1 \textsf{if} \, n \leq 0 \textsf{then} f \; n \textsf{else} \; 1 : \textsf{int}" class="latex" title="n : \textsf{int}, f : \textsf{int} \xrightarrow{k, k} \textsf{int} \vdash^{k+1}_1 \textsf{if} \, n \leq 0 \textsf{then} f \; n \textsf{else} \; 1 : \textsf{int}" />.<br />
But if we know that <img src="https://s0.wp.com/latex.php?latex=n&bg=ffffff&fg=000000&s=0" alt="n" class="latex" title="n" /> is known to be <img src="https://s0.wp.com/latex.php?latex=5&bg=ffffff&fg=000000&s=0" alt="5" class="latex" title="5" /> this can be refined (with the type <img src="https://s0.wp.com/latex.php?latex=n+%3A+%5Ctextsf%7Bint%7D%5B5%5D&bg=ffffff&fg=000000&s=0" alt="n : \textsf{int}[5]" class="latex" title="n : \textsf{int}[5]" /> in the indices, which will change the bound to $\vdash^{k+1}_{k+1}$.</p>
<p>The relational types are a little different. There is still a latent effect on function types, but is now a single relative cost <img src="https://s0.wp.com/latex.php?latex=%5Csigma+%5Cxrightarrow%7BD%7D+%5Ctau&bg=ffffff&fg=000000&s=0" alt="\sigma \xrightarrow{D} \tau" class="latex" title="\sigma \xrightarrow{D} \tau" />.<br />
Judgements are like the following, which says equal integers have zero cost: <img src="https://s0.wp.com/latex.php?latex=%5CGamma+%5Cvdash+n+%5Cominus+n+%5Cpreceq+0+%3A+%5Ctextsf%7Bint%7D_r&bg=ffffff&fg=000000&s=0" alt="\Gamma \vdash n \ominus n \preceq 0 : \textsf{int}_r" class="latex" title="\Gamma \vdash n \ominus n \preceq 0 : \textsf{int}_r" />.</p>
<p>The unary cost and relational cost are connected as follows:<br />
<img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%7C%5CGamma%7C+%5Cvdash%5EU_%7B-%7D+t_1+%3A+A+%5Cqquad+%7C%5CGamma%7C+%5Cvdash%5E%7B-%7D_L+t_2+%3A+A%7D%7B%5CGamma+%5Cvdash+t_1+%5Cominus+t_2+%5Cpreceq+U+-+L+%3A+%5Ctextsf%7BU%7D+A%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{|\Gamma| \vdash^U_{-} t_1 : A \qquad |\Gamma| \vdash^{-}_L t_2 : A}{\Gamma \vdash t_1 \ominus t_2 \preceq U - L : \textsf{U} A}" class="latex" title="\dfrac{|\Gamma| \vdash^U_{-} t_1 : A \qquad |\Gamma| \vdash^{-}_L t_2 : A}{\Gamma \vdash t_1 \ominus t_2 \preceq U - L : \textsf{U} A}" /><br />
Thus, we can drop into the unary cost to compute a worst and best run time individual, and then combine these into a relative cost.<br />
There is a box modality which captures terms which have the same cost and allow a reseting therefore of the differential cost, e.g., if we have identical terms then:<br />
<img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma+%5Cvdash+t+%5Cominus+t+%5Cpreceq+D+%3A+%5Ctau+%5Cqquad+%5Cforall+x+.+%5CGamma%28x%29+%5Csqsubseteq+%5CBox+%5CGamma%28x%29%7D%7B%5CGamma+%5Cvdash+t+%5Cominus+t+%5Cpreceq+0+%3A+%5CBox+%5Ctau%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma \vdash t \ominus t \preceq D : \tau \qquad \forall x . \Gamma(x) \sqsubseteq \Box \Gamma(x)}{\Gamma \vdash t \ominus t \preceq 0 : \Box \tau}" class="latex" title="\dfrac{\Gamma \vdash t \ominus t \preceq D : \tau \qquad \forall x . \Gamma(x) \sqsubseteq \Box \Gamma(x)}{\Gamma \vdash t \ominus t \preceq 0 : \Box \tau}" />.<br />
The type <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7BList%7D%5E%5Calpha%7BI%7D+%5Ctau&bg=ffffff&fg=000000&s=0" alt="\textsf{List}^\alpha{I} \tau" class="latex" title="\textsf{List}^\alpha{I} \tau" /> captures lists which differ in less than <img src="https://s0.wp.com/latex.php?latex=%5Calpha&bg=ffffff&fg=000000&s=0" alt="\alpha" class="latex" title="\alpha" /> positions. This interacts with the <img src="https://s0.wp.com/latex.php?latex=%5CBox&bg=ffffff&fg=000000&s=0" alt="\Box" class="latex" title="\Box" /> type in order to capture the notion of differing numbers of elements in the list.<br />
In the semantics, the relative cost is pushed inside the semantics (some step-indexed logical relations are used, details not shown).<br />
Going back to the earlier example of searching nested lists, we see a typing of <img src="https://s0.wp.com/latex.php?latex=1+%5Cleq+%7C%5Ctexttt%7Bfind1%7D%7C+%5Cleq+3n&bg=ffffff&fg=000000&s=0" alt="1 \leq |\texttt{find1}| \leq 3n" class="latex" title="1 \leq |\texttt{find1}| \leq 3n" /> and <img src="https://s0.wp.com/latex.php?latex=3n+%5Cleq+%7C%5Ctexttt%7Bfind2%7D%7C+%5Cleq+4n&bg=ffffff&fg=000000&s=0" alt="3n \leq |\texttt{find2}| \leq 4n" class="latex" title="3n \leq |\texttt{find2}| \leq 4n" /> meaning <img src="https://s0.wp.com/latex.php?latex=%5Ctexttt%7Bfind1%7D&bg=ffffff&fg=000000&s=0" alt="\texttt{find1}" class="latex" title="\texttt{find1}" /> has a relatively smaller cost. Plugging this into the higher-order function and doing the type derivation again gives a looser bound.</p>
<p>Approach (2) is the heavyweight approach was presented earlier at ICFP (A Relational Logic for Higher-order Programs), the language RHOL (Relational Higher Order Logic). Has relational and unary judgments, where the relational rule is <img src="https://s0.wp.com/latex.php?latex=%5CGamma+%5Cmid+%5CPsi+%5Cvdash+t_1+%3A+%5Ctau_1+%5Capprox+t_2+%3A+%5Ctau_2+%5Cmid+%5Ctheta&bg=ffffff&fg=000000&s=0" alt="\Gamma \mid \Psi \vdash t_1 : \tau_1 \approx t_2 : \tau_2 \mid \theta" class="latex" title="\Gamma \mid \Psi \vdash t_1 : \tau_1 \approx t_2 : \tau_2 \mid \theta" /> which contains term variable assertions <img src="https://s0.wp.com/latex.php?latex=%5CPsi&bg=ffffff&fg=000000&s=0" alt="\Psi" class="latex" title="\Psi" /> and relational assertion <img src="https://s0.wp.com/latex.php?latex=%5Ctheta&bg=ffffff&fg=000000&s=0" alt="\theta" class="latex" title="\theta" />. In the type system, some rules are synchronous (building up the related two terms with the same syntactic construct) or asynchronous (building up one of the terms).</p>
<p>HOL terms can be encoded into RHOL, and vice versa (a bit like the hyperproperty result of Clarkson/Schneider), but RHOL is much easier to work with for relational reasoning but also all the power of HOL can be embedded into RHOL.<br />
The system <img src="https://s0.wp.com/latex.php?latex=R%5EC&bg=ffffff&fg=000000&s=0" alt="R^C" class="latex" title="R^C" /> is a monadic meta language with a specific cost effect (with <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bcstep%7D_n%28m%29&bg=ffffff&fg=000000&s=0" alt="\textsf{cstep}_n(m)" class="latex" title="\textsf{cstep}_n(m)" /> introducing a cost. Intuition: if a term <img src="https://s0.wp.com/latex.php?latex=m&bg=ffffff&fg=000000&s=0" alt="m" class="latex" title="m" /> is closed and $m \Downarrow_n v$ then $m \cong \textsf{cstep}_n(v)$.We can define formulae in HOL which all us to reason about these explicit cost terms and their quantities: we can define what we need in the logic.</p>
<p>For computing a relative upper bound on the insertion sort, we want to prove <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7BUnsortedDiff%7D%28l_1%2C+l_2%2C+n%29+%5CRightarrow+%5Ctextsf%7Bcost%7D%28%5Ctextit%7Bisort%7D+%5C%2C+l_1%29+-%C2%A0%5Ctextsf%7Bcost%7D%28%5Ctextit%7Bisort%7D+%5C%2C+l_2%29+%5Cleq+n&bg=ffffff&fg=000000&s=0" alt="\textsf{UnsortedDiff}(l_1, l_2, n) \Rightarrow \textsf{cost}(\textit{isort} \, l_1) - \textsf{cost}(\textit{isort} \, l_2) \leq n" class="latex" title="\textsf{UnsortedDiff}(l_1, l_2, n) \Rightarrow \textsf{cost}(\textit{isort} \, l_1) - \textsf{cost}(\textit{isort} \, l_2) \leq n" />. Using the system , we can prove this property, using a suitable invariant/predicate over the input and output list(s).</p>
<p>Take home: type-based relational verification is a large research space that needs more work.</p>
<p>Q: what about recursion? In the first system, letrec is included straightforwardly, in the second system the assumption is terminating (general) recursion.<br />
Q: are the costs tied to a specific evaluation strategy? Yes, but one can encode different ones in the system. (Both systems are parametric in the way you count cost). In the first system, this is part of the typing, in the second, this comes where you put monadic cost operations <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bcstep%7D&bg=ffffff&fg=000000&s=0" alt="\textsf{cstep}" class="latex" title="\textsf{cstep}" />.</p>
<hr style="height: 4pt; color: #333;" />
<p>(FSCD) <a name="fscd-day4-talk1"></a><strong>Arrays and References in Resource Aware ML</strong>, <span style="text-decoration: underline;">Benjamin Lichtman</span>, Jan Hoffmann</p>
<p>Resource-aware ML (RAML) – model resource usages of programs with cost semantics, e.g., implement Quicksort and it produce the bound <img src="https://s0.wp.com/latex.php?latex=14n%5E2+%2B+19%5En+%2B+c&bg=ffffff&fg=000000&s=0" alt="14n^2 + 19^n + c" class="latex" title="14n^2 + 19^n + c" /> (I forgot what the constant c was). This work introduces references and arrays to enable analysis of programs where resource consumption depends on the data stored in mutable structures (references and arrays).</p>
<p>How does it work? Automatic amortized resource analysis. Assign “potential” functions to a data structures, which will pay the resource consumption cost, then hopefully there is some left over for later. This propagates backwards to give the overall upper bound cost.</p>
<p>Each variable in scope may contribute (carry) potential, e.g., recursive data types (such as <img src="https://s0.wp.com/latex.php?latex=x+%3A+%5Ctextsf%7Blist%7D%5E2%28%5Ctextsf%7Bint%7D%29&bg=ffffff&fg=000000&s=0" alt="x : \textsf{list}^2(\textsf{int})" class="latex" title="x : \textsf{list}^2(\textsf{int})" /> that is it can contribute <img src="https://s0.wp.com/latex.php?latex=2n&bg=ffffff&fg=000000&s=0" alt="2n" class="latex" title="2n" /> units of potential where <img src="https://s0.wp.com/latex.php?latex=n&bg=ffffff&fg=000000&s=0" alt="n" class="latex" title="n" /> is the length of the list.</p>
<p>Based on an affine type system (all variable used at most once). Subexpressions are bound to variables whenever possible (share-let normal form) in order to explicitly track when things are evaluated (via variables). When a variable carrying potential needs to be shared, the potential can be explicitly split <code style="font-size: 10.5pt; font-family: monospace;">share x as (x1, x2) in e</code> in order to use a variable more than once (gives you contraction). This splits the potential of <img src="https://s0.wp.com/latex.php?latex=x&bg=ffffff&fg=000000&s=0" alt="x" class="latex" title="x" /> across the two variables. Benjamin walked us through a nice example with appending to a list.</p>
<p>Now let’s add references.</p>
<pre style="font-size: 10.5pt; font-family: monospace; padding: 5px;">g l =
let t = ref l in
share r as (r1, r2) in -- r2 : L^1 (int)
let _ = h r1 in -- h : (L^q int) ref -> int
-- (need to make the type of this stronger)
append (!r2, []) -- append : L^1(int) * L^0(int) -> L^0(int)</pre>
<p>One approach is to carry around extra information about possible effects, but they want to keep effects out of the type system because they seem them as cumbersome and difficult to use. Instead, strengthen contracts by require that potential annotations of data inside references are fixed.</p>
<pre style="font-size: 10.5pt; font-family: monospace; padding: 5px;">g l =
let t = ref l in
share r as (r1, r2) in -- the restricted h means r1 and r2 have
-- to have the same type
let _ = h r1 in -- h : (L^1 int) ref -> int
append (!r2, [])</pre>
<p>Introduce a <code style="font-size: 10.5pt; font-family: monospace;">swap</code> command as the only way to update a cell, that behaves like:</p>
<pre style="font-size: 10.5pt; font-family: monospace; padding: 5px;">swap (r, l) = let x = !r in (let _ = r := l in x)</pre>
<p>Requires that data placed into a cell has the same type as data being taken out, ensures that potential annotations of mutable cells never change (the only way to update a cell). Thus, swap is the well-typed version of dereference in situations where you need to work with potential. Thus, require the previous program to use swap on <code style="font-size: 10.5pt; font-family: monospace;">r2</code> with an empty list instead of a direct dereference in the last line.</p>
<p>The paper has a detailed example of depth-first search (stateful) which shows more of the details related to how to work with state and potentials. This also utilises an idea of “wrapper” types.</p>
<p>Introduce a notion of “memory typing”, which tracks all memory locations pointed to by mutual references, this is require to prove the soundness of the cost analysis (based on the free potential of the system, the potential of the context, and the potential of each mutable memory reference).</p>
<p>Mutable arrays were then a natural extension from this basis, with an array-swap operation for updating an array cell.</p>
<p>In the future, they’d like to create some syntactic sugar so that the programming style is not affected as much by having to instead swap, aswap, and wrapper types. Would like to investigate how fixed potential annotations restrict the inhabitants of a type (<span style="color: #000080;">Yes, yes, this is very interesting</span>!)</p>
<p>I asked about what happens if you write down Landin’s Knot to get recursion via higher-order state: Benjamin answered that functions don’t get given potential so storing and updating functions in a reference would type correctly. I still wonder if this would let me get the correct potential bounds for a recursive algorithm (like DFS) if the recursion was implemented via higher-order state.</p>
<hr style="height: 4pt; color: #333;" />
<p>(FSCD) <a name="fscd-day4-talk2"></a><b>The Complexity of Principal Inhabitation</b>, Andrej Dudenhefner, Jakob Rehof</p>
<p>In the simply-typed lambda calculus (STLC), with principle types. We say <img src="https://s0.wp.com/latex.php?latex=%5Ctau&bg=ffffff&fg=000000&s=0" alt="\tau" class="latex" title="\tau" /> is the principle type of <img src="https://s0.wp.com/latex.php?latex=M&bg=ffffff&fg=000000&s=0" alt="M" class="latex" title="M" /> if <img src="https://s0.wp.com/latex.php?latex=%5Cvdash+M+%3A+%5Ctau&bg=ffffff&fg=000000&s=0" alt="\vdash M : \tau" class="latex" title="\vdash M : \tau" /> and for all types <img src="https://s0.wp.com/latex.php?latex=%5Csigma&bg=ffffff&fg=000000&s=0" alt="\sigma" class="latex" title="\sigma" /> such that <img src="https://s0.wp.com/latex.php?latex=M+%3A+%5Csigma&bg=ffffff&fg=000000&s=0" alt="M : \sigma" class="latex" title="M : \sigma" /> there exists a substitution <img src="https://s0.wp.com/latex.php?latex=S&bg=ffffff&fg=000000&s=0" alt="S" class="latex" title="S" /> such that <img src="https://s0.wp.com/latex.php?latex=S%28%5Ctau%29+%3D+%5Csigma&bg=ffffff&fg=000000&s=0" alt="S(\tau) = \sigma" class="latex" title="S(\tau) = \sigma" />.</p>
<p>Definition: (Normal Principal Inhabitant), We say that a term M in beta-normal form is a normal principal inhabitant of a type t, if t is the principal type of M.</p>
<p>This work shows that principal inhabitation for STLC is PSPACE-complete (in 1979 it was shown that inhabitation (not considering) principality is PSPACE-complete). Thus, this work seeks to answer the following: since principality is a global property of derivations, does it increase the complexity of inhabitation? This is also practically interesting for type-based program synthesis, since a normal principal inhabitant of a type is a natural implementation of the type.</p>
<p>For example, <img src="https://s0.wp.com/latex.php?latex=a+%5Crightarrow+a+%5Crightarrow+a&bg=ffffff&fg=000000&s=0" alt="a \rightarrow a \rightarrow a" class="latex" title="a \rightarrow a \rightarrow a" /> is inhabited by the K combinator <img src="https://s0.wp.com/latex.php?latex=K+%3D+%5Clambda+x+.+%5Clambda+y+.+%C2%A0x&bg=ffffff&fg=000000&s=0" alt="K = \lambda x . \lambda y . x" class="latex" title="K = \lambda x . \lambda y . x" /> but is not principally inhabited (<span style="color: #333399;">I haven’t understood why yet</span>).</p>
<p>The proof is based on the subformula calculus. The subformula calculus provides paths to index types (as trees) which are strings of either 1 or 2. This is useful for reasonable about subformulae, where the usual STLC rules are re-expressed in terms of type paths, e.g., <img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma%2C+x+%3A+1%5Cpi+%5Cvdash+e+%3A+2%5Cpi%7D%7B%5CGamma+%5Cvdash+%5Clambda+x+.+e+%3A+%5Cpi%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma, x : 1\pi \vdash e : 2\pi}{\Gamma \vdash \lambda x . e : \pi}" class="latex" title="\dfrac{\Gamma, x : 1\pi \vdash e : 2\pi}{\Gamma \vdash \lambda x . e : \pi}" />. Define relations on paths, <img src="https://s0.wp.com/latex.php?latex=R_M&bg=ffffff&fg=000000&s=0" alt="R_M" class="latex" title="R_M" /> as the minimal equivalence relation for a beta-normal term <img src="https://s0.wp.com/latex.php?latex=M&bg=ffffff&fg=000000&s=0" alt="M" class="latex" title="M" /> (capture constraints on subformulae of terms) and <img src="https://s0.wp.com/latex.php?latex=R_%5Ctau&bg=ffffff&fg=000000&s=0" alt="R_\tau" class="latex" title="R_\tau" /> which defines an equivalence relation on paths of paths with common subpaths (capture constraints on subformulae on types).</p>
<p>(I had to step out, so missed the end).</p>
<hr style="height: 4pt; color: #444;" />
<p>(FSCD) <a name="fscd-day4-talk3"></a><strong>Types as Resources for Classical Natural Deduction</strong>, Delia Kesner, Pierre Vial</p>
<p>Quantitative types seen as resources (can’t be duplicated); they provide simple arithmetical arguments to prove operational equivalences. These ideas are extended to classical logic in this talk.</p>
<p>In simple types, typability implies normalisation. With intersection types this is an equivalence, i.e., normalising also implies typability. In an intersection type system, a variable can be assigned several types, e.g. <img src="https://s0.wp.com/latex.php?latex=x+%3A+A+%5Cwedge+B+%5Cwedge+C+%5Cwedge+B&bg=ffffff&fg=000000&s=0" alt="x : A \wedge B \wedge C \wedge B" class="latex" title="x : A \wedge B \wedge C \wedge B" /> (note the two occurrences of <img src="https://s0.wp.com/latex.php?latex=B&bg=ffffff&fg=000000&s=0" alt="B" class="latex" title="B" />) where intersection is associative, commutative and (possibly) idempotent. In 1994, Gardner introduces a system which is not idempotent, where types are now multisets (rather than sets in the earlier formulations). This has the flavour of linear logic resources (<span style="color: #333399;">my interpretation: the types can then capture how many times a variable is used (quantitative) if we contraction takes the intersection of the two types and intersection is non-idempotent, e.g., <img src="https://s0.wp.com/latex.php?latex=x_1+%3A+A+%5Cwedge+B%2C+x_2+%3A+B+%5Cwedge+C+%5Cleadsto+x+%3A+A+%5Cwedge+B+%5Cwedge+B+%5Cwedge+C&bg=ffffff&fg=000000&s=0" alt="x_1 : A \wedge B, x_2 : B \wedge C \leadsto x : A \wedge B \wedge B \wedge C" class="latex" title="x_1 : A \wedge B, x_2 : B \wedge C \leadsto x : A \wedge B \wedge B \wedge C" />)</span>.</p>
<p>In the literature, we see how to build a computational interpretation of classical natural deduction: Intuitionistic Logic + Peirce’s law gives classical logic and Felleisen’s call-cc operator gives the computational interpretation. Relatedly, the lambda_mu calculus (Parigot 92) give a direct interpretation of classical natural deduction.</p>
<p>Types are strict where intersection can only appear on the left-hand side of a function arrow.</p>
<p>Interestingly, you can have a situation where the type of a lambda abstraction has an empty intersection of types in its source type. In order to get normalisation in this context, the system was extended a bit to deal with this (I didn’t capture this part, see paper). The full system had rules to ensure this didn’t happen.</p>
<p>A notion of “weighted” subject reduction is defined, where the size of a derivation tree is strictly decreasing during reduction.</p><br /> <a href="http://feeds.wordpress.com/1.0/gocomments/dorchard.wordpress.com/711/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/dorchard.wordpress.com/711/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=dorchard.wordpress.com&blog=10180011&post=711&subd=dorchard&ref=&feed=1" alt="" height="1" border="0" width="1" />Thu, 07 Sep 2017 10:58:33 +0000Dominic Orchard: ICFP / FSCD day 2 – rough noteshttp://dorchard.wordpress.com/?p=491
https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/
<p>(Blog posts for <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/">Day 1</a>, <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/">Day 2</a>, <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/">Day 3</a>, <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/">Day 4</a> (half day))</p>
<p>I decided to take electronic notes at <a href="http://icfp17.sigplan.org/program/program-icfp-2017">ICFP</a> and <a href="http://www.cs.ox.ac.uk/conferences/fscd2017/">FSCD</a> (colocated) this year, and following the example of various people who put their conference notes online (which I’ve found useful), I thought I would attempt the same. However, there is a big caveat: <span style="color: #ff6600;">my notes are going to be partial and may be incorrect;</span> my apologies to the speakers for any mistakes.</p>
<ul>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2-0">ICFP day 2 keynote – Challenges in Assuring AI</a>, John Launchbury</li>
<li>(FSCD) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#fscd-day2-keynote">FSCD day 2 keynote – Uniform resource analysis by rewriting: strengths and weakeness</a>, Georg Moser</li>
<li>(FSCD) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#fscd-day2-talk1">Continuation Passing Style for Effect Handlers</a>, Daniel Hillerström, Sam Lindley, Bob Atkey, KC Sivaramakrishnan</li>
<li>(ICFP <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2pm-1">How to Prove Your Calculus Is Decidable: Practical Applications of Second-Order Algebraic Theories and Computation</a>, Makoto Hamana</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2pm-2">A Relational Logic for Higher-Order Programs</a>, Alejandro Aguirre, Gilles Barthe, Marco Gaboardi, Deepak Garg, Pierre-Yves Strub</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2pm-3">Foundations of Strong Call by Need</a>, Thibaut Balabonski, Pablo Barenbaum, Eduardo Bonelli, Delia Kesner</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2pm-4">No-Brainer CPS Conversion</a>, Milo Davis, William Meehan, Olin Shivers</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2-pm2-talk1">Compiling to Categories</a>, Conal Elliott</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2-pm2-talk2">Visitors Unchained</a>, François Pottier</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/#icfp-day2-pm2-talk3">Staged Generic Programming</a>, Jeremy Yallop</li>
</ul>
<hr style="height: 5pt; color: #444;" />
<p><a name="icfp-day2-0"></a>(ICFP Keynote #2) – <strong>Challenges in Assuring AI</strong>, John Launchbury</p>
<p>The moment we think we’ve got our (verification) hands round the world’s systems (e.g., compilers), a whole new set of systems appear: AI.</p>
<p>The dual of “have I tested enough?” is “how good is my model?” (in a verification-based approach). What is the model we have for AI systems?</p>
<p>What is intelligence? Information processing. But there are different kinds of information processing ability; asking if a system is ‘intelligent’ is too coarse/binary. John breaks this down into ‘perceiving’ P, ‘learning’ L, ‘abstracting’ A (create new meanings), and ‘reasoning’ R (plan and decide).</p>
<p><strong>AI wave 1</strong> : rule-based, hand-crafted knowledge: P 1, L 0, A 0, R 3 (poor handling of uncertainty, no learning). Type systems are a little like this (perceiving, didn’t have to tell it everything). Can do some pretty amazing stuff: see smart security systems that analysis code and patch bugs.</p>
<p><strong>AI wave 2</strong>: statistical learning. Specific problem domains, train them on big data. Still requires a lot of engineering work (to get the right models and statistical techniques for the task). P 3, L 3, A 1, R 1 (less good at reasoning).</p>
<p>What does it mean to prove a vision system correct?</p>
<p><span style="text-decoration: underline;"><em>Manifold hypothesis</em></span><strong> – </strong>data coming in is high-dimensional but the features you are interested in form lower-dimensional structures. E.g., with cars, lots of data comes in but understanding the data means separating the manifolds (of low dimensionality).<br />
Another example, a 28×28 image is a 784 dimensional space (as in 784 numbers, 784-length vector). Variation in handwritten digits form 10 distinct manifolds within the 784 dimensional space. High-degree of coherence between all data samples for a particular digit. (<em><span style="color: #808080;">“Manifold” here may be slightly more informally used than the full mathematical definition</span>).</em></p>
<p>Imagine two interlocking spiral arms which are really 1-dimensional manifolds in a 2-dimensional space. By continuous transforms on the 2-dimensional space these can morphed into linearly separable components. Sometimes stretching into a new dimension enables enclosed manifolds to be separated.</p>
<p>Neural nets separate manifolds; each layer of a neural network stretches and squashes the data space until the manifolds are cleanly separated. Non-linear functions at each step (otherwise multiple linear transformations would collapse into just one layer). Use calculus to adjust the weights (error propagation backwards through the layers).</p>
<p>Structured neural nets have different kinds of layers, e.g., feature maps (which perform a local analysis over the whole input); generated potentially via convolutions (i.e., turn a 28×28 image into 20 24×24 images), do some subsampling. Programming these requires some trial-and-error.</p>
<p><span style="text-decoration: underline;"><em>Models to explain decisions</em></span>: “why did you classify this image as a cat?”; not just “I ran it through my calculations and ‘cat’ came up as highest”. A deep neural net could produce a set of words (fairly abstract) and then a language-generating recurrent neural net (RNN) translates these into captions (sentences). Currently, statistically impressive but individual unreliable.</p>
<p>Adversarial perturbations: Panda + <1% computed distortion, becomes classified as a Gibbon. Whilst the two images are indistinguishable to a human, the noise interferes with the manifold separation. There are even universal adversarial perturbations that throw-off classification of all images.</p>
<p>Assured control code (e.g., in a helicopter): a physical machine is modelled (system dynamics, power, inertia), a control system design is created, from which controller code is generated. But this does not produce a very nice to fly system. Need some kind of refinement back to the control code. Given the correct structure for the control code framework then the fine tuning from feedback on the actual system is very effective. This can be overlayed with different goals (different fitness functions), e.g., safety over learning.</p>
<p><strong>AI wave 3: </strong>combining wave 1 and 2 techniques. Perceive into a contextual model on which abstractions are computed and which can be tuned (learning) to produce reasoning. Aim to get a better intelligence profile: P 3, L 3, A 2, R 3. Hand assurance arguments on the contextual model part instead.<br />
e.g., build models that are specialised based on our knowledge, e.g., we know the probably number of strokes in each digit 0-9 and its trajectory. The generative model generates explanation of how a test character was generated (go from the model of whats its looking for to what it sees; how likely is the thing seen generatable from the model of a particular digit).</p>
<p>Need to be able to specify intent in a variety of ways; all real world specifications are partial and humans can’t always communicate the full intent of what they want (cf., trying to explain how to drive a car).</p>
<p>What are we trying to assure? Mistakes in sensing/reasoning leading to undesirable actions, undesirable emergent behaviours, hackable systems being subverted, misalignment between human/machine values.</p>
<p>(AlphaGo is a wave3-like system as it combined the neural net-based classification of moves with traditional planning/reasoning/tree-based pruning/search).</p>
<hr style="height: 3pt; color: #333;" />
<p><a name="fscd-day2-keynote"></a><br />
(FSCD Keynote #2) – <strong>Uniform Resource Analysis by Rewriting: </strong><strong>Strengths and Weaknesses, </strong>George Moser</p>
<p>Can we use complexity measures on term rewriting systems to make complexity arguments about higher-order functions?</p>
<p>The main scheme is to convert a program via a <em>complexity reflecting transformation</em> into a Term Rewriting System (TRS), then do automated resource analysis on the TRSS to get an asymptotic bound. The first half of the talk is some background on doing complexity analysis (<span style="color: #999999;">which was a bit fast for me so my notes are a bit incoherent below</span>). There are some really nice complexity analysis tools for programs coming out of this work (<span style="color: #999999;">links near the end</span>).</p>
<p>Some preliminaries: Example, set up list reverse as a finite set of rewrite rules (a Term Rewriting System, TRS):</p>
<pre style="font-size: 10.5pt; font-family: monospace;">rev(xs) -> rev'(xs, nil)
rev'(nil, acc) -> acc
rev'(x :: xs, acc) -> rev'(xs, x :: acc)</pre>
<p>A computation of this set of rules is the application of the rules from left to right. The rules are non-overlapping.</p>
<p>Can see rewriting as an abstraction of functional programming, or can see it as part of equational reasoning (universal algebra); but this really generalises functional programming (function symbols can appear nested on the left-hand side).</p>
<p><strong>Definition: </strong>A TRS is terminating if the rewriting relation is “well-founded”.<br />
<strong>Definition: </strong>A function symbol f is “defined” if it is the root symbol on the left of a rule (other f is a constructor).<br />
<strong>Definition: </strong>Runtime complexity wrt. a terminating TRS is defined:</p>
<p><img src="https://s0.wp.com/latex.php?latex=dh%28t%29+%3D+max+%5C%7B+n+%5Cmid+exists+u+.+t+%5Crightarrow%5En+u%5C%7D&bg=ffffff&fg=000000&s=0" alt="dh(t) = max \{ n \mid exists u . t \rightarrow^n u\}" class="latex" title="dh(t) = max \{ n \mid exists u . t \rightarrow^n u\}" /><br />
<img src="https://s0.wp.com/latex.php?latex=rc%28n%29+%3D+max+%5C%7B+dh%28t%29+%5Cmid+%5Cmathsf%7Bsize%7D%28t%29+%5Cleq+n+%5Cwedge+%5Ctextit%7Bn+and+t+are+%22basic%22%7D+%5C%7D&bg=ffffff&fg=000000&s=0" alt="rc(n) = max \{ dh(t) \mid \mathsf{size}(t) \leq n \wedge \textit{n and t are "basic"} \}" class="latex" title="rc(n) = max \{ dh(t) \mid \mathsf{size}(t) \leq n \wedge \textit{n and t are "basic"} \}" />.</p>
<p>RC is the runtime complexity; Q: is this a “natural notion” for rewriting? (<span style="color: #999999;">I don’t know how one defined natural here</span>).<br />
Derivational complexity has no restriction on the terms,<br />
<img src="https://s0.wp.com/latex.php?latex=dc%28n%29+%3D+max+%5C%7B+dh%28t%29+%5Cmid+%5Ctextsf%7Bsize%7D%28t%29+%5Cleq+n%5C%7D&bg=ffffff&fg=000000&s=0" alt="dc(n) = max \{ dh(t) \mid \textsf{size}(t) \leq n\}" class="latex" title="dc(n) = max \{ dh(t) \mid \textsf{size}(t) \leq n\}" /> This has been used mainly to make termination arguments.<br />
See “Termination proofs and the length of derivations” (Hofbauer, Lautemann, 1989).</p>
<p><strong>Definition</strong>: Multiset Path Order. A precedence order <img src="https://s0.wp.com/latex.php?latex=%3E&bg=ffffff&fg=000000&s=0" alt=">" class="latex" title=">" /> induces a multiset path order <img src="https://s0.wp.com/latex.php?latex=%3E_%7Bmpo%7D&bg=ffffff&fg=000000&s=0" alt=">_{mpo}" class="latex" title=">_{mpo}" />, (<span style="color: #999999;">whose definition I’m not going to transcribe, <a href="https://en.wikipedia.org/wiki/Path_ordering_(term_rewriting)">Wikipedia has a definition</a></span>) It basically says that <img src="https://s0.wp.com/latex.php?latex=s+%3D+f%28s_1%2C+...%2C+s_n%29+%3E+t+%3D+g%28t_1%2C...%2Ct_m%29&bg=ffffff&fg=000000&s=0" alt="s = f(s_1, ..., s_n) > t = g(t_1,...,t_m)" class="latex" title="s = f(s_1, ..., s_n) > t = g(t_1,...,t_m)" /> if <img src="https://s0.wp.com/latex.php?latex=s&bg=ffffff&fg=000000&s=0" alt="s" class="latex" title="s" /> is <img src="https://s0.wp.com/latex.php?latex=%3E&bg=ffffff&fg=000000&s=0" alt=">" class="latex" title=">" /> all of <img src="https://s0.wp.com/latex.php?latex=t_i&bg=ffffff&fg=000000&s=0" alt="t_i" class="latex" title="t_i" />, or <img src="https://s0.wp.com/latex.php?latex=s&bg=ffffff&fg=000000&s=0" alt="s" class="latex" title="s" /> is <img src="https://s0.wp.com/latex.php?latex=%5Cgeq&bg=ffffff&fg=000000&s=0" alt="\geq" class="latex" title="\geq" /> some <img src="https://s0.wp.com/latex.php?latex=t_i&bg=ffffff&fg=000000&s=0" alt="t_i" class="latex" title="t_i" />.</p>
<p>The Hydra Battle- is this terminating? (Dershowitz and Jouannaud designed the TRS as a termination problem, later rectified by Dershowitz): The beast is a finite tree, each leaf corresponds to a head. Hercules chops off heads of the Hydra, but the Hydra regrows:</p>
<ul>
<li>If the cut head has a pre-predecessor (grandmother) then the remaining subtree is multiplied by the stage of the game (heads regrowing and multiplying).</li>
<li>Otherwise, nothing happens (the head doesn’t regrow)</li>
</ul>
<p>Can show it is terminating by an inductive argument over transfinite numbers. (but complexity is quite bad). See <a href="https://link.springer.com/content/pdf/10.1007%2Fs00200-009-0094-4.pdf">“The Hydra battle and Cicho’s principle” (Moser, 2009). </a></p>
<p>The RTA list of open problem was mentioned: <a href="http://www.win.tue.nl/rtaloop/">http://www.win.tue.nl/rtaloop/</a></p>
<p>TRSes can be represented via a matrix interpretation (<span style="color: #999999;">details were too fast for me</span>).<br />
These techniques were developed into a fully automated complexity analysis tool for TRSes called <strong>tct-trs</strong> (modular complexity analysis framework).</p>
<p>Going back to the original goal: can we use this for higher-order programs? Consider the textbook example of <code style="font-size: 10.5pt; font-family: monospace;">reverse</code> defined by a left-fold. Need to do a complexity preserving conversion into a TRS (complexity preserving transformation ensures lower bounds, complexity reflexing ensures upper bounds). First, defunctionalise via a rewrite system into a first-order system. Built a tool <strong>tct-hoca</strong> which works well for various small higher-order functional programs (with complexity proofs taking up to 1 minute to compute). See the tools at:</p>
<ul>
<li><a href="https://github.com/ComputationWithBoundedResources/tct-trs">https://github.com/ComputationWithBoundedResources/tct-trs</a></li>
<li><a href="https://github.com/ComputationWithBoundedResources/tct-hoca">https://github.com/ComputationWithBoundedResources/tct-hoca</a></li>
</ul>
<p>What about “real” programs or “imperative” programs? Use ‘integer transition systems’ for modelling imperative programs, which is integrated into the tools now.</p>
<p>Some related work: <em>Multivariate amortized source analysis </em>(Hoffman, Aehlig, Hofmann, 2012) and a type-based approach “<em>Towards automatic resource bound analysis for OCaml</em>” (Hofmann, Das, Weng, POPL 2017) (<span style="color: #0000ff;">note to self: read this!</span>).</p>
<p>A previous challenge from Tobias Nipkow: how do you do this with data structures like splay trees (cf Tarjan)? There is some new work in progress. Build on having <em>sized types</em> so that we can account for tree size. An annotated signature then decorates function types with the size polynomial, e.g. <img src="https://s0.wp.com/latex.php?latex=%5Ctexttt%7Bsplay%7D+%3A+A+%5Ctimes+T_n+%5Cxrightarrow%7Bn%5E3+%2B+2%7D+T_n&bg=ffffff&fg=000000&s=0" alt="\texttt{splay} : A \times T_n \xrightarrow{n^3 + 2} T_n" class="latex" title="\texttt{splay} : A \times T_n \xrightarrow{n^3 + 2} T_n" /> (recursive function to create a splay tree). A type system for this was shown, with cost annotation judgment (no time to copy down any details).</p>
<p>The strength of the uniform resource analysis approach is its modularity and extensibility (different intermediate languages and complexity problems).<br />
Weaknesses: the extensibility/modularity required some abstraction which weakens the proving power, so adding new kinds of analysis (e.g., constant amortised / logarithmic amortised) requires a lot of work.<br />
Audience question: can I annotate my program with the expected complexity and get a counter-example if its false? (counter-examples, not yet).</p>
<hr style="height: 5pt; color: #444;" />
<p><a name="fscd-day2-talk1"></a>(FSCD) <b>Continuation Passing Style for Effect Handlers</b>, Daniel Hillerström, Sam Lindley, Bob Atkey, KC Sivaramakrishnan</p>
<p>Consider two effects: nondeterminism and exceptions, in the drunk toss example (a drunk person tosses a coin and either catches it and gets a bool for head or tails, or drops the coin [failure/exception]).</p>
<pre style="font-size: 10.5pt; font-family: monospace;">drunkToss : Toss ! {Choose:Bool; Fail:Zero}
drunkToss = if do Choose
then if do Choose then Heads else Tails
else absurd do Fail</pre>
<p>Induces a simple computation tree.</p>
<p>A modular handler, using row polymorphism (with variable <code style="font-size: 10.5pt; font-family: monospace;">r</code> below)</p>
<pre style="font-size: 10.5pt; font-family: monospace;">allChoices : a ! {Choose : Bool; r} => List a ! {r}
allChoices = return x |-> [x]
Choose k |-> k true ++ k false
fail : a ! {Fail: Zero; r} => List a ! {r}
fail = return x |-> [x]
Fail k |-> []</pre>
<p>The <code style="font-size: 10.5pt; font-family: monospace;">!</code> is an annotation explaining the effect operations, <code style="font-size: 10.5pt; font-family: monospace;">k</code> are continuations, <code style="font-size: 10.5pt; font-family: monospace;">r</code> are row variables (universally quantified). Two possible interpretations:</p>
<p>(1) handle the computation with <code style="font-size: 10.5pt; font-family: monospace;">fail</code> first, then handle that with <code style="font-size: 10.5pt; font-family: monospace;">allChoices</code><br />
returns the result <code style="font-size: 10.5pt; font-family: monospace;">[[Heads, Tails], []]</code>.</p>
<p>(2) handle with <code style="font-size: 10.5pt; font-family: monospace;">allChoices</code> first then <code style="font-size: 10.5pt; font-family: monospace;">fail</code> gives the result <code style="font-size: 10.5pt; font-family: monospace;">[]</code>.</p>
<p>So the order of the handling matters. The operational semantics for handlers is based on a substitution in the computation tree for the relevant operations.</p>
<p>How do handlers get compiled? Use CPS! (restricted subset of the lambda calculus, good for implementing control flow). Based on <img src="https://s0.wp.com/latex.php?latex=%5Clambda%5E%7B%5Crho%7D_%7B%5Ctextit%7Beff%7D%7D&bg=ffffff&fg=000000&s=0" alt="\lambda^{\rho}_{\textit{eff}}" class="latex" title="\lambda^{\rho}_{\textit{eff}}" /> (Hillerstrom, Lindley 2016) – lambda calculus with effect handlers (separate syntactic category but contains lambda terms), row polymorphism, and monadic metalanguage style constructs for effectful let binding and ‘return’. The CPS translation is a homomorphism on all the lambda calculus syntax, but continuation passing on the effectful fragment (effectful let and return). One part of the interpretation for the effectful operation term: <img src="https://s0.wp.com/latex.php?latex=%5B%5B%5Ctextbf%7Bdo%7D+%5C%2C+l+%5C%2C+V%5D%5D+%3D+%5Clambda+k+.+%5Clambda+h+.+%5C%3B+h+%28l+%5C%3B+%5Clangle%5B%5BV%5D%5D%2C+%5Clambda+x+.+k+%5C%3B+x+%5C%3B+h%5Crangle%29&bg=ffffff&fg=000000&s=0" alt="[[\textbf{do} \, l \, V]] = \lambda k . \lambda h . \; h (l \; \langle[[V]], \lambda x . k \; x \; h\rangle)" class="latex" title="[[\textbf{do} \, l \, V]] = \lambda k . \lambda h . \; h (l \; \langle[[V]], \lambda x . k \; x \; h\rangle)" /> (<span style="color: #808080;">see the paper for the full translation!</span>)</p>
<p>A key part of the translation is to preserve the stack of handlers so that the order is preserved and (the interpretation) of operations can access up and down the stack.<br />
A problem with the translation is that it yields administrative redexes (redundant beta-redexes) and is not tail-recursive. Use an uncurried CRPS where the stack of continuations is explicit (see Materzok and Biernacki 2012), this leads to a new translation which can manipulate the continuation stack directly (see paper). However, this still yields some administrative redexes. Solution: adopt a two-level calculus (Danvy, Nielsen 2033) which has two kinds of continuations: static (translation time) and dynamic (run time), where you can reify/reflect between the two. This let’s you squash out administrative redexes from the runtime to the static part, which can then be beta-reduced immediately at compile time (end up with no static redexes).<br />
The paper proves that this translation preserves the operational semantics.<br />
Implementation in the ‘links’ language: <a href="https://github.com/links-lang/links">https://github.com/links-lang/links</a></p>
<p>I was going to ask about doing this typed: this is shown in the paper.</p>
<hr style="height: 4pt; color: #444;" />
<p><a name="icfp-day2pm-1"></a>(ICFP) <strong>How to Prove Your Calculus Is Decidable: Practical Applications of Second-Order Algebraic Theories and Computation, </strong><a href="http://icfp17.sigplan.org/profile/makotohamana" class="navigate">Makoto Hamana</a></p>
<p>SOL (Second-Order Laboratory) – tool for analysing confluence and termination of second-order rewrite rules within Haskell. Provides an embedded Haskell DSL (via Template Haskell) for describing rewrite rules. SOL can then automatically check termination and confluence.</p>
<p>Confluence + termination => decidability. Can use this to prove a calculi is decidable.</p>
<p>Based on a higher-order version of Knuth-Bendix critical pair checking using extended HO pattern unification (Libal,Miller 2016).</p>
<p>For example, the <em>theory of monads</em> is decidable: given two well-typed terms <code style="font-size: 10.5pt; font-family: monospace;">s</code>, <code style="font-size: 10.5pt; font-family: monospace;">t</code> consistent of return, bind, and variables, the question is <code style="font-size: 10.5pt; font-family: monospace;">s = t</code> derivable from the three laws is decidable. How? First, orient the monad laws as rewrite laws. Check confluence (Church-Rosser, CR) and Strong Normalisation (SN)– together these imply their exists unique normal forms, which are then used to prove equivalence (reduce terms to their normal forms). So, how to prove CR and SN?</p>
<p>One way to prove Strong Normalisation is to assign weights and show that reductions reduce weights. Unfortunately, this does not work in the higher-order context of the monad operations. Another standard technique is to, use the “reducibility method” (see Tait,Girard and Lindley,Stark’05). SOL uses the ‘general schema criterion’ which works for general rewrite rules (Blanqui’00,’16) using various syntactic conditions (positivity, accessibility, safe use of recursive calls, metavariables). [sound but obviously not complete due to halting problem]. (Uses also <a href="https://en.wikipedia.org/wiki/Newman%27s_lemma">Newman’s Lemma</a>– a terminating TRS is confluent when it is locally confluent).</p>
<p>This tool can be applied to various calculi (see paper, 8 different systems). Sounds really useful when building new calculi. The tool was applied to a database of well know termination/confluence problems and could check 93/189 termination and 82/96 confluence problems.</p>
<p>“Let’s prove your calculus is decidable using SOL”<a href="https://www.cs.gunma-u.ac.jp/~hamana/"> (Makoto is happy to assist!)</a></p>
<hr style="height: 4pt; color: #444;" />
<p><a name="icfp-day2pm-2"></a>(ICFP) <strong>A Relational Logic for Higher-Order Programs</strong><br />
Alejandro Aguirre, Gilles Barthe, Marco Gaboardi, Deepak Garg, Pierre-Yves Strub</p>
<p>Relational properties are things like, if X and Y are related by R then F(X) and F(Y) are related by R, or over two properties, e.g., the results are related instead by S.</p>
<p>Relational refinement types: <img src="https://s0.wp.com/latex.php?latex=%5CGamma+%5Cvdash+t_1+%5Csim+t_2+%3A+%5C%7Bn+%5Cmid+n_1+%3D+n_2%5C%7D+&bg=ffffff&fg=000000&s=0" alt="\Gamma \vdash t_1 \sim t_2 : \{n \mid n_1 = n_2\} " class="latex" title="\Gamma \vdash t_1 \sim t_2 : \{n \mid n_1 = n_2\} " /> These can be used to express monotonicity properties, are syntax directed, and exploit structural similarities of code. However, say we want to prove the naturality of <code style="font-size: 10.5pt; font-family: monospace;">take</code>, but formulating the right relational refinement type is difficult because the computations are structurally different.</p>
<p>Start with a basic logic with lambda-terms over simple inductive types, and a separate layer of predicates</p>
<p>RHOL has judgements <img src="https://s0.wp.com/latex.php?latex=%5CGamma+%5Cmid+%5CPsi+%5Cvdash+t_1+%3A+%5Ctau_1+%5Csim+t_2+%3A+%5Ctau_2+%5Cmid+%5Cphi%28r_1%2C+r_2%29&bg=ffffff&fg=000000&s=0" alt="\Gamma \mid \Psi \vdash t_1 : \tau_1 \sim t_2 : \tau_2 \mid \phi(r_1, r_2)" class="latex" title="\Gamma \mid \Psi \vdash t_1 : \tau_1 \sim t_2 : \tau_2 \mid \phi(r_1, r_2)" /> where <img src="https://s0.wp.com/latex.php?latex=%5Cphi&bg=ffffff&fg=000000&s=0" alt="\phi" class="latex" title="\phi" /> is a binary predicate on properties of the two terms and <img src="https://s0.wp.com/latex.php?latex=%5CPsi&bg=ffffff&fg=000000&s=0" alt="\Psi" class="latex" title="\Psi" /> gives assertions about the free variables. A key idea here is that the types and assertions are separate.</p>
<p>An example rule two-sided rule (where we inductively construct terms with the same syntactic constructor on both sides of a relation):</p>
<p><img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma%2C+x_1+%3A+%5Csigma_1%2C+x_2+%3A+%5Csigma_2+%5Cmid+%5CPsi%2C+%5Cphi%27+%5Cvdash+t_1+%3A+%5Ctau_1+%5Csim+t_2+%3A+%5Ctau_2+%5Cmid+%5Cphi%7D+%7B%5CGamma+%5Cmid+%5CPsi+%5Cvdash+%5Clambda+x_1+%3A+t_1+%3A+%5Csigma_1+%5Crightarrow+%5Ctau_1+%5Csim+%5Clambda+x_2+.+t_2+%3A+%5Csigma_2+%5Crightarrow+%5Ctau_2+%C2%A0%5Cmid+%5Cforall+x_1+.+%5Cphi%27+%5CRightarrow+%5Cphi%5Br_1+x_1+%2F+r_1%2C+%5C%2C+r_2+x_2+%2F+r_2%5D%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma, x_1 : \sigma_1, x_2 : \sigma_2 \mid \Psi, \phi' \vdash t_1 : \tau_1 \sim t_2 : \tau_2 \mid \phi} {\Gamma \mid \Psi \vdash \lambda x_1 : t_1 : \sigma_1 \rightarrow \tau_1 \sim \lambda x_2 . t_2 : \sigma_2 \rightarrow \tau_2 \mid \forall x_1 . \phi' \Rightarrow \phi[r_1 x_1 / r_1, \, r_2 x_2 / r_2]}" class="latex" title="\dfrac{\Gamma, x_1 : \sigma_1, x_2 : \sigma_2 \mid \Psi, \phi' \vdash t_1 : \tau_1 \sim t_2 : \tau_2 \mid \phi} {\Gamma \mid \Psi \vdash \lambda x_1 : t_1 : \sigma_1 \rightarrow \tau_1 \sim \lambda x_2 . t_2 : \sigma_2 \rightarrow \tau_2 \mid \forall x_1 . \phi' \Rightarrow \phi[r_1 x_1 / r_1, \, r_2 x_2 / r_2]}" />.<br />
This seems quite natural to me: abstraction introduces a universal quantification over the constraints attached to the free variables.</p>
<p>Rules can also be one side (where you construct a term on just one side of a relation).<br />
The paper shows that RHOL is as expressive as HOL (by a mutual encoding).</p>
<p>Other relational typing system can be embedded into RHOL: Relational Refinement Types, DCC (Dependency Core Calculus, and RelCost (relational cost).</p>
<p>Q: if HOL and RHOL are equivalent (rather than their just being an embedding of HOL into RHOL say) then why have RHOL? Relational reasoning was gained which was not built into HO. <span style="color: #0000ff;">Update<span style="color: #000080;">:</span></span><span style="color: #000080;"> I discussed this further later with one of the authors, and they made the point that people have done relational-style reasoning in HOL but it requires taking a pairing of programs and building all the relational machinery by hand which is very cumbersome. RHOL has all this built in and the one-side and two-side rules let you reason about differently-structured terms much more easily. I can see this now.</span></p>
<hr style="height: 4pt; color: #444;" />
<p><a name="icfp-day2pm-3"></a>(ICFP) <strong><br />
Foundations of Strong Call by Need</strong><br />
Thibaut Balabonski, Pablo Barenbaum, Eduardo Bonelli, Delia Kesner</p>
<p>What does this lambda term reduce to: <img src="https://s0.wp.com/latex.php?latex=%28%5Clambda+x+%5C%2C+y.+x+%5C%2C+x%29+%5C%2C+%28id+%5C%2C+id%29&bg=ffffff&fg=000000&s=0" alt="(\lambda x \, y. x \, x) \, (id \, id)" class="latex" title="(\lambda x \, y. x \, x) \, (id \, id)" />? It depends on call-by-value, call-by-name, call-by-need, or full beta then you get different answers. The call-by-value, call-by-name, and call-by-need strategies are called <em>weak</em> because they do not reduce underneath lambda abstraction. As a consequence, they do not compute normal forms on their own.</p>
<p>Weak call-by-need strategy (the usual strategy outlined in the literature): <em>procrastinate </em>and <em>remember. </em>Beta reduction into a let binding (don’t evaluate the right-hand side of an application). (<span style="color: #333399;">There was some quite technical reduction work here, but I noted this interesting rule on “bubbling” reductions out by splitting a nested reduction into two:</span> <img src="https://s0.wp.com/latex.php?latex=t%5Bv%5Bu%2Fy%5D%2Fx%5D+%5Crightarrow+t%5Bu%2Fy%5D%5Bv%2Fx%5D&bg=ffffff&fg=000000&s=0" alt="t[v[u/y]/x] \rightarrow t[u/y][v/x]" class="latex" title="t[v[u/y]/x] \rightarrow t[u/y][v/x]" />).</p>
<p>Strong call-by-need provide the call-by-need flavour of only evaluating needed values once, <strong>but</strong> crucially computes normal forms (by reducing under a lambda). This strong approach has two properties (conservative) the strong strategy <em>first</em> does whatever the weak strategy will do; and (complete) if there is a beta-normal form it will reach (a representative of) it. This comes by reducing under the lambda at some points. Consequently, this approach only ever duplicates values, not computations.</p>
<p>Q: If you implemented this in Haskell, would it make it faster? I didn’t quite hear the answer, but this would completely change the semantics as Haskell is impure with respect to exceptions and non-termination. It sounded like it would be more useful for something pure like Coq/Agda (where it could provide some performance improvement).</p>
<hr style="height: 4pt; color: #444;" />
<p><a name="icfp-day2pm-4"></a>(ICFP) – <b>No-Brainer CPS Conversion</b>, <span style="text-decoration: underline;">Milo Davis</span>, William Meehan, Olin Shivers</p>
<p>The Fischer/Reynolds algorithm for CPS transformation introduces administrative redexes. Since then, other algorithms have improved on this considerably (Danvy/Filinsky). In this approach, some additional reductions are applied to the term to get a smaller CPS term out. The moral of the story (or the key idea) is to treat the CPS converter like a compiler (use abstract representations, symbol tables).</p>
<p>Rules of the game: make the terms strictly smaller (greedy reductions), should be strongly normalisation, and confluent, keep the original algorithmic complexity of the CPS transforms (linear time).</p>
<p>Approach:<br />
– Variables are the same size, so do beta redexes on variables<br />
– Beta reduce application of a lambda to a lambda <img src="https://s0.wp.com/latex.php?latex=%28%5Clambda+x+.+e%29+%28%5Clambda+y+.+e%27%29&bg=ffffff&fg=000000&s=0" alt="(\lambda x . e) (\lambda y . e')" class="latex" title="(\lambda x . e) (\lambda y . e')" /> only when the number of reference to <img src="https://s0.wp.com/latex.php?latex=x&bg=ffffff&fg=000000&s=0" alt="x" class="latex" title="x" /> in <img src="https://s0.wp.com/latex.php?latex=e&bg=ffffff&fg=000000&s=0" alt="e" class="latex" title="e" /> is less than or equal to 1 (so things don’t explode).<br />
– Eta reduce (??only when there are no references to the variable in the body??) (<span style="color: #333399;">I think this is what was said, but I think I might have made a mistake here</span>).</p>
<p>Relies on some machinery: doing reference counts (variable use count), abstract representation of continuations (halt, variables, function continuations, and application continuations) rather than going straight to syntax and syntax constructor functions (which can do reductions) to dovetail with this, and explicit environments to be used with function closures.</p>
<hr style="height: 4pt; color: #444;" />
<p><a name="icfp-day2-pm2-talk1"></a>(ICFP) <b>Compiling to Categories</b>, Conal Elliott</p>
<p>Why overload? Get a common vocabulary, laws for modular reasoning.</p>
<p>But generally doesn’t apply to lambda, variables, and application. So what to do? <span style="text-decoration: underline;">Eliminate them!</span> Rewrite lambda terms into combinators: const, id, apply, pairing of functions (fork), curry, and keep going…. and you can automate this via a compiler plugin (so you don’t have to write code like this yourself).</p>
<p>This is implemented by the interface (algebra) of a category (for function composition / identities) + Cartesian (product) structure + coproducts + exponents: giving you (bi)Cartesian-closed categories. Implement this as class/interface so you can give different realisations/compilations, e.g., a graph drawing implementation (Conal showed some pretty pictures generated from this).</p>
<p>Another implementation is to generate Verilog (hardware descriptions), or to graphics shader code.</p>
<p>What about compiling to the derivative of a program? Represent this via the type:<br />
<code style="font-size: 10.5pt; font-family: monospace;">newtype D a b = D (a -> b x (a -o b))</code> i.e., a differentiable function is a function which produces its result and its derivative as a linear map (an implementation was then shown, see paper). You can then compose interpretations, e.g., graph interpretation with derivative interpretation.</p>
<p>Another interpretation: interval analysis (e.g., <code style="font-size: 10.5pt; font-family: monospace;">Interval Double = (Double, Double)</code>). There are more examples in the paper, including constraint solving via SMT by writing functions to <code style="font-size: 10.5pt; font-family: monospace;">Bool</code> which are then compiled directly into SMT constraints (rather than using existing embeddings, e.g., SBV).</p>
<p>Q: what about recursion? Can be done (compiling from a <code style="font-size: 10.5pt; font-family: monospace;">letrec</code> into a fix-point combinator class).</p>
<hr style="height: 4pt; color: #444;" />
<p><a name="icfp-day2-pm2-talk2"></a><strong> </strong>(ICFP)<strong> Visitors Unchained</strong>, François Pottier</p>
<p>(Humorous tagline from Francois: Objects at ICFP?! Binders again?!)</p>
<p>Manipulating abstract syntax with binding can be a chore: <em><span style="text-decoration: underline;">nameplate </span></em>(boilerplate on names, term coined by James Cheney). Large and easy to get wrong (especially in weakly typed languages, non-dependent languages). This paper is a way of getting rid of nameplate in OCaml via a library and an syntax extension. Supports multiple representations, complex binding, is modular, and relies on just a small amount of code generation.</p>
<p>Several Haskell libraries support this via its support for datatype generic programming. This will use OCaml’s object technology. Annotating a data type with <code style="font-size: 10.5pt; font-family: monospace;">[@@deriving visitors { variety = 'map' }]</code> automatically generates a visitor pattern (a class) for doing a <code style="font-size: 10.5pt; font-family: monospace;">map</code>-operations on the datatype. To use this, create a local object inheriting the visitor class and override one or more methods, e.g., for a syntax tree type, override the visit method for the “add” node in order to implement the rewriting of <code style="font-size: 10.5pt; font-family: monospace;">e+0 -> e</code>.<br />
Visitors are quite versatile: easy to customise behaviour via inheritance which is something you can’t do in FP without a lot of redesign to your algorithms.</p>
<p>Want to traverse syntax with binders, considering three things (from three perspectives or “users” in the talk here):<br />
(1) “End user” describes the structure of an AST, parameterised by types of bound names and free names and using a parametric data type <code style="font-size: 10.5pt; font-family: monospace;">abs</code> whenever a binding is needed (e.g., in a lambda). This should also deriving the <code style="font-size: 10.5pt; font-family: monospace;">map</code> visitor(see above) with option <code style="font-size: 10.5pt; font-family: monospace;">["BindingForms.map"]</code>.<br />
(2) “Binding library” (provided as part of this work) defines the <code style="font-size: 10.5pt; font-family: monospace;">abs</code> type which capture the binding construct, but they leave the scope extrusion function <code style="font-size: 10.5pt; font-family: monospace;">extend</code> as a parameter (<code style="font-size: 10.5pt; font-family: monospace;">extend</code> gets used when you traverse a binder to go inside a scope, e.g., going into the body of a lambda and extending the context by the bound variable).<br />
(3) The last part defines how to represent environments (and lookups) via an overriding a visitor, nominal terms, and scope extrusion. (The work provides some implementations of these part in a library too).</p>
<p>As seen above, the structure of this library is fairly modular. The “end user” has to glue all these things together.<br />
One limitation is that nonlinear patterns (e.g., as in Erlang) can’t be represented (I didn’t immediately see why though).</p>
<hr />
<p><a name="icfp-day2-pm2-talk3"></a> (ICFP) – <b>Staged Generic Programming</b>, Jeremy Yallop</p>
<p>Advert at the start: <em>if you have an abstract library that is not very high-performance, then the techniques here could be useful to you.</em></p>
<p>Generic programming a la Scrap Your Boilerplate, for any data type get automatic traversals e.g., <code style="font-size: 10.5pt; font-family: monospace;">listify (matches c)</code> for some constant <code style="font-size: 10.5pt; font-family: monospace;">c</code>, gives you a list of all nodes in a data type matching <code style="font-size: 10.5pt; font-family: monospace;">c</code>; as a programming you don’t need to write <code style="font-size: 10.5pt; font-family: monospace;">listify</code> or <code style="font-size: 10.5pt; font-family: monospace;">matches</code>: they are generated. Requires internally a way to represent types as data and to map data types into a generic form, a collection of traversals over the general form, and generic schemes which plug together the traversals with parameter generic queries. (In Haskell see the <a href="http://hackage.haskell.org/package/syb">syb</a>). <strong><span style="text-decoration: underline;">However, the overhead of this roughly 20x writing the code manually!</span></strong> This overhead was quite portable: reimplementing Scrap Your Boilerplate into OCaml had about the same amount of slow down compared with manual traversals.</p>
<p>Why this slow? Type comparisons are slow, lots of indirection, and lots of unnecessary applications of applications. (<span style="color: #333399;">Dominic: to me it seems like it ends up a bit like the slowness involved in dynamically typed languages</span>).</p>
<p>Solution: use <em>staging</em> to optimise the datatype generic implementation. Staging lets you explain which parts of the program can be evaluated early, and which bits need to be evaluated later (due to waiting for inputs)– these are the quoted bits (inside <code style="font-size: 10.5pt; font-family: monospace;"></code> with the option to jump back into the early part by backquoting <code style="font-size: 10.5pt; font-family: monospace;">~</code>).</p>
<p>Naively staging SYB keeps the type representation unchanged, but the shallow traversals and recursion schemes can be quoted judiciously such that type comparison, indirect calls, and polymorphism are all eliminated. (Jeremy showed us the code output.. but it has lots of noise and opportunities for improvement).</p>
<p>Instead, use a special fixed-point operator that inlines non-recursive functions and perform (semantics preserving) code transformations as well: e.g., decomposing the generic map operation.</p>
<p>One technique is to allow equations on “partially-static data” e.g., if you have <code style="font-size: 10.5pt; font-family: monospace;">[] @ x @ []</code> (i.e., static empty list concat dynamic list <code style="font-size: 10.5pt; font-family: monospace;">x</code> concat static empty list <code style="font-size: 10.5pt; font-family: monospace;">[]</code>) this still can be statically rewritten into <code style="font-size: 10.5pt; font-family: monospace;">x</code>. Thus apply laws on the static parts as far as possible.<br />
Another technique was to eta reduces matches that have identical branches.<br />
(I think there were a few more thing here). Ends up with much tighter code which performs very close to or even better than the hand-written code.</p>
<p>Q: How does this compare to the<a href="https://hackage.haskell.org/package/TYB"> Template-your-Boilerplate library for Haskell</a>?<br />
<span style="color: #333399;">I completely missed the answer as I was looking up this library which I really should be using…!</span></p><br /> <a href="http://feeds.wordpress.com/1.0/gocomments/dorchard.wordpress.com/491/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/dorchard.wordpress.com/491/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=dorchard.wordpress.com&blog=10180011&post=491&subd=dorchard&ref=&feed=1" alt="" height="1" border="0" width="1" />Thu, 07 Sep 2017 10:48:08 +0000Dominic Orchard: ICFP / FSCD day 3 – rough noteshttp://dorchard.wordpress.com/?p=599
https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/
<p>(Blog posts for <a href="https://dorchard.wordpress.com/2017/09/04/icfp-fscd-day-1-rough-conference-notes/">Day 1</a>, <a href="https://dorchard.wordpress.com/2017/09/05/icfp-fscd-day-2-rough-notes/">Day 2</a>, <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/">Day 3</a>, <a href="https://dorchard.wordpress.com/2017/09/07/fscd-day-4-rough-notes/">Day 4</a> (half day))</p>
<p>I decided to take electronic notes at <a href="http://icfp17.sigplan.org/program/program-icfp-2017">ICFP</a> and <a href="http://www.cs.ox.ac.uk/conferences/fscd2017/">FSCD</a> (colocated) this year, and following the example of various people who put their conference notes online (which I’ve found useful), I thought I would attempt the same. However, there is a big caveat: <span style="color: #ff6600;">my notes are going to be partial and may be incorrect</span>; my apologies to the speakers for any mistakes.</p>
<ul>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-am1">A Specification for Dependent Types in Haskell</a>, Stephanie Weirich, Antoine Voizard, Pedro Henrique Avezedo de Amorim, Richard A. Eisenberg</li>
<li>(ICFP) <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-am2">Parametric Quantifiers for Dependent Type Theory</a> Andreas Nuyts, Andrea Vezzosi, Dominique Devriese</li>
<li>(ICFP) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-am3">Normalization by Evaluation for Sized Dependent Types</a>, Andreas Abel, Andrea Vezzosi, Theo Winterhalter</li>
<li>(ICFP) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-am4">A Metaprogramming Framework for Formal Verification</a>, Gabriel Ebner, Sebastian Ullrich, Jared Roesch, Jeremy Avigad, Leonardo De Moura</li>
<li>(FSCD) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-fscd-pm1">A Fibrational Framework for Substructural and Modal Logics, </a>Dan Licata, Michael Shulman, Mitchell Riley</li>
<li>(FSCD) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-fscd-pm2">Dinaturality between syntax and semantics</a>, Paolo Pistone</li>
<li>(FSCD) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-fscd-pm3">Models of Type Theory Based on Moore Paths</a>, Andrew Pitts, Ian Orton</li>
<li>(ICFP) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-pm2-talk1">Theorems for Free for Free: Parametricity, With and Without Types</a>, Amal Ahmed, Dustin Jamner, Jeremy G. Siek, Philip Wadler</li>
<li>(ICFP) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-pm2-talk2">On Polymorphic Gradual Typing</a>, Yuu Igarashi, Taro Sekiyama, Atsushi Igarashi</li>
<li>(ICFP) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-pm3-talk1">Constrained Type Families</a>, J. Garrett Morris, Richard A. Eisenberg</li>
<li>(ICFP) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-pm3-talk2">Automating Sized-Type Inference for Complexity Analysis</a>, Martin Avanzini, Ugo Dal Lago</li>
<li>(ICFP) – <a href="https://dorchard.wordpress.com/2017/09/06/icfp-fscd-day-3-rough-notes/#icfp-day3-pm3-talk3">Inferring Scope through Syntactic Sugar</a>, Justin Pombrio, Shriram Krishnamurthi, Mitchell Wand</li>
</ul>
<hr style="height: 4pt; color: #444;" />
<p><a name="icfp-day3-am1"></a>(ICFP) – <b>A Specification for Dependent Types in Haskell</b>, Stephanie Weirich, <span style="text-decoration: underline;">Antoine Voizard</span>, Pedro Henrique Avezedo de Amorim, Richard A. Eisenberg</p>
<p>Haskell already has dependent types! ish! (ICFP’14) But singletons are “gross”! (mock discussion between Stephanie and Richard). This work describes a semantics for dependent types in Haskell, along with a replacement for GHC’s internal (Core) language, along with the high-level type theory and fully mechanized metatheory.</p>
<p>But isn’t Haskell already dependently typed? Cf Haskell indexed type for vectors. The following cannot be written in GHC at the moment:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">vreplicate :: Pi(n :: Nat) -> Int -> Vec n
vreplicate Zero _ = Nil
vreplicate (Succ n) a = Cons a (vreplicate n a)</pre>
<p>The reason this can’t be typed is due to <code style="font-size: 10.5pt; font-family: monospace;">n</code> being erased (after compilation, i.e., it doesn’t appear in the run time) but it is needed dynamically in the first argument. In Dependent Haskell this can be done.</p>
<p>But… it’s not your usual system. For example, what about infinite data?</p>
<pre style="font-size: 10.5pt; font-family: monospace;">inf :: Nat
inf = S inf
vinf :: Vec inf
vinf = Cons 2 vinf -- yields equality demand Vec (S inf) = Vec inf</pre>
<p>Idealized Haskell (IH) = System F_omega + GADTs (has uncediable type checking!)</p>
<p>System FC (GHC Core) is a reified version of Idealized Haskell with enough annotations so that type checking is decidable, trivial, and syntax directed. Terms in Core can be see as type derivations of IH terms: contains type annotations and type equalities. (These can all be erased to get back into IH.) The first contribution of this work is a dependent System FC (DC). But its very complicated (needs also type soundness (progress/preservation), erasable coercions, ).<br />
Instead, they introduce System D, the dependently typed analog of Idealized Haskell: its a stripped down dependent Haskell in Curry style (implicit types) and similarly to IH its type checking is undecidable. It has dependent types, non-termination, and <span style="text-decoration: underline;">coercion abstraction.</span></p>
<p>Coercion abstraction: for example</p>
<pre style="font-size: 10.5pt; font-family: monospace;">g : Int -> Int |- \v -> \c -> \v -> v : Pi n . (g n ~ 7) -> Vec (g n) -> Vec 7</pre>
<p>Here the variable <code style="font-size: 10.5pt; font-family: monospace;">v</code> (which has type <code style="font-size: 10.5pt; font-family: monospace;">Vec (g n)</code> when passed in) is implicitly cast using the coercion <code style="font-size: 10.5pt; font-family: monospace;">c</code> (which is passed in as a parameter) to <code style="font-size: 10.5pt; font-family: monospace;">Vec 7</code>. This is translated into DC using an explicit coercion operator, applying <code style="font-size: 10.5pt; font-family: monospace;">c</code> to coerce <code style="font-size: 10.5pt; font-family: monospace;">v</code>.</p>
<p>System D and System DC were mechanized (in Coq) during the design using Ott (to input the rules of the language) & LNgen (for generating lemmas). Currently working on dependent pattern matching, a full implementation in GHC, and roles. (Hence, this is the specification of the type system so far).</p>
<p>Q: type inference for Dependent Haskell?<br />
A: Not sure yet (<span style="color: #333399;">but see Richard Eisenberg’s thesis</span>)<br />
Q: How do you get decidable type checking in the presence of infinite terms?<br />
<span style="color: #333399;">A: I didn’t understand the answer (will followup later).</span><br />
Q: Why is coercion abstraction explicit in D while coercion application is implicit?<br />
A: “Coercion application *is* explicit in D, it’s the use of coercion to cast a type that is implicit, but if you were to apply it to a value it would be explicit.”</p>
<hr style="height: 4pt; color: #333;" />
<p><a name="icfp-day3-am2"></a>(ICFP) – <b>Parametric Quantifiers for Dependent Type Theory</b> <span style="text-decoration: underline;">Andreas Nuyts</span>, Andrea Vezzosi, Dominique Devriese</p>
<p>A type variable is parameteric if is only used for type-checking (free well-behavedness theorems): i.e., can’t be inspected by pattern matching so you have the same algorithm on all types, e.g. <code style="font-size: 10.5pt; font-family: monospace;">flatten : forall X . Tree X -> List X</code>. Parametericity is well studied in the System F world. This work looks at parametericity in dependent type theory.; some results carry over, some can be proved internally, but some are lost. This work formulates a sound dependent type system ParamDTT which .</p>
<p>Parametricity gives representation independence (in System F):<br />
<img src="https://s0.wp.com/latex.php?latex=A+%5Crightarrow+B+%5Ccong+%28%5Cforall+X+.+%28X+%5Crightarrow+A%29+%5Crightarrow+%28X+%5Crightarrow+B%29&bg=ffffff&fg=000000&s=0" alt="A \rightarrow B \cong (\forall X . (X \rightarrow A) \rightarrow (X \rightarrow B)" class="latex" title="A \rightarrow B \cong (\forall X . (X \rightarrow A) \rightarrow (X \rightarrow B)" /><br />
(This result follows by parametricity, using the result that <img src="https://s0.wp.com/latex.php?latex=g+%3A+%5Cforall+X+.+%28X+%5Crightarrow+A%29+%5Crightarrow+%28X+%5Crightarrow+B%29&bg=ffffff&fg=000000&s=0" alt="g : \forall X . (X \rightarrow A) \rightarrow (X \rightarrow B)" class="latex" title="g : \forall X . (X \rightarrow A) \rightarrow (X \rightarrow B)" /> implies <img src="https://s0.wp.com/latex.php?latex=g+%5C%2C+X_0+%5C%2C+r_0+%5C%2C+x%5C%2C+%3D+g+%5C%2C+A+%5C%2C+%5Ctextit%7Bid%7D+%5C%2C+%28r_0+%5C%3B+x_0%29&bg=ffffff&fg=000000&s=0" alt="g \, X_0 \, r_0 \, x\, = g \, A \, \textit{id} \, (r_0 \; x_0)" class="latex" title="g \, X_0 \, r_0 \, x\, = g \, A \, \textit{id} \, (r_0 \; x_0)" /> which can be provided by relational parametricity, see Reynolds “related things map to related things”, applies identity extension lemma).</p>
<p>Can we do the same thing for DTT and can we prove the result internally?<br />
<img src="https://s0.wp.com/latex.php?latex=%5CPi&bg=ffffff&fg=000000&s=0" alt="\Pi" class="latex" title="\Pi" /> is however <span style="text-decoration: underline;">not parametric</span>. For example if we convert to <img src="https://s0.wp.com/latex.php?latex=%5CPi+%28X+%3A+U%29+.+%28X+%5Crightarrow+A%29+%5Crightarrow+%28X+%5Crightarrow+B%29&bg=ffffff&fg=000000&s=0" alt="\Pi (X : U) . (X \rightarrow A) \rightarrow (X \rightarrow B)" class="latex" title="\Pi (X : U) . (X \rightarrow A) \rightarrow (X \rightarrow B)" />. Suppose $B = U$ then we can leak details of the implementation (rerepresentation type is returned as data) <img src="https://s0.wp.com/latex.php?latex=%5Ctextit%7Bleak%7D+%5C%3B+X+%5C%3B+r+%5C%3B+x+%3D+X&bg=ffffff&fg=000000&s=0" alt="\textit{leak} \; X \; r \; x = X" class="latex" title="\textit{leak} \; X \; r \; x = X" /> (has this type). This violates the identity extension lemma used in the above proof for System F.</p>
<p>So instead, <strong>add </strong>a parametric quantifier to DTT to regain representation independence, making the above <img src="https://s0.wp.com/latex.php?latex=%5Ctextit%7Bleak%7D&bg=ffffff&fg=000000&s=0" alt="\textit{leak}" class="latex" title="\textit{leak}" /> ill typed. The proof of parametricity for <img src="https://s0.wp.com/latex.php?latex=g+%3A+%5Cforall+X+.+%28X+%5Crightarrow+A%29+%5Crightarrow+%28X+%5Crightarrow+B%29&bg=ffffff&fg=000000&s=0" alt="g : \forall X . (X \rightarrow A) \rightarrow (X \rightarrow B)" class="latex" title="g : \forall X . (X \rightarrow A) \rightarrow (X \rightarrow B)" /> can now be proved internally. This uses the relational interval type <img src="https://s0.wp.com/latex.php?latex=0+--+1+%3A+%5Cmathbb%7BI%7D&bg=ffffff&fg=000000&s=0" alt="0 -- 1 : \mathbb{I}" class="latex" title="0 -- 1 : \mathbb{I}" /> (cf Bernardy, Coquand, and Moulin 2015, on bridge/path-based proofs) which gives a basis for proving the core idea of “related things map to related things” where <img src="https://s0.wp.com/latex.php?latex=0&bg=ffffff&fg=000000&s=0" alt="0" class="latex" title="0" /> is connected with the type <img src="https://s0.wp.com/latex.php?latex=X&bg=ffffff&fg=000000&s=0" alt="X" class="latex" title="X" /> (in the above type) and <img src="https://s0.wp.com/latex.php?latex=1&bg=ffffff&fg=000000&s=0" alt="1" class="latex" title="1" /> is connected to the type <img src="https://s0.wp.com/latex.php?latex=A&bg=ffffff&fg=000000&s=0" alt="A" class="latex" title="A" /> via a switching term <img src="https://s0.wp.com/latex.php?latex=%2F+r%2F+%3A+%5Cmathbb%7BI%7D+%5Crightarrow+U&bg=ffffff&fg=000000&s=0" alt="/ r/ : \mathbb{I} \rightarrow U" class="latex" title="/ r/ : \mathbb{I} \rightarrow U" /> (see paper for the proof) (I think this is analogous to setting up a relation between <img src="https://s0.wp.com/latex.php?latex=X&bg=ffffff&fg=000000&s=0" alt="X" class="latex" title="X" /> and <img src="https://s0.wp.com/latex.php?latex=A&bg=ffffff&fg=000000&s=0" alt="A" class="latex" title="A" /> in the usual relational parametricity proof).</p>
<p>They extended Agda with support for this.</p>
<hr style="height: 4pt; color: #333;" />
<p><a name="icfp-day3-am3"></a>(ICFP) – <b>Normalization by Evaluation for Sized Dependent Types</b><br />
<span style="text-decoration: underline;">Andreas Abel</span>, Andrea Vezzosi, Theo Winterhalter</p>
<p>(context, DTT a la Martin-Lof, and you can add subtpying to this, where definitional equality implies subtyping). Definitional equality is decideable (for the purposes of type checking), but the bigger the better (want to be able to know as many things equal as possible).</p>
<p>Termination is needed for consistency, general <code style="font-size: 10.5pt; font-family: monospace;">fix</code> gives you inconsistency. Instead, you can used data types indexed by tree height, where <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7BNat%7D%5Ei+%3D+%5C%7B+n+%5Cmid+n+%3C+i%5C%7D&bg=ffffff&fg=000000&s=0" alt="\textsf{Nat}^i = \{ n \mid n < i\}" class="latex" title="\textsf{Nat}^i = \{ n \mid n < i\}" />, you can define <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bfix%7D+%3A+%28%5Cforall+i+%5Crightarrow+%28%5Ctextsf%7BNat%7D%5E%7Bi%7D+%5Crightarrow+C%29+%5Crightarrow+%28%5Ctextsf%7BNat%7D%5E%7Bi%2B1%7D+%5Crightarrow+C%29%29+%5Crightarrow+%5Cforall+j+%5Crightarrow%C2%A0%5Ctextsf%7BNat%7D%5Ej+%5Crightarrow+C&bg=ffffff&fg=000000&s=0" alt="\textsf{fix} : (\forall i \rightarrow (\textsf{Nat}^{i} \rightarrow C) \rightarrow (\textsf{Nat}^{i+1} \rightarrow C)) \rightarrow \forall j \rightarrow \textsf{Nat}^j \rightarrow C" class="latex" title="\textsf{fix} : (\forall i \rightarrow (\textsf{Nat}^{i} \rightarrow C) \rightarrow (\textsf{Nat}^{i+1} \rightarrow C)) \rightarrow \forall j \rightarrow \textsf{Nat}^j \rightarrow C" />.</p>
<p>However, size expressions<em> are not unique,</em> which is problematic for proofs. e.g., <img src="https://s0.wp.com/latex.php?latex=suc+i+%3A%C2%A0%5Ctextsf%7BNat%7D%5Ei+%5Crightarrow%C2%A0%5Ctextsf%7BNat%7D%5E%5Cinfty&bg=ffffff&fg=000000&s=0" alt="suc i : \textsf{Nat}^i \rightarrow \textsf{Nat}^\infty" class="latex" title="suc i : \textsf{Nat}^i \rightarrow \textsf{Nat}^\infty" /> but also <img src="https://s0.wp.com/latex.php?latex=suc+%5Cinfty+%3A%C2%A0%5Ctextsf%7BNat%7D%5Ei+%5Crightarrow%C2%A0%5Ctextsf%7BNat%7D%5E%5Cinfty&bg=ffffff&fg=000000&s=0" alt="suc \infty : \textsf{Nat}^i \rightarrow \textsf{Nat}^\infty" class="latex" title="suc \infty : \textsf{Nat}^i \rightarrow \textsf{Nat}^\infty" />. Intuition: sizes are irrelevant in terms, but relevant in types only.</p>
<p>Andreas set up some standard definitions of inductive <code style="font-size: 10.5pt; font-family: monospace;">Nat</code> then tried to define a Euclidean division on <code style="font-size: 10.5pt; font-family: monospace;">Nat</code> (where <code style="font-size: 10.5pt; font-family: monospace;">monus</code> is minus cutting off at <code style="font-size: 10.5pt; font-family: monospace;">0</code>).</p>
<pre style="font-size: 10.5pt; font-family: monospace;">div : Nat → Nat → Nat
div zero y = zero
div (suc x) y = {! suc (div (monus x y) y) !}</pre>
<p>However, Agda fails to termination check this. The solution is to “size type”-up everything, i.e., redefine the usual inductive <code style="font-size: 10.5pt; font-family: monospace;">Nat</code> to have a size index:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">data Nat : Size → Set where
zero : ∀ i → Nat (↑ i)
suc : ∀ i → Nat i → Nat (↑ i)</pre>
<p>All the other definitions were given sized type parameters (propogated through) and then <code style="font-size: 10.5pt; font-family: monospace;">div</code> was redefined (type checking to):</p>
<pre style="font-size: 10.5pt; font-family: monospace;">div : ∀ i → Nat i → Nat ∞ → Nat i
div .(↑ i) (zero i) y = zero i
div .(↑ i) (suc i x) y = suc _ (div i (monus i _ x y) y)</pre>
<p><span style="color: #333399;">So far this is the usual technique. <span style="color: #000000;">However, now we have difficulty proving a lemma (which was straightforward before sizing that minus of <code style="font-size: 10.5pt; font-family: monospace;">x</code> with itself gives <code style="font-size: 10.5pt; font-family: monospace;">0</code>. Now this looks like:<br />
</span></span></p>
<pre style="font-size: 10.5pt; font-family: monospace;">monus-diag : ∀ i (x : Nat i) → Eq ∞ (monus i i x x) (zero ∞)</pre>
<p>In the case: <code style="font-size: 10.5pt; font-family: monospace;">monus-diag .(↑ i) (suc i x) = monus-diag i x</code> Agda gets the error <code style="font-size: 10.5pt; font-family: monospace;">i != ↑ i of type Size</code>.</p>
<p>In the Church-style, dependent type functions = polymorphic functions, so you can’t have irrelevant arguments. In Curry-style (with forall) this is okay. (<span style="color: #333399;">see the previous talk, this could be a possible alternate solution</span>?)</p>
<p>Instead, we want “Churry”-style where size arguments are used for type checking but can be ignored during equality checking: thus we want typing rules for irrelevant sizes. This was constructed via a special “irrelevant” modality <img src="https://s0.wp.com/latex.php?latex=%5Cbullet&bg=ffffff&fg=000000&s=0" alt="\bullet" class="latex" title="\bullet" />, where sizes uses as indexes can be marked as irrelevant in the context. Here is one of the rules for type formation when abstracting over an irrelevantly typed argument:</p>
<p><img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma+%5Cvdash+A+%3A+Type+%5Cquad+%5CGamma%2C+%5Cbullet+%5Cbullet+x+%3A+A+%5Cvdash+B+%3A+%5Ctextsf%7Btype%7D%7D%7B%5CGamma+%5Cvdash+%28x+%3A+%5Cbullet+A%29+%5Crightarrow+B+%3A+%5Ctextsf%7Btype%7D%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma \vdash A : Type \quad \Gamma, \bullet \bullet x : A \vdash B : \textsf{type}}{\Gamma \vdash (x : \bullet A) \rightarrow B : \textsf{type}}" class="latex" title="\dfrac{\Gamma \vdash A : Type \quad \Gamma, \bullet \bullet x : A \vdash B : \textsf{type}}{\Gamma \vdash (x : \bullet A) \rightarrow B : \textsf{type}}" /></p>
<p>This seemed to work really nicely (<span style="color: #333399;">I couldn’t get all the details down, but the typing rules were nice and clean and made a lot of sense as a way of adding irrelevancy).</span></p>
<hr />
<p><a name="icfp-day3-am4"></a>(ICFP) – <b>A Metaprogramming Framework for Formal Verification</b><br />
Gabriel Ebner, <span style="text-decoration: underline;">Sebastian Ullrich</span>, Jared Roesch, Jeremy Avigad, Leonardo De Moura</p>
<p>(Context: Lean is a dependently-typed theorem prover that aims to bridge the gap between interactive and automatic theorem proving. It has a very small trusted kernel (no pattern matching and termination checking), based on Calculus of Inductive Constructions + proof irrelevance + quotient types.)</p>
<p>They want to extend Lean with tactics, but its written in C++ (most tactic approaches are defined internally, cf Coq). Goal: to extend Lean using Lean by making Lean a reflective metaprogramming language (a bit like Idris) does, in order to build and run tactics within Lean. Do this by exposing Lean internals to Lean (Unification, type inference, type class resolution, etc.). Also needed an efficient evaluator to run meta programs.</p>
<pre style="font-size: 10.5pt; font-family: monospace;">lemma simple (p q : Prop) (hp : p) (hq : q) : q :=
by assumption</pre>
<p>Assumption is a tactic, a meta program in the <code style="font-size: 10.5pt; font-family: monospace;">tactic</code> monad:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">meta def assumption : tactic unit := do
ctx <- local_context,
t <- target
h <- find t ctx,
exact h</pre>
<p>(seems like a really nice DSL for building the tactics). Terms are reflected so there is a meta-level inductive definition of Lean terms, as well as quote and unquote primitives built in, but with a shallow (constant-time) reflection and reification mechanism.</p>
<p>The paper shows an example that defines a robust simplifier that is only about 40 locs (applying a congruence closure). They built a bunch of other things, including a bigger prover (~3000 loc) as well as command-line VM debugger as a meta program (~360 loc).</p>
<p>Next need a DSL for actually working with tactics (not just defining them). Initially quite ugly (with lots of quoting/unquoting). So to make things cleaner they let the tactics define their own parsers (which can then hook into the reflection mechanism). They then reused this to allow user-defined notations.</p>
<hr />
<p><a name="icfp-day3-fscd-pm1"></a>(FSCD) <strong>A Fibrational Framework for Substructural and Modal Logics, </strong>Dan Licata, Michael Shulman, Mitchell Riley</p>
<p>Background/context: model logic core <img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5Cemptyset+%5Cvdash+A%7D%7B%5CGamma+%5Cvdash+%5CBox+A%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\emptyset \vdash A}{\Gamma \vdash \Box A}" class="latex" title="\dfrac{\emptyset \vdash A}{\Gamma \vdash \Box A}" /> for necessity (true without any hypotheses) <img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma+%5Cvdash+%5CDiamond+A%7D%7B%5CDiamond+%5CGamma+%5Cvdash+%5CDiamond+B%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma \vdash \Diamond A}{\Diamond \Gamma \vdash \Diamond B}" class="latex" title="\dfrac{\Gamma \vdash \Diamond A}{\Diamond \Gamma \vdash \Diamond B}" /> (if something A is possible true then possibility can propagate to the context– doesn’t change the “possibleness”). Various intuitionistic substructural and modal logics/type systems: linear/affine, relevant, ordered, bunched (separation logic), coeffects, etc. Cohesive HoTT: take dependent type theory and add modalities (int, sharp, flat). S-Cohesion (Finster, Licata, Morehouse, Riley: a comonad and monad that themselves are adjoint, ends up with two kind of products which the modality maps between). Motivation: what are the common patterns in substructural and modal logics? How do we construct these things more systematically from a a small basic calculus.</p>
<p>With S4 modality (in sequent calculus), its common to make two contexts, one of modality formulae and one of normal, e.g., <img src="https://s0.wp.com/latex.php?latex=%5CGamma%3B+%5CDelta+%5Cvdash+A&bg=ffffff&fg=000000&s=0" alt="\Gamma; \Delta \vdash A" class="latex" title="\Gamma; \Delta \vdash A" /> which is modelled by <img src="https://s0.wp.com/latex.php?latex=%5CBox+%5CGamma+%5Ctimes+%5CDelta+%5Crightarrow+A&bg=ffffff&fg=000000&s=0" alt="\Box \Gamma \times \Delta \rightarrow A" class="latex" title="\Box \Gamma \times \Delta \rightarrow A" />. In the sequent calculus style, use the notion of context to form the type. Linear logic ! is similar (but with no weakening). The rules for <img src="https://s0.wp.com/latex.php?latex=%5Cotimes&bg=ffffff&fg=000000&s=0" alt="\otimes" class="latex" title="\otimes" /> follow the same kind of pattern: e.g., <img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma%2C+A%2C+B+%5Cvdash+C%7D%7B%5CGamma%2C+A%5Cotimes+B+%5Cvdash+C%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma, A, B \vdash C}{\Gamma, A\otimes B \vdash C}" class="latex" title="\dfrac{\Gamma, A, B \vdash C}{\Gamma, A\otimes B \vdash C}" /> where the context “decays” into the logical operator (rather than being literally equivalent): the types inherent the properties of the context (cf. also exchange).</p>
<p><em><span style="text-decoration: underline;">The general pattern: operations on contexts, with explicit or admissible structure properties, then have a type constructor (an operator) that internalises the implementation and inherits the properties.</span></em> This paper provides a framework for doing this, and abstracting the common aspects of many intuitionistic substructural and modal logicas based on a sequent calculus (has cut elimination for all its connectives, equational theory, and categorical semantics).</p>
<p>Modal operators are decomposed in the same way as adjoint linear logic (Benton&Wadler’94), see also Atkey’s lambda-calculus for resource logic (2004) and Reed (2009).</p>
<p>So.. how? Sequents are completely Cartesian but are annotated with a description of the context and its properties/substructurality by a first order term (a context descriptor) <img src="https://s0.wp.com/latex.php?latex=%5CGamma+%5Cvdash_a+A&bg=ffffff&fg=000000&s=0" alt="\Gamma \vdash_a A" class="latex" title="\Gamma \vdash_a A" /> where <img src="https://s0.wp.com/latex.php?latex=a&bg=ffffff&fg=000000&s=0" alt="a" class="latex" title="a" /> is this context descriptor. Context descriptors are built on “modes”. e.g., <img src="https://s0.wp.com/latex.php?latex=a%3A+A%2C+b+%3A+B%2C+c+%3A+C%2C+d+%3A+D+%5Cvdash_%7B%28a+%5Cotimes+b%29+%5Cotimes+%28c+%5Cotimes+d%29%7D+X&bg=ffffff&fg=000000&s=0" alt="a: A, b : B, c : C, d : D \vdash_{(a \otimes b) \otimes (c \otimes d)} X" class="latex" title="a: A, b : B, c : C, d : D \vdash_{(a \otimes b) \otimes (c \otimes d)} X" /> the meaning of this sequent depends on the equations/inequations on the context descriptor:, e.g.,if you have associativity of <img src="https://s0.wp.com/latex.php?latex=%5Cotimes&bg=ffffff&fg=000000&s=0" alt="\otimes" class="latex" title="\otimes" /> you get ordered logic, if you have contraction <img src="https://s0.wp.com/latex.php?latex=a+%5CRightarrow+a+%5Cotimes+a&bg=ffffff&fg=000000&s=0" alt="a \Rightarrow a \otimes a" class="latex" title="a \Rightarrow a \otimes a" /> then you get a relevant logic, etc. Bunched implication will have two kinds of nodes in the context descriptor. For a modality, unary function symbols, e.g. <img src="https://s0.wp.com/latex.php?latex=%5Ctextbf%7Br%7D%28a%29+%5Cotimes+b&bg=ffffff&fg=000000&s=0" alt="\textbf{r}(a) \otimes b" class="latex" title="\textbf{r}(a) \otimes b" /> means that the left part of the context is wrapped by a modality.</p>
<p>A subtlety of this is “weakening over weakening”, e.g., <img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma+%5Cvdash_a+B%7D%7B%5CGamma%2C+x+%3A+A+%5Cvdash_a+B%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma \vdash_a B}{\Gamma, x : A \vdash_a B}" class="latex" title="\dfrac{\Gamma \vdash_a B}{\Gamma, x : A \vdash_a B}" /> where we have weakend in the sequent calculus but the context descriptor is unchanged (i.e., we can’t use <img src="https://s0.wp.com/latex.php?latex=x+%3A+A&bg=ffffff&fg=000000&s=0" alt="x : A" class="latex" title="x : A" /> to prove <img src="https://s0.wp.com/latex.php?latex=B&bg=ffffff&fg=000000&s=0" alt="B" class="latex" title="B" />).</p>
<p>The F types give you a product structured according to a context descriptor, e.g. <img src="https://s0.wp.com/latex.php?latex=A+%5Cotimes+B+%3A%3D+F_%7Bx+%5Cotimes+y%7D+%28x+%3A+A%2C+y+%3A+B%29&bg=ffffff&fg=000000&s=0" alt="A \otimes B := F_{x \otimes y} (x : A, y : B)" class="latex" title="A \otimes B := F_{x \otimes y} (x : A, y : B)" />. All of the left rules become instances of one rule on F: <img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma%2C+%5CDelta+%5Cvdash_%7B%5Cbeta%5B%5Calpha+%2F+x%5D%7D+B%7D%7B%5CGamma%2C+x+%3A+F_%5Calpha%28%5CDelta%29+%5Cvdash+B%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma, \Delta \vdash_{\beta[\alpha / x]} B}{\Gamma, x : F_\alpha(\Delta) \vdash B}" class="latex" title="\dfrac{\Gamma, \Delta \vdash_{\beta[\alpha / x]} B}{\Gamma, x : F_\alpha(\Delta) \vdash B}" /> which gives you all your left-rules (a related dual rule gives the right sequent rules).</p>
<p>Categorical semantics is based on a cartesian 2-multicategory (see paper!)<br />
Q: is this like display logics? Yes in the sense of “punctuating” the shape of the context, but without the “display” (I don’t know what this means though).</p>
<p>This is really beautiful stuff. It looks like it could extend okay with graded modalities for coeffects, but I’ll have to go away and try it myself.</p>
<hr style="height: 4pt; color: #333;" />
<p><a name="icfp-day3-fscd-pm2"></a>(FSCD) – <b>Dinaturality between syntax and semantics</b>, Paolo Pistone</p>
<p>Multivariate functions <img src="https://s0.wp.com/latex.php?latex=F+X+X&bg=ffffff&fg=000000&s=0" alt="F X X" class="latex" title="F X X" /> and <img src="https://s0.wp.com/latex.php?latex=G+Y+Y&bg=ffffff&fg=000000&s=0" alt="G Y Y" class="latex" title="G Y Y" /> where the first parameter is contravariant and second is covariant may have <em>dinatural transformations</em> between them. (Dinaturality is weaker than naturality). Yields a family of maps <img src="https://s0.wp.com/latex.php?latex=%5Ctheta_X+%5Cin+Hom_C%28F+X+X%2C+G+X+X%29&bg=ffffff&fg=000000&s=0" alt="\theta_X \in Hom_C(F X X, G X X)" class="latex" title="\theta_X \in Hom_C(F X X, G X X)" /> which separate the contravariant and covariant actions in the coherence condition (see <a href="https://en.wikipedia.org/wiki/Dinatural_transformation">Wiki</a> for the hexagonal diagram which is the generalisation of the usual naturality property into the dinatural setting).</p>
<p>A standard approach in categorical semantics is to interpret open formula by multivariant functor <img src="https://s0.wp.com/latex.php?latex=F+X+Y+%3D+X+%5Crightarrow+Y&bg=ffffff&fg=000000&s=0" alt="F X Y = X \rightarrow Y" class="latex" title="F X Y = X \rightarrow Y" /> and proofs/terms by dinautral transformations. Due to Girard, Scedrov, Scott 1992 there is the theorem that: if <img src="https://s0.wp.com/latex.php?latex=M&bg=ffffff&fg=000000&s=0" alt="M" class="latex" title="M" /> is closed and <img src="https://s0.wp.com/latex.php?latex=%5Cvdash+M+%3A+%5Csigma&bg=ffffff&fg=000000&s=0" alt="\vdash M : \sigma" class="latex" title="\vdash M : \sigma" /> then <img src="https://s0.wp.com/latex.php?latex=M&bg=ffffff&fg=000000&s=0" alt="M" class="latex" title="M" /> is dinatural. The aim of the paper is to <em>prove the converse result</em>: if <img src="https://s0.wp.com/latex.php?latex=M&bg=ffffff&fg=000000&s=0" alt="M" class="latex" title="M" /> is closed and <img src="https://s0.wp.com/latex.php?latex=%5Cbeta%5Ceta&bg=ffffff&fg=000000&s=0" alt="\beta\eta" class="latex" title="\beta\eta" />-normal and (syntactically) dinatural then <img src="https://s0.wp.com/latex.php?latex=%5Cvdash+M+%3A+%5Csigma&bg=ffffff&fg=000000&s=0" alt="\vdash M : \sigma" class="latex" title="\vdash M : \sigma" /> (i.e., implies typability).</p>
<p>Syntactic dinaturality examples where shown (lambda terms which exhibit the dinaturality property). Imagine a transformation on a Church integer <img src="https://s0.wp.com/latex.php?latex=%28X+%5Crightarrow+X%29+%5Cxrightarrow%7Bh%7D+%28X+%5Crightarrow+X%29&bg=ffffff&fg=000000&s=0" alt="(X \rightarrow X) \xrightarrow{h} (X \rightarrow X)" class="latex" title="(X \rightarrow X) \xrightarrow{h} (X \rightarrow X)" />– this is a dinatural transformation and its dinaturality hexagon is representable by lambda terms.<span style="color: #333399;"> (I didn’t get a chance to grok all the details here and serialise them).</span></p>
<p><span style="color: #333333;">Typability of a lambda term was characterised by its tree structure and properties of its shape and size (e.g., <img src="https://s0.wp.com/latex.php?latex=%5Clambda+x_1+.+%5Cldots+%5Clambda+x_n+.+M+%3A+%5Csigma_1+%5Crightarrow+%5Cldots+%5Csigma_h+%5Crightarrow+%5Ctau&bg=ffffff&fg=000000&s=0" alt="\lambda x_1 . \ldots \lambda x_n . M : \sigma_1 \rightarrow \ldots \sigma_h \rightarrow \tau" class="latex" title="\lambda x_1 . \ldots \lambda x_n . M : \sigma_1 \rightarrow \ldots \sigma_h \rightarrow \tau" /> then $n \leq h$, there are various others; see paper).<br />
</span></p>
<p><span style="color: #333399;"><span style="color: #333333;">The theorem then extends from typability to parametricity (implies that the syntactically dinatural terms are parametric) when the type is “interpretable” (see paper for definition).</span></span></p>
<p>(<span style="color: #000080;">sorry my notes don’t go further as I got a bit lost, entirely my fault, I was flagging after lunch</span>).</p>
<hr style="height: 4pt; color: #333;" />
<p>(FSCD) – <a name="icfp-day3-fscd-pm3"></a><b>Models of Type Theory Based on Moore Paths</b>, Andrew Pitts, Ian Orton</p>
<p>In MLTT (Martin-Lof Type Theory), univalence is an extensional property (e.g., <img src="https://s0.wp.com/latex.php?latex=%28%5Cforall+x+.+f+x+%3D+g+x%29+%C2%A0%5Crightarrow+%28f+%3D+g%29&bg=ffffff&fg=000000&s=0" alt="(\forall x . f x = g x) \rightarrow (f = g)" class="latex" title="(\forall x . f x = g x) \rightarrow (f = g)" />) of types in a universe <img src="https://s0.wp.com/latex.php?latex=U&bg=ffffff&fg=000000&s=0" alt="U" class="latex" title="U" />: given <img src="https://s0.wp.com/latex.php?latex=X%2C+Y+%3A+U&bg=ffffff&fg=000000&s=0" alt="X, Y : U" class="latex" title="X, Y : U" /> every identification <img src="https://s0.wp.com/latex.php?latex=p+%3A+Id_U+X+Y&bg=ffffff&fg=000000&s=0" alt="p : Id_U X Y" class="latex" title="p : Id_U X Y" /> induces an (Id)-isomorphism <img src="https://s0.wp.com/latex.php?latex=X+%5Ccong+Y&bg=ffffff&fg=000000&s=0" alt="X \cong Y" class="latex" title="X \cong Y" />. U is univalent “if all Id-isomorphisms <img src="https://s0.wp.com/latex.php?latex=X+%5Ccong+Y&bg=ffffff&fg=000000&s=0" alt="X \cong Y" class="latex" title="X \cong Y" /> is U are induced by some identification <img src="https://s0.wp.com/latex.php?latex=P+%3A+Id_U+X+Y&bg=ffffff&fg=000000&s=0" alt="P : Id_U X Y" class="latex" title="P : Id_U X Y" />” (want this expressed in type theory) (this is a simpler restating of Voevodsky’s original definition due to Licata, Shulman, et al.). This paper: towards answering, are there models for usual type theory which contain such univalent universes?</p>
<p>Univalence is inconsistent with extensional type theory where <img src="https://s0.wp.com/latex.php?latex=p+%3A+Id_A+X+Y&bg=ffffff&fg=000000&s=0" alt="p : Id_A X Y" class="latex" title="p : Id_A X Y" /> implies <img src="https://s0.wp.com/latex.php?latex=x+%3D+y+%5Cwedge+p+%3D+%5Ctextsf%7Brefl%7D&bg=ffffff&fg=000000&s=0" alt="x = y \wedge p = \textsf{refl}" class="latex" title="x = y \wedge p = \textsf{refl}" />. Univalence is telling you that the notion of identification is richer subsuming ETT where there is an identification between two types then they are already equal, which is much smaller/thinner.</p>
<p>Need a source of models of “intensional” identification types (i.e., not the extensional ones ^^^): homotopy type theory provides a source of such models via paths: <img src="https://s0.wp.com/latex.php?latex=p+%3A+Id_A+x+y&bg=ffffff&fg=000000&s=0" alt="p : Id_A x y" class="latex" title="p : Id_A x y" /> are paths from point <img src="https://s0.wp.com/latex.php?latex=x&bg=ffffff&fg=000000&s=0" alt="x" class="latex" title="x" /> to point <img src="https://s0.wp.com/latex.php?latex=y&bg=ffffff&fg=000000&s=0" alt="y" class="latex" title="y" /> in a space <img src="https://s0.wp.com/latex.php?latex=A&bg=ffffff&fg=000000&s=0" alt="A" class="latex" title="A" /> where <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Brefl%7D_A&bg=ffffff&fg=000000&s=0" alt="\textsf{refl}_A" class="latex" title="\textsf{refl}_A" /> is a “constant path” (transporting along such a path does nothing to the element).</p>
<p>So we need to find some structure <img src="https://s0.wp.com/latex.php?latex=Path_A+x+y&bg=ffffff&fg=000000&s=0" alt="Path_A x y" class="latex" title="Path_A x y" /> for these identifications, but we need to restrict to families of type <img src="https://s0.wp.com/latex.php?latex=%28B+%5C%3B+x+%5Cmid+x+%3A+A%29&bg=ffffff&fg=000000&s=0" alt="(B \; x \mid x : A)" class="latex" title="(B \; x \mid x : A)" /> carrying structure to guarantee substitution operations (in this DTT context). “Holes” is diagrams are filled using path reversals and path composition. Solution is “Moore” paths.</p>
<p>A Moore path from <img src="https://s0.wp.com/latex.php?latex=x&bg=ffffff&fg=000000&s=0" alt="x" class="latex" title="x" /> to <img src="https://s0.wp.com/latex.php?latex=y&bg=ffffff&fg=000000&s=0" alt="y" class="latex" title="y" /> in a topological space <img src="https://s0.wp.com/latex.php?latex=X&bg=ffffff&fg=000000&s=0" alt="X" class="latex" title="X" /> is specified by a shape (its length) <img src="https://s0.wp.com/latex.php?latex=%7Cp%7C&bg=ffffff&fg=000000&s=0" alt="|p|" class="latex" title="|p|" /> in <img src="https://s0.wp.com/latex.php?latex=%5Cmathbb%7BR%7D_%2B&bg=ffffff&fg=000000&s=0" alt="\mathbb{R}_+" class="latex" title="\mathbb{R}_+" /> and an extent function <img src="https://s0.wp.com/latex.php?latex=p%40+%3A+%5Cmathbb%7BR%7D_%2B+%5Crightarrow+X&bg=ffffff&fg=000000&s=0" alt="p@ : \mathbb{R}_+ \rightarrow X" class="latex" title="p@ : \mathbb{R}_+ \rightarrow X" /> satisfying <img src="https://s0.wp.com/latex.php?latex=p%40+0+%3D+x&bg=ffffff&fg=000000&s=0" alt="p@ 0 = x" class="latex" title="p@ 0 = x" /> and <img src="https://s0.wp.com/latex.php?latex=%28%5Cforall+i+%5Cgeq+%7Cp%7C%29+.+p%40i+%3D+y&bg=ffffff&fg=000000&s=0" alt="(\forall i \geq |p|) . p@i = y" class="latex" title="(\forall i \geq |p|) . p@i = y" />. These paths have a reversal operator by truncated subtraction <img src="https://s0.wp.com/latex.php?latex=%5Ctextit%7Brev%7D%28p%40i%29+%3D+p%40%28%7Cp%7C+-+i%29&bg=ffffff&fg=000000&s=0" alt="\textit{rev}(p@i) = p@(|p| - i)" class="latex" title="\textit{rev}(p@i) = p@(|p| - i)" />. (note this is idempotent on constant paths as required, allowing holes in square paths to be filled (I had to omit the details, too many diagrams)). But topological spaces don’t give us a model of MLTT, so they replace it by an arbitrary topos (modelling extensional type theory) and instead of <img src="https://s0.wp.com/latex.php?latex=%5Cmathbb%7BR%7D&bg=ffffff&fg=000000&s=0" alt="\mathbb{R}" class="latex" title="\mathbb{R}" /> for path length/shape use any totally ordered commutative ring. All the definition/properties of Moore paths are the expressed in the internal language of the topos. This successfully gives you a category with families (see Dybjer) modelling intensional Martin-Lof type theory (with Pi). This also satisfies <em>dependent functional extensionality</em> thus its probably a good model as univalence should imply extensionality. But haven’t yet got a proof that this does contain a univalence universe (todo).</p>
<p>Q: can it just be done with semirings? Yes.</p>
<hr style="height: 4pt; color: #333;" />
<p><a name="icfp-day3-pm2-talk1"></a>(ICFP) – <strong>Theorems for Free for Free: Parametricity, With and Without Types, </strong>Amal Ahmed, Dustin Jamner, Jeremy G. Siek, Philip Wadler</p>
<p>Want gradual typing AND parametricity. There has been various work integrating polymorphic types and gradual typing (with blame) but so far there have been no (published) proofs of parametricity. This paper proposes a variant of the polymorphic blame calculus <img src="https://s0.wp.com/latex.php?latex=%5Clambda+B&bg=ffffff&fg=000000&s=0" alt="\lambda B" class="latex" title="\lambda B" /> [Ahmed, Findler, Siek, Wadler POPL’11] (fixed a few problems) and proves parametricity for it.</p>
<p>Even dynamically typed code cast to universal type should behave parametrically., e.g. consider the increment function which is typed dynamically and cast to a parametric type <img src="https://s0.wp.com/latex.php?latex=%5Clambda+x+.+x+%2B+1+%3A+%5Cast+%5CRightarrow%5Ep+%5Cforall+X+.+X+%5Crightarrow+X%29+%5Bint%5D+4&bg=ffffff&fg=000000&s=0" alt="\lambda x . x + 1 : \ast \Rightarrow^p \forall X . X \rightarrow X) [int] 4" class="latex" title="\lambda x . x + 1 : \ast \Rightarrow^p \forall X . X \rightarrow X) [int] 4" /> returns 5 but we want to raise a “blame” for the coercion <img src="https://s0.wp.com/latex.php?latex=p&bg=ffffff&fg=000000&s=0" alt="p" class="latex" title="p" /> here (coercing a monomorphic to polymorphic type is bad!).</p>
<p>The language has coercions and blame, and values can be tagged with a coercion (same for function values). Some rules:<br />
<img src="https://s0.wp.com/latex.php?latex=%28v+%3A+G+%5CRightarrow%5Ep+%5Cast+%5CRightarrow%5Eq+G%29+%5Cmapsto+v&bg=ffffff&fg=000000&s=0" alt="(v : G \Rightarrow^p \ast \Rightarrow^q G) \mapsto v" class="latex" title="(v : G \Rightarrow^p \ast \Rightarrow^q G) \mapsto v" /> (that is, casts to the dynamic type and back again cancel out).<br />
But, if you cast to a different type (tag) then blame pops out: <img src="https://s0.wp.com/latex.php?latex=%28v+%3A+G+%5CRightarrow%5Ep+%5Cast+%5CRightarrow%5Eq+G%27%29+%5Cmapsto+%5Ctextsf%7Bblame%7D+p&bg=ffffff&fg=000000&s=0" alt="(v : G \Rightarrow^p \ast \Rightarrow^q G') \mapsto \textsf{blame} p" class="latex" title="(v : G \Rightarrow^p \ast \Rightarrow^q G') \mapsto \textsf{blame} p" /> (<span style="color: #333399;">or maybe that was blame q?</span>)</p>
<p><img src="https://s0.wp.com/latex.php?latex=add+%3D+%5CLambda+X+.+%5Clambda+%28x+%3A+X%29+.+3+%2B+%28x+%3A+X+%5CRightarrow%5Ep+%5Cast+%5CRightarrow%5Eq+%5Ctextsf%7Bint%7D%29&bg=ffffff&fg=000000&s=0" alt="add = \Lambda X . \lambda (x : X) . 3 + (x : X \Rightarrow^p \ast \Rightarrow^q \textsf{int})" class="latex" title="add = \Lambda X . \lambda (x : X) . 3 + (x : X \Rightarrow^p \ast \Rightarrow^q \textsf{int})" /> this has a type that makes it look like a constant function <img src="https://s0.wp.com/latex.php?latex=%5Clambda+X+.+X+%5Crightarrow+%5Ctextsf%7Bint%7D&bg=ffffff&fg=000000&s=0" alt="\lambda X . X \rightarrow \textsf{int}" class="latex" title="\lambda X . X \rightarrow \textsf{int}" />. When executing this we want to raise blame rather than actually produce a value (e.g. for <img src="https://s0.wp.com/latex.php?latex=add+3+%5B%5Ctextsf%7Bint%7D%5D+2&bg=ffffff&fg=000000&s=0" alt="add 3 [\textsf{int}] 2" class="latex" title="add 3 [\textsf{int}] 2" />). Solution: use runtime type generation rather than substitution for type variables.</p>
<p>A type-name store <img src="https://s0.wp.com/latex.php?latex=%5CSigma&bg=ffffff&fg=000000&s=0" alt="\Sigma" class="latex" title="\Sigma" /> provides a dictionary mapping type variables to types.<br />
<img src="https://s0.wp.com/latex.php?latex=%5CSigma+%5Crhd+%28%5CLambda+X+.+v%29+%5BB%5D+%5Cmapsto+%5CSigma%2C+%5Calpha+%3A%3D+B+%5Crhd+v%5B%5Calpha%2FX%5D&bg=ffffff&fg=000000&s=0" alt="\Sigma \rhd (\Lambda X . v) [B] \mapsto \Sigma, \alpha := B \rhd v[\alpha/X]" class="latex" title="\Sigma \rhd (\Lambda X . v) [B] \mapsto \Sigma, \alpha := B \rhd v[\alpha/X]" /> i.e., instead of substituting the <img src="https://s0.wp.com/latex.php?latex=B&bg=ffffff&fg=000000&s=0" alt="B" class="latex" title="B" /> in the application, create a <span style="text-decoration: underline;">fresh type</span> name <img src="https://s0.wp.com/latex.php?latex=%5Calpha&bg=ffffff&fg=000000&s=0" alt="\alpha" class="latex" title="\alpha" /> that is then mapped to <img src="https://s0.wp.com/latex.php?latex=B&bg=ffffff&fg=000000&s=0" alt="B" class="latex" title="B" /> in the store. Then we extend this with types and coercion: <img src="https://s0.wp.com/latex.php?latex=%5CSigma+%5Crhd+%28%5CLambda+X+.+v%29+%5BB%5D+%5Cmapsto+%5CSigma%2C+%5Calpha+%3A%3D+B+%5Crhd+v%5B%5Calpha%2FX%5D+%3A+A%5B%5Calpha%2FX%5D+%5CRightarrow%5E%7B%2B%5Calpha%7D+A%5BB%2FX%5D&bg=ffffff&fg=000000&s=0" alt="\Sigma \rhd (\Lambda X . v) [B] \mapsto \Sigma, \alpha := B \rhd v[\alpha/X] : A[\alpha/X] \Rightarrow^{+\alpha} A[B/X]" class="latex" title="\Sigma \rhd (\Lambda X . v) [B] \mapsto \Sigma, \alpha := B \rhd v[\alpha/X] : A[\alpha/X] \Rightarrow^{+\alpha} A[B/X]" /> where <img src="https://s0.wp.com/latex.php?latex=%2B%5Calpha&bg=ffffff&fg=000000&s=0" alt="+\alpha" class="latex" title="+\alpha" /> (<span style="color: #ff6600;">I think!</span>) means replace occurrences of B with <img src="https://s0.wp.com/latex.php?latex=%5Calpha&bg=ffffff&fg=000000&s=0" alt="\alpha" class="latex" title="\alpha" />.</p>
<p>Casts out a polymorphic type are instantiated with the dynamic type. Casts into polymorphic types are delayed like function casts.</p>
<p>Parametricity is proved via step-indexed Kripke logical relations. The set of worlds <img src="https://s0.wp.com/latex.php?latex=W&bg=ffffff&fg=000000&s=0" alt="W" class="latex" title="W" /> are tuples of the type-name stores, and relations about when concealed values should be related. See paper: also shows soundness of the logical relations wrt. contextual equivalence. There was some cool stuff with “concealing” that I missed really.</p>
<hr style="height: 4pt; color: #333;" />
<p>(ICFP) – <a name="icfp-day3-pm2-talk2"></a><b>On Polymorphic Gradual Typing</b>, Yuu Igarashi, Taro Sekiyama, Atsushi Igarashi</p>
<p>Want to smoothly integrate polymorphism/gradual. e.g., apply dynamically-typed arguments into polmorphically typed parameters, and vice versa. Created two systems: System Fg (polymorphic gradually typed lambda cal) which translates to System Fc.</p>
<p>Want conservativity: A non-dynamically typed term in System F should be convertible into System Fg, and vice versa. The following rule shows the gradual typing “consistency” rule for application (in System Fg and gradual lambda calculus):<br />
<img src="https://s0.wp.com/latex.php?latex=%5Cdfrac%7B%5CGamma+%5Cvdash+s+%3A+T_%7B11%7D+%5Crightarrow+T_%7B12%7D+%5Cquad+%5CGamma+%5Cvdash+t+%3A+T_1+%5Cquad+T_1+%5Csim+T_%7B11%7D%7D%7B%5CGamma+%5Cvdash+s+%5C%3B+t+%3A+T_2%7D&bg=ffffff&fg=000000&s=0" alt="\dfrac{\Gamma \vdash s : T_{11} \rightarrow T_{12} \quad \Gamma \vdash t : T_1 \quad T_1 \sim T_{11}}{\Gamma \vdash s \; t : T_2}" class="latex" title="\dfrac{\Gamma \vdash s : T_{11} \rightarrow T_{12} \quad \Gamma \vdash t : T_1 \quad T_1 \sim T_{11}}{\Gamma \vdash s \; t : T_2}" /> where <img src="https://s0.wp.com/latex.php?latex=%5Csim&bg=ffffff&fg=000000&s=0" alt="\sim" class="latex" title="\sim" /> defines type consistency. Idea: extend this to include notions of consistency between polymorphic and dynamic types. An initial attempt breaks conservativity since there are ill-typed System F terms which are well-typed in System Fg then.</p>
<p>The gradual guarantee conjecture: coercions without changes to the operational semantics.</p>
<p>They introduce “type precision” (which I think replaces consistency) which is not symmetric where <img src="https://s0.wp.com/latex.php?latex=A+%5Csqsubseteq+%5Cast&bg=ffffff&fg=000000&s=0" alt="A \sqsubseteq \ast" class="latex" title="A \sqsubseteq \ast" /> – <span style="color: #0000ff;">I didn’t yet get the importance of this asymmetry</span>). But there are some issues again… a solution is to do fresh name generation to prevent non-parametric behaviour (Ahmed et al. ’11, ’17 <strong><span style="color: #0000ff;">the last talk</span></strong>) But making types less precise still changes behaviour (create blame exceptions). Their approach to fix this is to restrict the term precision relation <img src="https://s0.wp.com/latex.php?latex=%5Csqsubseteq&bg=ffffff&fg=000000&s=0" alt="\sqsubseteq" class="latex" title="\sqsubseteq" /> so that it can not be used inside of <img src="https://s0.wp.com/latex.php?latex=%5CLambda&bg=ffffff&fg=000000&s=0" alt="\Lambda" class="latex" title="\Lambda" /> (term level) and <img src="https://s0.wp.com/latex.php?latex=%5Cforall&bg=ffffff&fg=000000&s=0" alt="\forall" class="latex" title="\forall" /> (type level).<br />
Still need to prove the gradual guarantee for this new approach (and adapt the parametricity results from Ahmed et al. ’17).</p>
<hr style="height: 4pt; color: #333;" />
<p>(ICFP) – <a name="icfp-day3-pm3-talk1"></a><b>Constrained Type Families</b>, J. Garrett Morris, Richard A. Eisenberg</p>
<p>(Recall type families = type-level functions)<br />
Contribution 1: Discovery: GHC assumes that all type families are total<br />
Contribution 2: First type-safety proof with non-termination and non-linear patterns<br />
Contribution 2.5 (on the way): simplified metatheory<br />
The results are applicable to any language with partiality and type-level functions (e.g. Scala)<br />
(Haskellers leave no feature unabused!)</p>
<pre style="font-size: 10.5pt; font-family: monospace;">-- Example type family:
type family Pred n
type instance Pred (S n) = n
type instance Pred Z = Z</pre>
<p>Can define a closed version:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">type family Pred n where
Pred (S n) = n
Pred Z = Z</pre>
<p>Definition: A <span style="text-decoration: underline;">ground type</span> has no type families.<br />
Definition: A <span style="text-decoration: underline;">total type family,</span> when applied to ground types, always produces a ground type.</p>
<p><span style="text-decoration: underline;">In the new proposal: all partial type families must be associated type families.</span></p>
<pre style="font-size: 10.5pt; font-family: monospace;">type family F a -- with no instances
x = fst (5, undefined :: F Int) -- is typeable, but there are no instance of F!</pre>
<p>Instead, put it into a class:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">class CF a where
type F a
x = fst (5, undefined :: F Int) -- now get error "No instance for (CF Int)"</pre>
<p>Can use closed type families for type equality.</p>
<pre style="font-size: 10.5pt; font-family: monospace;">type family EqT a b where
EqT a a = Char
EqT a b = Bool
f :: a -> EqT a (Maybe a) -- This currently fails to type check but it should work
f _ = False</pre>
<p>Another example:</p>
<pre style="font-size: 10.5pt; font-family: monospace;">type family Maybes a
type instance Maybes a = Maybe (Maybes a)
justs = Just justs -- GHC says we can't have an infinite type a ~ Maybe a
-- But we clearly do with this type family!</pre>
<p>Why not just ban <code style="font-size: 10.5pt; font-family: monospace;">Maybes</code>? Sometimes we need recursive type families.</p>
<p>By putting (partial) type classes inside a class means we are giving a constraint that explains there should be a type at the bottom of (once we’ve evaluated) a type family. Therefore we escape the totality trap and avoid the use of bogus types.</p>
<p>Total type families need not be associated. Currently GHC has a termination checker, but it is quite weak, so often people turn it off with <code style="font-size: 10.5pt; font-family: monospace;">UndecideableInstances</code>.<br />
But, wrinkle: backwards compatibility (see paper) (lots of partial type families that are not associated).</p>
<p>Forwards compatibility: dependent types, terminating checking, is Girard’s paradox encodable?</p>
<p>This work also lets us simplify injective type family constraints (this is good, hard to understand at the moment). It makes closed type families more powerful (also great).</p>
<hr style="height: 4pt; color: #333;" />
<p>(ICFP) – <a name="icfp-day3-pm3-talk2"></a><b>Automating Sized-Type Inference for Complexity Analysis</b>, Martin Avanzini, Ugo Dal Lago</p>
<p>(for higher-order functional programs automatically). Instrument a program with a step counter by turning them into state monad which gives a dynamic count.<br />
Sized typed (e.g., sized vectors), let you type ‘reverse’ of a list as <img src="https://s0.wp.com/latex.php?latex=%5Cforall+i+.+L_i+%5C%2C+a+%5Crightarrow+L_i+%5C%2C+a&bg=ffffff&fg=000000&s=0" alt="\forall i . L_i \, a \rightarrow L_i \, a" class="latex" title="\forall i . L_i \, a \rightarrow L_i \, a" />. But inference is highly non-trivial.</p>
<p>Novel sized-type system with polymorphic recursion and arbitrary rank polymorphism <span style="text-decoration: underline;">on sizes</span> with <strong>inference procedure</strong> for the size types. Constructor definitions define size measure, e.g., for lists of nats: <img src="https://s0.wp.com/latex.php?latex=%C2%A0%5Ctextsf%7Bnil%7D+%3A+L_1&bg=ffffff&fg=000000&s=0" alt=" \textsf{nil} : L_1" class="latex" title=" \textsf{nil} : L_1" /> and <img src="https://s0.wp.com/latex.php?latex=%5Ctextsf%7Bcons%7D+%3A+%5Cforall+i+j+.+N_i+%5Crightarrow+L_j+%5Crightarrow+L_%7Bi+%2B+j+%2B+1%7D&bg=ffffff&fg=000000&s=0" alt="\textsf{cons} : \forall i j . N_i \rightarrow L_j \rightarrow L_{i + j + 1}" class="latex" title="\textsf{cons} : \forall i j . N_i \rightarrow L_j \rightarrow L_{i + j + 1}" /> (here the natural number has a size too).<br />
Can generalise size types in the application rules, (abs, app, and var rules deal with abstraction and instantiation of size parameters).</p>
<p>Theorem (soundness of sizes): (informally) the sizes attached to the a well-typed program gives you a bound on the number of execution steps that actually happen dynamically (via the state encoding). [also implies termination].</p>
<p>Inference step 1: add template annotations, e.g., <img src="https://s0.wp.com/latex.php?latex=%5Ctextit%7Breverse%7D+%3A+%5Cforall+k+.+L_k+%5Crightarrow+L_%7Bg%28k%29%7D&bg=ffffff&fg=000000&s=0" alt="\textit{reverse} : \forall k . L_k \rightarrow L_{g(k)}" class="latex" title="\textit{reverse} : \forall k . L_k \rightarrow L_{g(k)}" />. Then we need to find a concrete interpretation of <img src="https://s0.wp.com/latex.php?latex=g&bg=ffffff&fg=000000&s=0" alt="g" class="latex" title="g" />.<br />
Step 2: constraint generation, which arise from subtyping which approximate sizes, plus the constructor sizes. These collect together to constraint the meta variables introduced in step 1.<br />
Step 3: constraint solving: replace meta variable by Skolem terms. Then use an SMT solver (the constraints are non-linear integer constraints). (from a question) The solution may not be affine (but if it is then it is easier to find).</p>
<p>The system does well at inferring the types of lots of standard higher-order functions with lists. But cross product on lists produces a type which is too polymorphic: with a little more annotation you can give it a useful type.<br />
Q: What happens with quicksort? Unfortunately it can’t be done in the current system. Insertion sort is fine though.<br />
Q: What about its relation to RAML? The method is quite different (RAML uses an amortized analysis). The examples that each can handle are quite different.</p>
<p>(ICFP) – <a name="icfp-day3-pm3-talk3"></a><b>Inferring Scope through Syntactic Sugar</b>, Justin Pombrio, Shriram Krishnamurthi, Mitchell Wand</p>
<p>[<a href="https://github.com/brownplt/scope-graph">Github</a>]</p>
<p>Lots of languages use syntactic sugar, but then its hard to understand what the scoping rules are for these syntactic sweets.</p>
<p>E.g. list comprehension <code>[e | p <- L]</code> desugars to <code>map (\p . e) L</code>. The approach in this work is a procedure of <span style="text-decoration: underline;">scope inference</span> which takes these rewrite patterns (where <code>e</code> and <code>p</code> and <code>L</code> are variables/parameters). We know the scoping rules for the core language (i.e., the binding on the right) [which parameterises the scope inference algorithm]. Desugaring shouldn’t change scope (i.e., if there is path between two variables then it should be preserved). So the scope structure is homomorphic to the desugared version, but smaller. (<span style="color: #333399;">A scope preorder was defined, I didn’t get to write down the details</span>).</p>
<p>Theorem: if resugaring succeeds, then desugaring will preserve scope.</p>
<p>Evaluation: the techniques was tested on Pyret (educational language), Haskell (list comprehensions), and Scheme (all of the binding constructs in R5RS). It worked for everything apart from ‘do’ in R5RS.</p><br /> <a href="http://feeds.wordpress.com/1.0/gocomments/dorchard.wordpress.com/599/" rel="nofollow"><img src="http://feeds.wordpress.com/1.0/comments/dorchard.wordpress.com/599/" alt="" border="0" /></a> <img src="https://pixel.wp.com/b.gif?host=dorchard.wordpress.com&blog=10180011&post=599&subd=dorchard&ref=&feed=1" alt="" height="1" border="0" width="1" />Thu, 07 Sep 2017 10:46:20 +0000Sandy Maguire: Modeling Musichttp://reasonablypolymorphic.com//blog/modeling-music
http://reasonablypolymorphic.com//blog/modeling-music
<div class="main">
<article>
<header>
<h1><a href="http://reasonablypolymorphic.com/blog/modeling-music">Modeling Music</a></h1>
</header>
<p class="meta">
<span class="prev">
<a href="http://reasonablypolymorphic.com/blog/information-effects">←</a>
</span>
<span class="next">
<a href="http://reasonablypolymorphic.com/blog/recursion-schemes">→</a>
</span>
<time>September 6, 2017</time>
<span class="tags">
</span>
</p>
<div class="content">
<p>One of my long-term goals since forever has been to get good at music. I can sightread music, and I can play music by ear – arguably I can play music well. But this isn’t to say that I am good at music; I’m lacking any theory which might take me from “following the path” of music to “navigating” music.</p>
<p>Recently I took another stab at learning this stuff. Every two years or so I make an honest-to-goodness attempt at learning music theory, but inevitably run into the same problems over and over again. The problem is that I have yet to find any music education resources that communicate on my wavelength.</p>
<p>Music education usually comes in the form of “here are a bunch of facts about music; memorize them and you will now know music.” As someone who got good at math because it was the only subject he could find that didn’t require a lot of memorization, this is a frustrating situation to be in for me. Math education, in other words, presents too many theorems and too few axioms.</p>
<p>My learning style prefers to know the governing fundamentals, and derive results when they’re needed. It goes without saying that this is not the way most music theory is taught.</p>
<p>Inspired by my recent forays into learning more mathematics, I’ve had an (obvious) insight into how to learn things, and that’s to model them in systems I already understand. I’m pretty good at functional programming, so it seemed like a pretty reasonable approach.</p>
<p>I’ve still got a <em>long</em> way to go, but this post describes my first attempt at modeling music, and, vindicating my intuitions, shows how we can derive value out of this model.</p>
<h2 id="music-from-first-principles">Music from First Principles</h2>
<p>I wanted to model music, but it wasn’t immediately obviously how to actually go about doing that. I decided to write down the few facts about music theory I <em>did</em> know: there are notes.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">data</span> <span class="dt">Note</span> <span class="fu">=</span> <span class="dt">C</span> <span class="fu">|</span> <span class="dt">C'</span> <span class="fu">|</span> <span class="dt">D</span> <span class="fu">|</span> <span class="dt">D'</span> <span class="fu">|</span> <span class="dt">E</span> <span class="fu">|</span> <span class="dt">F</span> <span class="fu">|</span> <span class="dt">F'</span> <span class="fu">|</span> <span class="dt">G</span> <span class="fu">|</span> <span class="dt">G'</span><span class="fu">|</span> <span class="dt">A</span> <span class="fu">|</span> <span class="dt">A'</span> <span class="fu">|</span> <span class="dt">B</span>
<span class="kw">deriving</span> (<span class="dt">Eq</span>, <span class="dt">Ord</span>, <span class="dt">Show</span>, <span class="dt">Enum</span>, <span class="dt">Bounded</span>, <span class="dt">Read</span>)</code></pre></div>
<p>Because Haskell doesn’t let you use <code>#</code> willy-nilly, I decided to mark sharps with apostrophes.</p>
<p>I knew another fact, which is that the sharp keys can also be described as flat keys – they are <a href="https://en.wikipedia.org/wiki/Enharmonic">enharmonic</a>. I decided to describe these as pattern synonyms, which may or may not have been a good idea. Sometimes the name of the note matters, but sometimes it doesn’t, and I don’t have a great sense of when that is. I resolved to reconsider this decision if it caused issues down the road.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">{-# LANGUAGE PatternSynonyms #-}</span>
pattern <span class="dt">Ab</span> <span class="fu">=</span> <span class="dt">G'</span>
pattern <span class="dt">Bb</span> <span class="fu">=</span> <span class="dt">A'</span>
pattern <span class="dt">Db</span> <span class="fu">=</span> <span class="dt">C'</span>
pattern <span class="dt">Eb</span> <span class="fu">=</span> <span class="dt">D'</span>
pattern <span class="dt">Gb</span> <span class="fu">=</span> <span class="dt">F'</span></code></pre></div>
<p>The next thing I knew was that notes have some notion of <em>distance</em> between them. This distance is measured in <strong>semitones</strong>, which correspond to the pitch difference you can play on a piano. This distance is called an <strong>interval</strong>, and the literature has standard names for intervals of different sizes:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">data</span> <span class="dt">Interval</span>
<span class="fu">=</span> <span class="dt">Uni</span> <span class="co">-- 0 semitones</span>
<span class="fu">|</span> <span class="dt">Min2</span> <span class="co">-- 1 semitone</span>
<span class="fu">|</span> <span class="dt">Maj2</span> <span class="co">-- 2 semitones</span>
<span class="fu">|</span> <span class="dt">Min3</span> <span class="co">-- etc.</span>
<span class="fu">|</span> <span class="dt">Maj3</span>
<span class="fu">|</span> <span class="dt">Perf4</span>
<span class="fu">|</span> <span class="dt">Tri</span>
<span class="fu">|</span> <span class="dt">Perf5</span>
<span class="fu">|</span> <span class="dt">Min6</span>
<span class="fu">|</span> <span class="dt">Maj6</span>
<span class="fu">|</span> <span class="dt">Min7</span>
<span class="fu">|</span> <span class="dt">Maj7</span>
<span class="kw">deriving</span> (<span class="dt">Eq</span>, <span class="dt">Ord</span>, <span class="dt">Show</span>, <span class="dt">Enum</span>, <span class="dt">Bounded</span>, <span class="dt">Read</span>)</code></pre></div>
<p>It’s pretty obvious that intervals add in the usual way, since they’re really just names for different numbers of semitones. We can define addition over them, with the caveat that if we run out of interval names, we’ll loop back to the beginning. For example, this will mean we’ll call an octave a <code>Uni</code>son, and a 13th a <code>Perf4</code>. Since this is “correct” if you shift down an octave every time you wrap around, we decide not to worry about it:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">iAdd ::</span> <span class="dt">Interval</span> <span class="ot">-></span> <span class="dt">Interval</span> <span class="ot">-></span> <span class="dt">Interval</span>
iAdd a b <span class="fu">=</span> toEnum <span class="fu">$</span> (fromEnum a <span class="fu">+</span> fromEnum b) <span class="ot">`mod`</span> <span class="dv">12</span></code></pre></div>
<p>We can similarly define subtraction.</p>
<p>This “wrapping around” structure while adding should remind us of our group theory classes; in fact intervals are exactly the group <span class="math inline">\(\mathbb{Z}/12\mathbb{Z}\)</span> – a property shared by the hours on a clock where <span class="math inline">\(11 + 3 = 2\)</span>. That’s certainly interesting, no?</p>
<p>If intervals represent distances between notes, we should be able to subtract two notes to get an interval, and add an interval to a note to get another note.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">iBetween ::</span> <span class="dt">Note</span> <span class="ot">-></span> <span class="dt">Note</span> <span class="ot">-></span> <span class="dt">Interval</span>
iBetween a b <span class="fu">=</span> toEnum <span class="fu">$</span> (fromEnum a <span class="fu">-</span> fromEnum b) <span class="ot">`mod`</span> <span class="dv">12</span>
<span class="ot">iAbove ::</span> <span class="dt">Note</span> <span class="ot">-></span> <span class="dt">Interval</span> <span class="ot">-></span> <span class="dt">Note</span>
iAbove a b <span class="fu">=</span> toEnum <span class="fu">$</span> (fromEnum a <span class="fu">+</span> fromEnum b) <span class="ot">`mod`</span> <span class="dv">12</span></code></pre></div>
<p>Let’s give this all a try, shall we?</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> iAdd <span class="dt">Maj3</span> <span class="dt">Min3</span>
<span class="dt">Perf5</span>
<span class="fu">></span> iBetween <span class="dt">E</span> <span class="dt">C</span>
<span class="dt">Maj3</span>
<span class="fu">></span> iAbove <span class="dt">D</span> <span class="dt">Maj3</span>
<span class="dt">F'</span></code></pre></div>
<p>Looks good so far! Encouraged by our success, we can move on to trying to model a scale.</p>
<h2 id="scales">Scales</h2>
<p>This was my first stumbling block – what exactly is a scale? I can think of a few: C major, E major, Bb harmonic minor, A melodic minor, and plenty others! My first attempt was to model a scale as a <em>list of notes</em>.</p>
<p>Unfortunately, this doesn’t play nicely with our mantra of “axioms over theorems”. Represented as a list of notes, it’s hard to find the common structure between C major and D major.</p>
<p>Instead, we can model a scale as a list of <em>intervals</em>. Under this lens, all major scales will be represented identically, which is a promising sign. I didn’t know what those intervals happened to be, but I did know what C major looked like:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">cMajor ::</span> [<span class="dt">Note</span>]
cMajor <span class="fu">=</span> [<span class="dt">C</span>, <span class="dt">D</span>, <span class="dt">E</span>, <span class="dt">F</span>, <span class="dt">G</span>, <span class="dt">A</span>, <span class="dt">B</span>]</code></pre></div>
<p>We can now write a simple helper function to extract the intervals from this:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">intsFromNotes ::</span> [<span class="dt">Note</span>] <span class="ot">-></span> [<span class="dt">Interval</span>]
intsFromNotes notes <span class="fu">=</span> fmap (\x <span class="ot">-></span> x <span class="ot">`iBetween`</span> head notes) notes
<span class="ot">major ::</span> [<span class="dt">Interval</span>]
major <span class="fu">=</span> intsFromNotes cMajor</code></pre></div>
<p>To convince ourselves it works:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> major
[<span class="dt">Uni</span>,<span class="dt">Maj2</span>,<span class="dt">Maj3</span>,<span class="dt">Perf4</span>,<span class="dt">Perf5</span>,<span class="dt">Maj6</span>,<span class="dt">Maj7</span>]</code></pre></div>
<p>Seems reasonable; the presence of all those major intervals is probably why they call it a major scale. But while memorizing the intervals in a scale is likely a fruitful exercise, it’s no good to me if I want to actually <em>play</em> a scale. We can write a function to add the intervals in a scale to a tonic in order to get the actual <em>notes</em> of a scale:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">transpose ::</span> <span class="dt">Note</span> <span class="ot">-></span> [<span class="dt">Interval</span>] <span class="ot">-></span> [<span class="dt">Note</span>]
transpose n <span class="fu">=</span> fmap (iAbove n)</code></pre></div>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> transpose <span class="dt">A</span> major
[<span class="dt">A</span>,<span class="dt">B</span>,<span class="dt">C'</span>,<span class="dt">D</span>,<span class="dt">E</span>,<span class="dt">F'</span>,<span class="dt">G'</span>]</code></pre></div>
<p>Looking good!</p>
<h2 id="modes">Modes</h2>
<p>The music theory I’m actually trying to learn with all of this is jazz theory, and my jazz theory book talks a lot about modes. A mode of a scale, apparently, is playing the same notes, but starting on a different one. For example, G mixolydian is actually just the notes in C major, but starting on G (meaning it has an F♮, rather than F#).</p>
<p>By rote, we can scribe down the names of the modes:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">data</span> <span class="dt">Mode</span>
<span class="fu">=</span> <span class="dt">Ionian</span>
<span class="fu">|</span> <span class="dt">Dorian</span>
<span class="fu">|</span> <span class="dt">Phrygian</span>
<span class="fu">|</span> <span class="dt">Lydian</span>
<span class="fu">|</span> <span class="dt">Mixolydian</span>
<span class="fu">|</span> <span class="dt">Aeolian</span>
<span class="fu">|</span> <span class="dt">Locrian</span>
<span class="kw">deriving</span> (<span class="dt">Eq</span>, <span class="dt">Ord</span>, <span class="dt">Show</span>, <span class="dt">Enum</span>, <span class="dt">Bounded</span>, <span class="dt">Read</span>)</code></pre></div>
<p>If you think about it, playing the same notes as a different scale but starting on a different note sounds a lot like rotating the order of the notes you’re playing. I got <a href="https://stackoverflow.com/a/16379034/5617054">an algorithm for rotating a list off stack overflow</a>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">rotate ::</span> <span class="dt">Int</span> <span class="ot">-></span> [a] <span class="ot">-></span> [a]
rotate _ [] <span class="fu">=</span> []
rotate n xs <span class="fu">=</span> zipWith const (drop n (cycle xs)) xs</code></pre></div>
<p>which we can then use in our dastardly efforts:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">modeOf ::</span> <span class="dt">Mode</span> <span class="ot">-></span> [a] <span class="ot">-></span> [a]
modeOf mode <span class="fu">=</span> rotate (fromEnum mode)</code></pre></div>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> modeOf <span class="dt">Mixolydian</span> <span class="fu">$</span> transpose <span class="dt">C</span> major
[<span class="dt">G</span>,<span class="dt">A</span>,<span class="dt">B</span>,<span class="dt">C</span>,<span class="dt">D</span>,<span class="dt">E</span>,<span class="dt">F</span>]</code></pre></div>
<p>That has a F♮, all right. Everything seems to be proceeding according to our plan!</p>
<p>Something that annoys me about modes is that “G mixolydian” has the notes of C, not of G. This means the algorithm I need to carry out in my head to jam with my buddies goes something as follows:</p>
<ul>
<li>G mixolydian?</li>
<li>Ok, mixolydian is the fifth mode.</li>
<li>So what’s a major fifth below G?</li>
<li>It’s C!</li>
<li>What’s the C major scale?</li>
<li>OK, got it.</li>
<li>So I want to play the C major scale but starting on a different note.</li>
<li>What was I doing again?</li>
</ul>
<p>That’s a huge amount of thinking to do on a key change. Instead, what I’d prefer is to think of “mixolydian” as a transformation on G, rather than having to backtrack to C. I bet there’s an easier mapping from modes to the notes they play. Let’s see if we can’t tease it out!</p>
<p>So to figure out what are the “mode rules”, I want to compare the intervals of C major (ionian) to C whatever, and report back any which are different. As a sanity check, we know from thinking about G mixolydian that the mixolydian rules should be <code>Maj7 => Min7</code> in order to lower the F# to an F♮.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">modeRules ::</span> <span class="dt">Mode</span> <span class="ot">-></span> [(<span class="dt">Interval</span>, <span class="dt">Interval</span>)]
modeRules m <span class="fu">=</span> filter (uncurry (<span class="fu">/=</span>))
<span class="fu">.</span> zip (intsFromNotes <span class="fu">$</span> transpose <span class="dt">C</span> major)
<span class="fu">.</span> intsFromNotes
<span class="fu">.</span> transpose <span class="dt">C</span>
<span class="fu">$</span> modeOf m major</code></pre></div>
<p>What this does is construct the notes in C ionian, and then in C whatever, turns both sets into intervals, and then removes any groups which are the same. What we’re left with is pairs of intervals that have changed while moving modes.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> modeRules <span class="dt">Mixolydian</span>
[(<span class="dt">Maj7</span>,<span class="dt">Min7</span>)]
<span class="fu">></span> modeRules <span class="dt">Dorian</span>
[(<span class="dt">Maj3</span>,<span class="dt">Min3</span>), (<span class="dt">Maj7</span>,<span class="dt">Min7</span>)]</code></pre></div>
<p>Very cool. Now I’ve got something actionable to memorize, and it’s saved me a bunch of mental effort to compute on my own. My new strategy for determining D dorian is “it’s D major but with a minor 3rd and 7th”.</p>
<h2 id="practicing">Practicing</h2>
<p>My jazz book suggests that practicing every exercise along the circle of fifths would be formative. The circle of fifths is a sequence of notes you get by successively going up or down a perfect 5th starting from C. In jazz allegedly it is more valuable to go down, so we will build that:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">circleOfFifths ::</span> [<span class="dt">Note</span>]
circleOfFifths <span class="fu">=</span> iterate (<span class="ot">`iMinus`</span> <span class="dt">Perf5</span>) <span class="dt">C</span></code></pre></div>
<p>This is an infinite list, so we’d better be careful when we look at it:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> take <span class="dv">5</span> circleOfFifths
[<span class="dt">C</span>,<span class="dt">F</span>,<span class="dt">A'</span>,<span class="dt">D'</span>,<span class="dt">G'</span>]</code></pre></div>
<p>Side note, we get to every note via the circle of fifths because there are 12 distinct notes (one for each semitone on C). A major fifth, being 7 semitones, is semi-prime with 12, meaning, meaning it will never get into a smaller cycle. Math!</p>
<p>Ok, great! So now I know which notes to start my scales on. An unfortunate property of the jazz circle of fifths is that going down by fifths means you quickly get into the freaky scales they don’t teach 7 year olds. You get into the weeds where the scales start on black notes and don’t adhere to your puny human intuitions about fingerings.</p>
<p>A quick google search suggested that there is no comprehensive reference for “what’s the fingering for scale X”. However, that same search did provide me with a heuristic – “don’t use your thumb on a black note.”</p>
<p>That’s enough for me to go on! Let’s see if we can’t write a program to solve this problem for us. It wasn’t immediately obvious to me how to generate potential fingerings, but it seems like we’ll need to know which notes are black:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">isBlack ::</span> <span class="dt">Note</span> <span class="ot">-></span> <span class="dt">Bool</span>
isBlack <span class="dt">A'</span> <span class="fu">=</span> <span class="dt">True</span>
isBlack <span class="dt">C'</span> <span class="fu">=</span> <span class="dt">True</span>
isBlack <span class="dt">D'</span> <span class="fu">=</span> <span class="dt">True</span>
isBlack <span class="dt">F'</span> <span class="fu">=</span> <span class="dt">True</span>
isBlack <span class="dt">G'</span> <span class="fu">=</span> <span class="dt">True</span>
isBlack _ <span class="fu">=</span> <span class="dt">False</span></code></pre></div>
<p>For the next step, I thought I’d play it safe and hard code the list of fingering patterns for the right hand that I already know.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">fingerings ::</span> [[<span class="dt">Int</span>]]
fingerings <span class="fu">=</span> [ [<span class="dv">1</span>, <span class="dv">2</span>, <span class="dv">3</span>, <span class="dv">1</span>, <span class="dv">2</span>, <span class="dv">3</span>, <span class="dv">4</span>, <span class="dv">5</span>] <span class="co">-- C major</span>
, [<span class="dv">1</span>, <span class="dv">2</span>, <span class="dv">3</span>, <span class="dv">4</span>, <span class="dv">1</span>, <span class="dv">2</span>, <span class="dv">3</span>, <span class="dv">4</span>] <span class="co">-- F major</span>
]</code></pre></div>
<p>That’s it. That’s all the fingerings I know. Don’t judge me. It’s obvious that none of my patterns as written will avoid putting a thumb on a black key in the case of, for example, Bb major, so we’ll make a concession and say that you can start anywhere in the finger pattern you want.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">allFingerings ::</span> [[<span class="dt">Int</span>]]
allFingerings <span class="fu">=</span> <span class="kw">do</span>
amountToRotate <span class="ot"><-</span> [<span class="dv">0</span><span class="fu">..</span><span class="dv">7</span>]
fingering <span class="ot"><-</span> fingerings
pure <span class="fu">$</span> rotate amountToRotate fingering</code></pre></div>
<p>With this list of mighty potential fingerings, we’re ready to find one that fits a given scale!</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">fingeringOf ::</span> [<span class="dt">Note</span>] <span class="ot">-></span> [<span class="dt">Int</span>]
fingeringOf notes <span class="fu">=</span> head <span class="fu">$</span> <span class="kw">do</span>
fingers <span class="ot"><-</span> allFingerings
guard <span class="fu">.</span> not
<span class="fu">.</span> or
<span class="fu">.</span> fmap (\(n, f) <span class="ot">-></span> isBlack n <span class="fu">&&</span> f <span class="fu">==</span> <span class="dv">1</span>)
<span class="fu">.</span> zip notes
<span class="fu">$</span> fingers
pure fingers</code></pre></div>
<p>We can test it:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> fingeringOf <span class="fu">$</span> transpose <span class="dt">C</span> major
[<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">3</span>,<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">3</span>,<span class="dv">4</span>,<span class="dv">5</span>]
<span class="fu">></span> fingeringOf <span class="fu">$</span> transpose <span class="dt">F</span> major
[<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">3</span>,<span class="dv">4</span>,<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">3</span>,<span class="dv">4</span>]
<span class="fu">></span> fingeringOf <span class="fu">$</span> transpose <span class="dt">A'</span> major
[<span class="dv">2</span>,<span class="dv">3</span>,<span class="dv">1</span>,<span class="dv">2</span>,<span class="dv">3</span>,<span class="dv">4</span>,<span class="dv">5</span>,<span class="dv">1</span>]</code></pre></div>
<p>So it doesn’t work <em>amazingly</em>, but it does in fact find fingerings that avoid putting a thumb on a black key. We could tweak how successful this function is by putting more desirable fingerings earlier in <code>allFingerings</code>, but as a proof of concept this is good enough.</p>
<p>That’s about as far as I’ve taken this work so far, but it’s already taught me more about music theory than I’d learned in 10 years of lessons (in which, admittedly, I skipped the theory sections). More to come on this topic, probably.</p>
<p>As usual, <a href="https://github.com/isovector/jazz/">you can find the associated code on Github</a>.</p>
</div>
<p class="meta">
<span class="prev">
<a href="http://reasonablypolymorphic.com/blog/information-effects">←</a>
</span>
<span class="next">
<a href="http://reasonablypolymorphic.com/blog/recursion-schemes">→</a>
</span>
</p>
</article>
</div>Wed, 06 Sep 2017 00:00:00 +0000Alp Mestanogullari: Quick and minimal Haskell hacking with Nixhttp://alpmestan.com/posts/2017-09-06-quick-haskell-hacking-with-nix.html
http://alpmestan.com/posts/2017-09-06-quick-haskell-hacking-with-nix.html
<div class="post-content">
<p>This post will not explain in detail how <a href="https://nixos.org/nix/">Nix</a> can build haskell projects or set up development environments, but simply show one workflow that I use a lot. Please refer to <a href="https://nixos.org/nixpkgs/manual/#users-guide-to-the-haskell-infrastructure">this manual</a>, <a href="https://github.com/Gabriel439/haskell-nix#nix-and-haskell-in-production">Gabriel Gonzalez’s guide</a> and various blog posts out there to find out more about Nix and Haskell.</p>
<p>Alright, so I often end up wanting to quickly experiment with some ideas in a ghci session or a standalone Haskell module. Sometimes I can get away by simply prototyping my idea with data types and functions from base, but sometimes (most of the time, really) I want (or need) to use other libraries. I’d like to have a magic command to which I would give a ghc version and a list of haskell packages, and I would end up with an environment with ghc/ghci and all the aforementionned packages pre-installed in that ghc’s package database, so that just running <code>ghci</code> would be enough for being able to use those packages. Similarly, standalone modules could be built just with <code>ghc --make</code> for example. No need for a cabal file, cabal-install or stack.</p>
<p>Well, this is possible with Nix. The <code>nix-shell</code> program allows us to enter some environment that possibly requires downloading, building and running several things. The precise provisioning process depends a lot on what the environment consists in. It turns out that the Haskell infrastructure in Nix provides some functions out of the box for describing environments that consist in a ghc install along with some Haskell packages from Hackage. Making use of them is as simple as:</p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash">$ <span class="ex">nix-shell</span> -p <span class="st">"haskell.packages.ghc821.ghcWithPackages (pkgs: [pkgs.aeson pkgs.dlist])"</span></code></pre></div>
<p>This enters a shell with GHC 8.2.1 with aeson and dlist (and their transitive dependencies) installed in its package database. You might even fetch all those things from a binary cache with a little bit of luck!</p>
<p>The text between double quotes is a function call, in the Nix programming language. It calls a function, <code>haskell.packages.ghc821.ghcWithPackages</code> (<code>.</code> here is simply field/attribute access, like in many OO languages), provided by the standard Haskell infrastructure in <code>nixpkgs</code>, with an argument that’s itself a function: <code>pkgs: [pkgs.aeson pkgs.dlist]</code> is equivalent to (pseudo Haskell code) <code>\pkgs -> [aeson pkgs, dlist pkgs]</code> where <code>aeson</code> and <code>dlist</code> would be fields of a big record containing all Haskell packages. So this <code>ghcWithPackages</code> function lets us provision a ghc package database using a package set that it is providing us with, <code>pkgs</code>.</p>
<p><em>Note</em>: a haskell package set in nix is basically any record that provides the packages you need, where packages must be declared under a very precise shape. The one we’re using here (<code>haskell.packages.ghc821</code>) is derived from the latest stackage LTS (9.x) but we could very well be calling <code>thebestpackageset.ghcWithPackages</code> instead, provided that we define <code>thebestpackageset</code> somewhere and that its definition is valid. For example, you could simply extend the latest LTS with a few packages of yours that nix should get from github, by simply using the record extension syntax of Nix. Or you could override just a few package versions from the LTS. Or you could even put together an entire package set yourself.</p>
<p>In that new shell, you can verify that you indeed get the GHC you asked for and that you can use the said packages:</p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash">[<span class="ex">nix-shell</span>:~]$ ghc --version
<span class="ex">The</span> Glorious Glasgow Haskell Compilation System, version 8.2.1
[<span class="ex">nix-shell</span>:~]$ ghc-pkg list
<span class="ex">/nix/store/wrrw9c2dsw3d4vmfck7dfx3br33c6pd1-ghc-8.2.1-with-packages/lib/ghc-8.2.1/package.conf.d</span>
<span class="ex">Cabal-2.0.0.2</span>
<span class="ex">aeson-1.1.2.0</span>
<span class="ex">array-0.5.2.0</span>
<span class="ex">attoparsec-0.13.1.0</span>
<span class="ex">base-4.10.0.0</span>
<span class="ex">base-compat-0.9.3</span>
<span class="ex">binary-0.8.5.1</span>
<span class="ex">bytestring-0.10.8.2</span>
<span class="ex">containers-0.5.10.2</span>
<span class="ex">deepseq-1.4.3.0</span>
<span class="ex">directory-1.3.0.2</span>
<span class="ex">dlist-0.8.0.3</span>
<span class="ex">filepath-1.4.1.2</span>
<span class="ex">ghc-8.2.1</span>
<span class="ex">ghc-boot-8.2.1</span>
<span class="ex">ghc-boot-th-8.2.1</span>
<span class="ex">ghc-compact-0.1.0.0</span>
<span class="ex">ghc-prim-0.5.1.0</span>
<span class="ex">ghci-8.2.1</span>
<span class="ex">hashable-1.2.6.1</span>
<span class="ex">haskeline-0.7.4.0</span>
<span class="ex">hoopl-3.10.2.2</span>
<span class="ex">hpc-0.6.0.3</span>
<span class="ex">integer-gmp-1.0.1.0</span>
<span class="ex">integer-logarithms-1.0.2</span>
<span class="ex">old-locale-1.0.0.7</span>
<span class="ex">pretty-1.1.3.3</span>
<span class="ex">primitive-0.6.2.0</span>
<span class="ex">process-1.6.1.0</span>
<span class="ex">random-1.1</span>
<span class="ex">rts-1.0</span>
<span class="ex">scientific-0.3.5.1</span>
<span class="ex">tagged-0.8.5</span>
<span class="ex">template-haskell-2.12.0.0</span>
<span class="ex">terminfo-0.4.1.0</span>
<span class="ex">text-1.2.2.2</span>
<span class="ex">time-1.8.0.2</span>
<span class="ex">time-locale-compat-0.1.1.3</span>
<span class="ex">transformers-0.5.2.0</span>
<span class="ex">transformers-compat-0.5.1.4</span>
<span class="ex">unix-2.7.2.2</span>
<span class="ex">unordered-containers-0.2.8.0</span>
<span class="ex">uuid-types-1.0.3</span>
<span class="ex">vector-0.12.0.1</span>
<span class="ex">xhtml-3000.2.2</span>
[<span class="ex">nix-shell</span>:~]$ ghci
<span class="ex">GHCi</span>, version 8.2.1: http://www.haskell.org/ghc/ :? for help
<span class="ex">Prelude</span><span class="op">></span> import Data.Aeson
<span class="ex">Prelude</span> Data.Aeson<span class="op">></span> import Data.DList
<span class="ex">Prelude</span> Data.Aeson Data.DList<span class="op">></span></code></pre></div>
<p>Finally, you can define a very handy shell function for making it easier to spin up a new shell without having to remember the precise syntax for the <code>nix-shell</code> call above. Add this to your <code>~/.bash_profile</code>, <code>~/.bashrc</code> or what have you:</p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="kw">function</span><span class="fu"> nix-haskell()</span> <span class="kw">{</span>
<span class="kw">if [[</span> <span class="va">$#</span> <span class="ot">-lt</span> 2<span class="kw"> ]]</span>;
<span class="kw">then</span>
<span class="bu">echo</span> <span class="st">"Must provide a ghc version (e.g ghc821) and at least one package"</span>
<span class="bu">return</span> 1<span class="kw">;</span>
<span class="kw">else</span>
<span class="va">ghcver=$1</span>
<span class="va">pkgs=${@:2}</span>
<span class="bu">echo</span> <span class="st">"Starting haskell shell, ghc = </span><span class="va">$ghcver</span><span class="st">, pkgs = </span><span class="va">$pkgs</span><span class="st">"</span>
<span class="ex">nix-shell</span> -p <span class="st">"haskell.packages.</span><span class="va">$ghcver</span><span class="st">.ghcWithPackages (pkgs: with pkgs; [</span><span class="va">$pkgs</span><span class="st">])"</span>
<span class="kw">fi</span>
<span class="kw">}</span></code></pre></div>
<p>The <code>with pkgs;</code> bit simply allows us to say <code>[aeson dlist]</code> instead of <code>[pkgs.aeson pkgs.dlist]</code>, it just “opens” the <code>pkgs</code> “namespace”, like <code>RecordWildCards</code> in Haskell, when you write <code>SomeRecord{..}</code> to bring all the fields of a record in scope, with the field selector name as variable name for each field.</p>
<p>Now entering our previous shell is even simpler!</p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash">$ <span class="ex">nix-haskell</span> ghc821 aeson dlist</code></pre></div>
<p>Of course, if you need private dependencies or fancier environments, you do need to resort to writing a <code>shell.nix</code> that carefully puts everything together (not that it’s that complicated, but definitely not as simple as this <code>nix-haskell</code> command). However, for a quick hacking session or exploration of an idea, I’m not aware of a nicer solution.</p>
</div>
<div class="row">
<span class="date col-md-5 text-left">
<i class="fa fa-calendar"></i> Posted on September 6, 2017
</span>
<span style="width: 100%;" class="tags text-right">
<ul class="list-inline"><li><i class="fa fa-tags"></i></li><li><a href="http://alpmestan.com/tags/haskell.html">haskell</a></li><li><a href="http://alpmestan.com/tags/nix.html">nix</a></li></ul>
</span>
</div>Wed, 06 Sep 2017 00:00:00 +0000Douglas M. Auclair (geophf): August 2017 1HaskellADay problems and solutionstag:blogger.com,1999:blog-4650294074444534066.post-1909717347457427681
http://logicaltypes.blogspot.com/2017/09/august-2017-1haskelladay-problems-and.html
<ul><li>August 31st, 2017: Today's #haskell exercise is by way of @ahnqir. What are the <a>percentage of facebook users by country</a>? Today's #haskell solution <a>charts FB users by country</a>. GO INDIA! <div style="clear: both; text-align: center;" class="separator"><a style="margin-left: 1em; margin-right: 1em;" href="https://3.bp.blogspot.com/-7T2ei9ckEjs/WalS6ctQPcI/AAAAAAAAB0A/0iHOBYQpNuYcI0uZALR-4AJ51Hiz3Et5gCLcBGAs/s1600/fb-users-by-country.png"><img src="https://3.bp.blogspot.com/-7T2ei9ckEjs/WalS6ctQPcI/AAAAAAAAB0A/0iHOBYQpNuYcI0uZALR-4AJ51Hiz3Et5gCLcBGAs/s320/fb-users-by-country.png" height="189" border="0" width="320" /></a></div></li><li>August 4th, 2017: For today's #haskell problem we are looking at the <a href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M08/D04/Exercise.hs">caesar cipher</a> as a better rot13. And today's #haskell solution with <a href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M08/D04/Solution.hs">rot, rot, roslein rot</a> everywhere. <div style="clear: both; text-align: center;" class="separator"><iframe allowfullscreen="allowfullscreen" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/kLy_35bW5H8/0.jpg" frameborder="0" height="266" src="http://logicaltypes.blogspot.com/feeds/posts/default?feature=player_embedded" width="320"></iframe></div></li><li>August 1st, 2017: For today's #haskell problem we are <a href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M08/D01/Exercise.hs">parsing a string into word-tokens</a> and munging/demunging them with rot13! WOOT! <a href="https://github.com/geophf/1HaskellADay/blob/master/exercises/HAD/Y2017/M08/D01/Solution.hs">Munging plaintext and demunging cyphertext</a>, oh, my! </li></ul>Fri, 01 Sep 2017 12:31:37 +0000noreply@blogger.com (geophf)Mark Jason Dominus: Miscellanea about 24 puzzlestag:,2017:/math/24-puzzle-3
https://blog.plover.com/math/24-puzzle-3.html
<p>This is a collection of leftover miscellanea about twenty-four
puzzles. In case you forgot what that is:</p>
<blockquote>
<p>The puzzle «4 6 7 9 ⇒ 24»
means that one should take the numbers 4, 6, 7, and 9, and combine
them with the usual arithmetic operations of addition, subtraction,
multiplication, and division, to make the number 24. In this case the
unique solution is $$6\times\frac{7 + 9}{4}.$$ When the target number is
24, as it often is, we omit it and just write «4 6 7 9».</p>
<p>Prior articles on this topic:</p>
<ul>
<li><a href="https://blog.plover.com/math/17-puzzle.html">A simple but difficult arithmetic puzzle</a> (July 2016)</li>
<li><a href="https://blog.plover.com/math/24-puzzle.html">Solving twenty-four puzzles</a> (March 2017)</li>
<li><a href="https://blog.plover.com/math/24-puzzle-2.html">Recognizing when two arithmetic expressions are essentially the same</a> (August 2017)</li>
</ul>
</blockquote>
<h2>How many puzzles have solutions?</h2>
<p>For each value of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24T%24" />, there are 715 puzzles
«a b c d ⇒ T».
(I discussed this digression in two more earlier articles:
<a href="https://blog.plover.com/math/increasing-sequences.html">[1]</a>
<a href="https://blog.plover.com/math/increasing-sequences-2.html">[2]</a>.)
When the target <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24T%20%3d%2024%24" />, 466 of the 715 puzzles have solutions. Is this
typical? Many solutions of
«a b c d»
puzzles end with a multiplication of 6 and 4, or of 8 and 3, or
sometimes of 12 and 2—so many that one quickly learns to look for
these types of solutions right away. When <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24T%3d23%24" />, there won't be
any solutions of this type, and we might expect that relatively few
puzzles with prime targets have solutions.</p>
<p>This turns out to be the case:</p>
<p><img src="https://pic.blog.plover.com/math/24-puzzle-3/scatterplot.svg" alt="ALT attribute suggestions welcome!" /></p>
<p>The <em>x</em>-axis is the target number <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24T%24" />, with 0 at the left, 300 at right,
and vertical guide lines every 25. The <em>y</em> axis is the number of
solvable puzzles out of the maximum possible of 715, with 0 at the
bottom, 715 at the top, and horizontal guide lines every 100.</p>
<p>Dots representing prime number targets are colored black. Dots for
numbers with two prime factors (4, 6, 9, 10, 14, 15, 21, 22, etc.) are red;
dots with three, four, five, six, and seven prime factors are orange,
yellow, green, blue, and purple respectively.</p>
<p>Two countervailing trends are obvious: Puzzles with smaller targets
have more solutions, and puzzles with highly-composite targets have
more solutions. No target number larger than 24 has as many as 466
solvable puzzles.</p>
<p>These are only trends, not hard rules. For example, there are 156
solvable puzzles with the target 126 (4 prime factors) but only 93
with target 128 (7 prime factors). Why? (I don't know. Maybe because
there is some correlation with the number of <em>different</em> prime
factors? But 72, 144, and 216 have many solutions, and only two
different prime factors.)</p>
<p>The smallest target you can't hit is 417. The following
numbers 418 and 419
are also impossible. But there are 8 sets
of four digits that can be used to make 416 and 23 sets
that can be used to make 420. The largest target that
can be hit is obviously <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%246561%20%3d%209%e2%81%b4%24" />; the largest target with two
solutions is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%242916%20%3d%204%c2%b79%c2%b79%c2%b79%20%3d%206%c2%b76%c2%b79%c2%b79%24" />.</p>
<p><a href="https://pic.blog.plover.com/math/24-puzzle-3/number-of-solvable-puzzles.txt">(The raw data are available
here)</a>.</p>
<p>There is a lot more to discover here. For example, from looking at
the chart, it seems that the locally-best target numbers often have
the form <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%242%5en3%5em%24" />. What would we see if we colored the dots
according to their largest prime factor instead of according to their
number of prime factors? (I tried doing this, and it didn't look like
much, but maybe it could have been done better.)</p>
<h3>Making zero</h3>
<p>As the chart shows, 705 of the 715 puzzles of the type «a b c d ⇒ 0»,
are solvable. This suggests an interesting inverse puzzle that Toph
and I enjoyed: find four digits <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2cb%2cc%2c%20d%24" /> that <em>cannot</em> be used to
make zero. (<a href="https://pic.blog.plover.com/math/24-puzzle-3/SOLUTIONS-zeroes.txt">The answers</a>).</p>
<h2>Identifying interesting or difficult problems</h2>
<p><b>(Caution: this section contains spoilers for many of the most
interesting puzzles.)</b></p>
<p>I spent quite a while trying to get the computer to rank puzzles by
difficulty, with indifferent success.</p>
<h3>Fractions</h3>
<p>Seven puzzles require the use of fractions. One of these is the
notorious «3 3 8 8» that I mentioned before. This is probably the
single hardest of this type. The other six are:</p>
<pre><code> «1 3 4 6»
«1 4 5 6»
«1 5 5 5»
«1 6 6 8»
«3 3 7 7»
«4 4 7 7»
</code></pre>
<p>(<a href="https://pic.blog.plover.com/math/24-puzzle-3/solutions-fractions.png">Solutions to these (formatted image)</a>;
<a href="https://pic.blog.plover.com/math/24-puzzle-3/solutions-fractions.txt">solutions to these (plain text)</a>)</p>
<p>«1 5 5 5» is somewhat easier than the others, but they all follow
pretty much the same pattern. The last two are pleasantly
symmetrical.</p>
<h3>Negative numbers</h3>
<p>No puzzles require the use of negative intermediate values. This
surprised me at first, but it is not hard to see why.
Subexpressions with negative intermediate values can always
be rewritten to have positive intermediate values instead.</p>
<p>For instance,
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%243%20%c3%97%20%289%20%2b%20%283%20%2d%204%29%29%24" /> can be rewritten as
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%243%20%c3%97%20%289%20%2d%20%284%20%2d%203%29%29%24" />
and
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%285%20%2d%208%29%c3%97%281%20%2d9%29%24" />
can be rewritten as
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%288%20%2d%205%29%c3%97%289%20%2d1%29%24" />.</p>
<h3>A digression about tree shapes</h3>
<p>In one of the earlier articles I asserted that there are only two
possible shapes for the expression trees of a puzzle solution:</p>
<p align="center"></p><table><tbody><tr>
<td align="center"><img src="https://pic.blog.plover.com/math/24-puzzle-3/tree1.png" alt="(((a # b) # c) # d)" />
</td><td align="center"><img src="https://pic.blog.plover.com/math/24-puzzle-3/tree2.png" alt="((a # b) # (c # d))" />
</td></tr><tr><td align="center">Form <i>A</i> </td><td align="center">Form <i>B</i>
</td></tr></tbody></table><p></p>
<p>(Pink square nodes contain operators and green round nodes contain
numbers.)</p>
<p>Lindsey Kuper pointed out that there are five possible shapes, not
two. Of course, I was aware of this (it is a Catalan number), so what
did I mean when I said there were only two? It's because I had the
idea that any tree that wasn't already in one of those two forms could
be put into form <em>A</em> by using transformations like the ones in the
previous section.</p>
<p>For example, the expression <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%284%c3%97%28%281%2b2%29%c3%b73%29%29%24" /> isn't in either form,
but we can commute the × to get the equivalent <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28%281%2b2%29%c3%b73%29%c3%974%24" />, which
has form <em>A</em>. Sometimes one uses the associative laws, for example to
turn <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%c3%b7%20%28b%20%c3%97%20c%29%24" /> into <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%20%c3%b7%20b%29%20%c3%b7%20c%24" />.</p>
<p>But I was mistaken; not every expression can be put into either of
these forms. The expression <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%288%c3%97%289%2d%282%c2%b73%29%29%24" /> is an example.</p>
<h3>Unusual intermediate values</h3>
<p>The most interesting thing I tried was to look for puzzles whose
solutions require unusual intermediate numbers.</p>
<p>For example, the puzzle «3 4 4 4» looks easy (the other puzzles
with just 3s and 4s are all pretty easy) but it is rather tricky
because its only solution goes through the unusual intermediate number
28: <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%20%283%20%2b%204%29%20%2d%204%24" />.</p>
<p>I ranked puzzles as follows: each possible intermediate number appears in a
certain number of puzzle solutions; this is the score for that
intermediate number.
(Lower scores are better, because they represent rarer intermediate numbers.)
The score
for a single expression is the score of its rarest intermediate
value. So for example <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%20%283%20%2b%204%29%20%2d%204%24" /> has the intermediate
values 7 and 28. 7 is extremely common, and 28 is quite unusual,
appearing in only 151 solution expressions, so <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%20%283%20%2b%204%29%20%2d%204%24" />
receives a fairly low score of 151 because of the intermediate 28.</p>
<p>Then each puzzle received a difficulty
score which was the score of its <em>easiest</em> solution expression. For
example,
«2 2 3 8»
has two solutions, one (<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%288%2b3%29%c3%972%2b2%24" />) involving the quite unusual
intermediate value 22, which has a very good score of only 79. But
this puzzle doesn't
count as difficult because it also admits the obvious solution
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%248%c2%b73%c2%b7%5cfrac22%24" /> and this is the solution that gives it its extremely
bad score of 1768.</p>
<p>Under this ranking, the best-scoring twenty-four puzzles, and their
scores, were:</p>
<pre><code> «1 2 7 7» 3
* «4 4 7 7» 12
* «1 4 5 6» 13
* «3 3 7 7» 14
* «1 5 5 5» 15
«5 6 6 9» 23
«2 5 7 9» 24
«2 2 5 8» 25
«2 5 8 8» 45
«5 8 8 8» 45
«2 2 2 9» 47
* «1 3 4 6» 59
* «1 6 6 8» 59
«2 4 4 9» 151
«3 4 4 4» 151
* «3 3 8 8» 152
«6 8 8 9» 152
«2 2 2 7» 155
«2 2 5 7» 155
«2 3 7 7» 155
«2 4 7 7» 155
«2 5 5 7» 155
«2 5 7 7» 156
«4 4 8 9» 162
</code></pre>
<p>(Something is not quite right here. I think «2 5 7 7» and «2 5 5 7»
should have the same score, and I don't know why they don't. But I
don't care enough to do it over.)</p>
<p>Most of these are at least a little bit interesting. The seven
puzzles that require the use of fractions appear; I have marked them
with stars. The top item is «1 2 7 7», whose only solution goes
through the extremely rare intermediate number 49. The next items
require fractions, and the one after that is «5 6 6 9», which I found
difficult. So I think there's some
value in this procedure.</p>
<p>But is there enough value? I'm not sure. The last item on the list,
«4 4 8 9», goes through the unusual number 36. Nevertheless I don't
think it is a hard puzzle.</p>
<p>(I can also imagine that someone might see the answer to «5 6 6 9» right
off, but find «4 4 8 9» difficult. The whole exercise is subjective.)</p>
<h3>Solutions with unusual tree shapes</h3>
<p>I thought about looking for solutions that involved unusual sequences
of operations. Division is much less common than the other three
operations.</p>
<p>To get it right, one needs to normalize the form of expressions, so
that the shapes <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%20%2b%20b%29%20%2b%20%28c%20%2b%20d%29%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20%28b%20%2b%20%28c%20%2b%20d%29%29%24" /> aren't
counted separately. The Ezpr library can help here. But I didn't go
that far because the preliminary results weren't encouraging.</p>
<p>There are very few expressions totaling 24 that have the form
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%c3%b7b%29%c3%b7%28c%c3%b7d%29%24" />. But if someone gives you a puzzle with a solution in
that form, then <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%c3%97d%29%c3%b7%28b%c3%97c%29%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%c3%97d%29%20%c3%b7%20%28b%c3%b7c%29%24" /> are also
solutions, and one or another is usually very easy to see. For
example, the puzzle «1 3 8 9» has the solution <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%288%c3%b71%29%c3%b7%283%c3%b79%29%24" />,
which has an unusual form. But this is an <em>easy</em> puzzle; someone with
even a little experience will find the solution <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%248%20%c3%97%20%5cfrac93%20%c3%97%201%24" />
immediately.</p>
<p>Similarly there are relatively few solutions of the form
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%c3%b7%28%28b%2dc%29%c3%b7d%29%24" />, but they can all be transformed into
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%c3%97d%c3%b7%28b%2dc%29%24" /> which is not usually hard to find. Consider
$$\frac 8{\left(\frac{6 - 4}6\right)}.$$ This is pretty
weird-looking, but when you're trying to solve it
one of the first things you might notice is the 8, and then you would
try to turn the rest of the digits into a 3 by solving
«4 6 6 ⇒ 3»,
at which point it wouldn't take long to think of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac6%7b6%2d4%7d%24" />. Or, coming at
it from the other direction, you might see the sixes and start looking
for a way to make «4 6 8 ⇒ 4», and it wouldn't take long to think of
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac8%7b6%2d4%7d%24" />. </p>
<h3>Ezpr shape</h3>
<p>Ezprs (<a href="https://blog.plover.com/math/24-puzzle-2.html">see previous article</a>)
correspond more closely than abstract syntax trees do with our
intuitive notion of how expressions ought to work, so looking at the
shape of the Ezpr version of a solution might give better results than
looking at the shape of the expression tree. For example, one might
look at the number of nodes in the Ezpr or the depth of the Ezpr.</p>
<h3>Ad-hockery</h3>
<p>When trying to solve one of these puzzles, there are a few things I
always try first. After adding up the four numbers, I then look for
ways to make <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%248%c2%b73%2c%206%c2%b74%2c%24" /> or <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%2412%c2%b72%24" />; if that doesn't work I start
branching out looking for something of the type <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24ab%5cpm%20c%24" />.</p>
<p>Suppose we take a list of all solvable puzzles, and remove all the very
easy ones: the puzzles where one of the inputs is zero, or where one of
the inputs is 1 and there is a solution of the form <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24E%c3%971%24" />.</p>
<p>Then take the remainder and mark them as “easy” if they have solutions of
the form <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2bb%2bc%2bd%2c%208%c2%b73%2c%206%c2%b74%2c%24" /> or <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%2412%c2%b72%24" />. Also eliminate puzzles
with solutions of the type <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24E%20%2b%20%28c%20%2d%20c%29%24" /> or <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24E%c3%97%5cleft%28%5cfrac%0acc%5cright%29%24" />.</p>
<p>How many are eliminated in this way? Perhaps most? The remaining
puzzles ought to have at least intermediate difficulty, and perhaps
examining just those will suggest a way to separate them further into
two or three ranks of difficulty.</p>
<h3>I give up</h3>
<p>But by this time I have solved so many twenty-four
puzzles that I am no longer sure which ones are hard and which ones
are easy. I suspect that I have seen and tried to solve most of the
466 solvable puzzles; certainly more than half. So my brain is no
longer a reliable gauge of which puzzles are hard and which are easy.</p>
<p>Perhaps looking at puzzles with five inputs would work better for me
now. These tend to be easy, because you have more to work with. But
there are 2002 puzzles and probably some of them are hard.</p>
<h2>Close, but no cigar</h2>
<p>What's the closest you can get to 24 without hitting it exactly? The
best I could do was <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%245%c2%b75%20%2d%20%5cfrac89%24" />. Then I asked the computer,
which confirmed that this is optimal, although I felt foolish when I
saw the simpler solutions that are equally good: <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%246%c2%b74%20%5cpm%5cfrac%2019%24" />.</p>
<p>The paired solutions
$$5 × \left(4 + \frac79\right) < 24 < 7 × \left(4 - \frac59\right)$$
are very handsome.</p>
<h2>Phone app</h2>
<p>The search program that tells us when a puzzle has solutions is only
useful if we can take it with us in the car and ask it about license
plates. A phone app is wanted. <a href="https://studio.code.org/projects/applab/32u24dDEOBMCO1vGVVw1YA">I built one with <em>Code Studio</em></a>.</p>
<p><a href="https://studio.code.org/projects/public">Code Studio is great</a>. It has a nice web interface, and beginners can
write programs by dragging blocks around. It looks very much like <a href="https://scratch.mit.edu/">MIT's
<em>scratch</em> project</a>, which is much
better-known. But Code Studio is a much better tool than Scratch. In
Scratch, once you reach the limits of what it can do, you are stuck,
and there is no escape. In Code Studio when you drag around those
blocks you are actually
writing JavaScript underneath, and you can click a button and see and
edit the
underlying JavaScript code you have written.</p>
<p>Suppose you need to convert <code>A</code> to 1 and <code>B</code> to 2 and so on. Scratch
does not provide an <code>ord</code> function, so with Scratch you are pretty
much out of luck; your only choice is to write a 26-way if-else tree,
which means dragging around something like 104 stupid blocks. In Code
Studio, you can drop down the the JavaScript level and type in <code>ord</code>
to use the standard <code>ord</code> function. Then if you go back to blocks,
the <code>ord</code> will look like any other built-in function block.</p>
<p>In Scratch, if you want to use a data structure other than an array,
you are out of luck, because that is all there is. In Code Studio,
you can drop down to the JavaScript level and use or build any data
structure available in JavaScript.</p>
<p>In Scratch, if you want to
initialize the program with bulk data, say a precomputed table of the
solutions of the 466
twenty-four puzzles, you are out of luck. In Code Studio, you can
upload a CSV file with up to 1,000 records, which then becomes
available to your program as a data structure.</p>
<p>In summary, you spend a lot of your time in Scratch working around the
limitations of Scratch, and what you learn doing that is of very
limited applicability. Code Studio is real programming and if it
doesn't do exactly what you want out of the box, you can get what you
want by learning a little more JavaScript, which is likely to be
useful in other contexts for a long time to come.</p>
<p>Once you finish your Code Studio app, you can click a button to send
the URL to someone via SMS. They can follow the link in their phone's
web browser and then use the app.</p>
<p>Code Studio is what Scratch should have been. <a href="https://studio.code.org/projects/public">Check it out</a>.</p>
<h2>Thanks</h2>
<p>Thanks to everyone who contributed to this article, including:</p>
<ul>
<li>my daughters Toph and Katara</li>
<li>Shreevatsa R.</li>
<li>Dr. Lindsey Kuper</li>
<li>Darius Bacon</li>
<li>everyone else who emailed me</li>
</ul>Mon, 28 Aug 2017 03:36:00 +0000mjd@plover.com (Mark Dominus)Marcos Pividori: 3 Libraries for communicating with Mobile devices through Push Notifications.tag:blogger.com,1999:blog-1745485961030073287.post-7278418580874134808
http://gsoc2013cwithmobiledevices.blogspot.com/2013/09/3-libraries-for-communicating-with.html
I announce the result of 3 months working on the GSoC project: "Communicating with mobile devices". After reading many documentation, thinking about a good abstraction, learning about mobile apps and networks connections, and really valuable recommendations from my mentor, (I really want to thank Michael Snoyman for his availability and kindness) I finished developing 3 libraries to make it easy to send push notifications to mobile devices. They are available on Hackage:<br /><br /><b>* push-notify-0.1.0.0 : </b>This library offers a simple API for sending notifications through APNS, GCM and MPNS. <a href="http://hackage.haskell.org/package/push-notify-0.1.0.0">[1]</a><br /><br /><b>* push-notify-ccs-0.1.0.0 :</b> This library offers an API for sending/receiving notifications through CCS (XMPP - Google Cloud Messaging). <a href="http://hackage.haskell.org/package/push-notify-ccs-0.1.0.0">[2]</a><br /><br /><b>* push-notify-general-0.1.0.0 : </b>This library offers a general API for sending/receiving notifications, and handling the registration of devices on the server. It provides a general abstraction which can be used to communicate through different services as APNS, GCM, MPNS, hiding as much as possible, the differences between them. <a href="http://hackage.haskell.org/package/push-notify-general">[3]</a><br /><br />Now, it is easy to send/receive information with mobile devices, so you can easily develop server applications which interact with them.<br />So, to clarify this, as part of this project, I developed many test examples, including Android/WPhone and Yesod apps. The code is available on GitHub: <a href="https://github.com/MarcosPividori/GSoC-Communicating-with-mobile-devices">[4]</a> and they are running online on: <a href="http://gsoc.marcospividori.com.ar/">[5]</a> , so can be downloaded and tested on mobile devices. (BackAndForth Messaging and Connect4 apps)<br />Every feedback is welcome!<br /><br />[1] <a href="http://hackage.haskell.org/package/push-notify-0.1.0.0">http://hackage.haskell.org/package/push-notify-0.1.0.0</a><br />[2] <a href="http://hackage.haskell.org/package/push-notify-ccs-0.1.0.0">http://hackage.haskell.org/package/push-notify-ccs-0.1.0.0</a><br />[3] <a href="http://hackage.haskell.org/package/push-notify-general">http://hackage.haskell.org/package/push-notify-general</a><br />[4] <a href="https://github.com/MarcosPividori/GSoC-Communicating-with-mobile-devices">https://github.com/MarcosPividori/GSoC-Communicating-with-mobile-devices</a><br />[5] <a href="http://gsoc.marcospividori.com.ar/">http://gsoc.marcospividori.com.ar</a>Sat, 26 Aug 2017 22:13:59 +0000noreply@blogger.com (Marcos)Twan van Laarhoven: Traversing syntax treeshttp://twanvl.nl/blog/haskell/traversing-syntax-trees
http://twanvl.nl/blog/haskell/traversing-syntax-trees
<p>When working with syntax trees (such as in <a href="https://www.twanvl.nl/blog/hott/indexed-equality-implementation">a type theory interpreter</a>) you often want to apply some operation to all subtrees of a node, or to all nodes of a certain type. Of course you can do this easily by writing a recursive function. But then you would need to have a case for every constructor, and there can be many constructors.
</p><p>Instead of writing a big recursive function for each operation, it is often easier to use a traversal function. Which is what this post is about. In particular, I will describe my favorite way to handle such traversal, in the hope that it is useful to others as well.
</p><p>As a running example we will use the following data type, which represents expressions in a simple lambda calculus
</p><pre class="haskell"><span class="comment">-- Lambda calculus with de Bruijn indices</span>
<span class="keyword">data</span> <span class="conid">Exp</span>
<span class="keyglyph">=</span> <span class="conid">Var</span> <span class="varop">!</span><span class="conid">Int</span>
<span class="keyglyph">|</span> <span class="conid">Lam</span> <span class="conid">Exp</span>
<span class="keyglyph">|</span> <span class="conid">App</span> <span class="conid">Exp</span> <span class="conid">Exp</span>
<span class="keyglyph">|</span> <span class="conid">Global</span> <span class="conid">String</span>
<span class="keyword">deriving</span> <span class="conid">Show</span>
<div class="empty-line"></div>
<span class="varid">example<sub>1</sub></span> <span class="keyglyph">::</span> <span class="conid">Exp</span>
<span class="varid">example<sub>1</sub></span> <span class="keyglyph">=</span> <span class="conid">Lam</span> <span class="varop">$</span> <span class="conid">Var</span> <span class="num">0</span> <span class="comment">-- The identity function</span>
<div class="empty-line"></div>
<span class="varid">example<sub>2</sub></span> <span class="keyglyph">::</span> <span class="conid">Exp</span>
<span class="varid">example<sub>2</sub></span> <span class="keyglyph">=</span> <span class="conid">Lam</span> <span class="varop">$</span> <span class="conid">Lam</span> <span class="varop">$</span> <span class="conid">Var</span> <span class="num">1</span> <span class="comment">-- The const function</span>
<div class="empty-line"></div>
<span class="varid">example<sub>3</sub></span> <span class="keyglyph">::</span> <span class="conid">Exp</span>
<span class="varid">example<sub>3</sub></span> <span class="keyglyph">=</span> <span class="conid">Lam</span> <span class="varop">$</span> <span class="conid">Lam</span> <span class="varop">$</span> <span class="conid">Lam</span> <span class="varop">$</span> <span class="conid">App</span> (<span class="conid">Var</span> <span class="num">2</span>) (<span class="conid">App</span> (<span class="conid">Var</span> <span class="num">1</span>) (<span class="conid">Var</span> <span class="num">0</span>)) <span class="comment">-- Function composition</span>
</pre><p>Now, what do I mean by a traversal function?
The base library comes with the <tt><span class="conid">Traversable</span></tt> class, but that doesn't quite fit our purposes, because that class is designed for containers that can contain any type a. But expressions can only contain other sub-expressions.
Instead we need a monomorphic variant of <tt><span class="varid">traverse</span></tt> for our expression type:
</p><pre class="haskell"><span class="varid">traverseExp</span> <span class="keyglyph">::</span> <span class="conid">Applicative</span> <span class="varid">f</span> <span class="keyglyph">=></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>)
</pre><p>The idea is that <tt><span class="varid">traverseExp</span></tt> applies a given function to all direct children of an expression.
</p><p>The <a href="http://hackage.haskell.org/package/uniplate">uniplate</a> package defines a similar function, <a href="http://hackage.haskell.org/package/uniplate-1.6.12/docs/Data-Generics-Uniplate-Operations.html#v:descendM"><tt><span class="varid">descendM</span></tt></a>. But it has two problems: 1) <tt><span class="varid">descendM</span></tt> has a <tt><span class="conid">Monad</span></tt> constraint instead of <tt><span class="conid">Applicative</span></tt>, and 2) the class actually requires you to implement a <tt><span class="varid">uniplate</span></tt> method, which is more annoying to do.
</p><p>The ever intimidating <a href="http://hackage.haskell.org/package/lens">lens</a> package has a closer match in <a href="http://hackage.haskell.org/package/lens-4.15.4/docs/Control-Lens-Plated.html"><tt><span class="varid">plate</span></tt></a>. But aside from the terrible name, that function also lacks a way to keep track of bound variables.
</p><p>For a language with binders, like the lambda calculus, many operations need to know which variables are bound.
In particular, when working with de Bruijn indices, it is necessary to keep track of the number of bound variables. To do that we define
</p><pre class="haskell"><span class="keyword">type</span> <span class="conid">Depth</span> <span class="keyglyph">=</span> <span class="conid">Int</span>
<span class="comment">-- Traverse over immediate children, with depth</span>
<span class="varid">traverseExpD</span> <span class="keyglyph">::</span> <span class="conid">Applicative</span> <span class="varid">f</span> <span class="keyglyph">=></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>)
<span class="varid">traverseExpD</span> <span class="varid">_</span> <span class="varid">_</span> (<span class="conid">Var</span> <span class="varid">i</span>) <span class="keyglyph">=</span> <span class="varid">pure</span> (<span class="conid">Var</span> <span class="varid">i</span>)
<span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span> (<span class="conid">Lam</span> <span class="varid">x</span>) <span class="keyglyph">=</span> <span class="conid">Lam</span> <span class="varop"><$></span> <span class="varid">f</span> (<span class="varid">d</span><span class="varop">+</span><span class="num">1</span>) <span class="varid">x</span>
<span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span> (<span class="conid">App</span> <span class="varid">x</span> <span class="varid">y</span>) <span class="keyglyph">=</span> <span class="conid">App</span> <span class="varop"><$></span> <span class="varid">f</span> <span class="varid">d</span> <span class="varid">x</span> <span class="varop"><*></span> <span class="varid">f</span> <span class="varid">d</span> <span class="varid">y</span>
<span class="varid">traverseExpD</span> <span class="varid">_</span> <span class="varid">_</span> (<span class="conid">Global</span> <span class="varid">x</span>) <span class="keyglyph">=</span> <span class="varid">pure</span> (<span class="conid">Global</span> <span class="varid">x</span>)
</pre><p>Once we have written this function, other traversals can be defined in terms of <tt><span class="varid">traverseExpD</span></tt>
</p><pre class="haskell"><span class="comment">-- Traverse over immediate children</span>
<span class="varid">traverseExp</span> <span class="keyglyph">::</span> <span class="conid">Applicative</span> <span class="varid">f</span> <span class="keyglyph">=></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>)
<span class="varid">traverseExp</span> <span class="varid">f</span> <span class="keyglyph">=</span> <span class="varid">traverseExpD</span> (<span class="varid">const</span> <span class="varid">f</span>) <span class="num">0</span>
</pre><p>And map and fold are just traversals with a specific applicative functor, <tt><span class="conid">Identity</span></tt> and <tt class="complex"><span class="conid">Const</span> <span class="varid">a</span></tt> respectively. Recent versions of GHC are smart enough to know that it is safe to <tt><span class="varid">coerce</span></tt> from a traversal function to a mapping or folding one.
</p><pre class="haskell"><span class="comment">-- Map over immediate children, with depth</span>
<span class="varid">mapExpD</span> <span class="keyglyph">::</span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>)
<span class="varid">mapExpD</span> <span class="keyglyph">=</span> <span class="varid">coerce</span> (<span class="varid">traverseExpD</span> <span class="keyglyph">::</span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Identity</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Identity</span> <span class="conid">Exp</span>))
<div class="empty-line"></div>
<span class="comment">-- Map over immediate children</span>
<span class="varid">mapExp</span> <span class="keyglyph">::</span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>)
<span class="varid">mapExp</span> <span class="keyglyph">=</span> <span class="varid">coerce</span> (<span class="varid">traverseExp</span> <span class="keyglyph">::</span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Identity</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Identity</span> <span class="conid">Exp</span>))
<div class="empty-line"></div>
<span class="comment">-- Fold over immediate children, with depth</span>
<span class="varid">foldExpD</span> <span class="keyglyph">::</span> <span class="keyword">forall</span> <span class="varid">a</span><span class="varop">.</span> <span class="conid">Monoid</span> <span class="varid">a</span> <span class="keyglyph">=></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">a</span>) <span class="keyglyph">-></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">a</span>)
<span class="varid">foldExpD</span> <span class="keyglyph">=</span> <span class="varid">coerce</span> (<span class="varid">traverseExpD</span> <span class="keyglyph">::</span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Const</span> <span class="varid">a</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Const</span> <span class="varid">a</span> <span class="conid">Exp</span>))
<div class="empty-line"></div>
<span class="comment">-- Fold over immediate children</span>
<span class="varid">foldExp</span> <span class="keyglyph">::</span> <span class="keyword">forall</span> <span class="varid">a</span><span class="varop">.</span> <span class="conid">Monoid</span> <span class="varid">a</span> <span class="keyglyph">=></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">a</span>) <span class="keyglyph">-></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">a</span>)
<span class="varid">foldExp</span> <span class="keyglyph">=</span> <span class="varid">coerce</span> (<span class="varid">traverseExp</span> <span class="keyglyph">::</span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Const</span> <span class="varid">a</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Const</span> <span class="varid">a</span> <span class="conid">Exp</span>))
</pre><p>After doing all this work, it is easy to answer questions like "how often is a variable used?"
</p><pre class="haskell"><span class="varid">varCount</span> <span class="keyglyph">::</span> <span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Sum</span> <span class="conid">Int</span>
<span class="varid">varCount</span> <span class="varid">i</span> (<span class="conid">Var</span> <span class="varid">j</span>)
<span class="keyglyph">|</span> <span class="varid">i</span> <span class="varop">==</span> <span class="varid">j</span> <span class="keyglyph">=</span> <span class="conid">Sum</span> <span class="num">1</span>
<span class="varid">varCount</span> <span class="varid">i</span> <span class="varid">x</span> <span class="keyglyph">=</span> <span class="varid">foldExpD</span> <span class="varid">varCount</span> <span class="varid">i</span> <span class="varid">x</span>
</pre><p>or "what is the set of all free variables?"
</p><pre class="haskell"><span class="varid">freeVars</span> <span class="keyglyph">::</span> <span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Set</span> <span class="conid">Int</span>
<span class="varid">freeVars</span> <span class="varid">d</span> (<span class="conid">Var</span> <span class="varid">i</span>)
<span class="keyglyph">|</span> <span class="varid">i</span> <span class="varop"><</span> <span class="varid">d</span> <span class="keyglyph">=</span> <span class="conid">Set</span><span class="varop">.</span><span class="varid">empty</span> <span class="comment">-- bound variable</span>
<span class="keyglyph">|</span> <span class="varid">otherwise</span> <span class="keyglyph">=</span> <span class="conid">Set</span><span class="varop">.</span><span class="varid">singleton</span> (<span class="varid">i</span> <span class="varop">-</span> <span class="varid">d</span>) <span class="comment">-- free variable</span>
<span class="varid">freeVars</span> <span class="varid">d</span> <span class="varid">x</span> <span class="keyglyph">=</span> <span class="varid">foldExpD</span> <span class="varid">freeVars</span> <span class="varid">d</span> <span class="varid">x</span>
</pre><p>Or to perform (silly) operations like changing all globals to lower case
</p><pre class="haskell"><span class="varid">lowerCase</span> <span class="keyglyph">::</span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>
<span class="varid">lowerCase</span> (<span class="conid">Global</span> <span class="varid">x</span>) <span class="keyglyph">=</span> <span class="conid">Global</span> (<span class="varid">map</span> <span class="varid">toLower</span> <span class="varid">x</span>)
<span class="varid">lowerCase</span> <span class="varid">x</span> <span class="keyglyph">=</span> <span class="varid">mapExp</span> <span class="varid">lowerCase</span> <span class="varid">x</span>
</pre><p>These functions follows a common pattern of specifying how a particular constructor, in this case <tt><span class="conid">Var</span></tt> or <tt><span class="conid">Global</span></tt>, is handled, while for all other constructors traversing over the child expressions.
</p><p>As another example, consider substitution, a very important operation on syntax trees.
In its most general form, we can combine substitution with raising expressions to a larger context (also called weakening).
And we should also consider leaving the innermost, bound, variables alone. This means that there are three possibilities for what to do with a variable.
</p><pre class="haskell"><span class="varid">substRaiseByAt</span> <span class="keyglyph">::</span> <span class="listcon">[</span><span class="conid">Exp</span><span class="listcon">]</span> <span class="keyglyph">-></span> <span class="conid">Int</span> <span class="keyglyph">-></span> <span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>
<span class="varid">substRaiseByAt</span> <span class="varid">ss</span> <span class="varid">r</span> <span class="varid">d</span> (<span class="conid">Var</span> <span class="varid">i</span>)
<span class="keyglyph">|</span> <span class="varid">i</span> <span class="varop"><</span> <span class="varid">d</span> <span class="keyglyph">=</span> <span class="conid">Var</span> <span class="varid">i</span> <span class="comment">-- A bound variable, leave it alone</span>
<span class="keyglyph">|</span> <span class="varid">i</span><span class="varop">-</span><span class="varid">d</span> <span class="varop"><</span> <span class="varid">length</span> <span class="varid">ss</span> <span class="keyglyph">=</span> <span class="varid">raiseBy</span> <span class="varid">d</span> (<span class="varid">ss</span> <span class="varop">!!</span> (<span class="varid">i</span><span class="varop">-</span><span class="varid">d</span>)) <span class="comment">-- substitution</span>
<span class="keyglyph">|</span> <span class="varid">otherwise</span> <span class="keyglyph">=</span> <span class="conid">Var</span> (<span class="varid">i</span> <span class="varop">-</span> <span class="varid">length</span> <span class="varid">ss</span> <span class="varop">+</span> <span class="varid">r</span>) <span class="comment">-- free variable, raising</span>
<span class="varid">substRaiseByAt</span> <span class="varid">ss</span> <span class="varid">r</span> <span class="varid">d</span> <span class="varid">x</span> <span class="keyglyph">=</span> <span class="varid">mapExpD</span> (<span class="varid">substRaiseByAt</span> <span class="varid">ss</span> <span class="varid">r</span>) <span class="varid">d</span> <span class="varid">x</span>
</pre><p>Similarly to <tt><span class="varid">varCount</span></tt>, we use <tt><span class="varid">mapExpD</span></tt> to handle all constructors besides variables.
Plain substitution and raising are just special cases.
</p><pre class="haskell"><span class="comment">-- Substitute the first few free variables, weaken the rest</span>
<span class="varid">substRaiseBy</span> <span class="keyglyph">::</span> <span class="listcon">[</span><span class="conid">Exp</span><span class="listcon">]</span> <span class="keyglyph">-></span> <span class="conid">Int</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>
<span class="varid">substRaiseBy</span> <span class="varid">ss</span> <span class="varid">r</span> <span class="keyglyph">=</span> <span class="varid">substRaiseByAt</span> <span class="varid">ss</span> <span class="varid">r</span> <span class="num">0</span>
<div class="empty-line"></div>
<span class="varid">raiseBy</span> <span class="keyglyph">::</span> <span class="conid">Int</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>
<span class="varid">raiseBy</span> <span class="varid">r</span> <span class="keyglyph">=</span> <span class="varid">substRaiseBy</span> <span class="listcon">[</span><span class="listcon">]</span> <span class="varid">r</span>
<div class="empty-line"></div>
<span class="varid">subst</span> <span class="keyglyph">::</span> <span class="listcon">[</span><span class="conid">Exp</span><span class="listcon">]</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="conid">Exp</span>
<span class="varid">subst</span> <span class="varid">ss</span> <span class="keyglyph">=</span> <span class="varid">substRaiseBy</span> <span class="varid">ss</span> <span class="num">0</span>
</pre><pre class="ghci"><span class="input">λ></span> <span class="varid">raiseBy</span> <span class="num">2</span> (<span class="conid">App</span> (<span class="conid">Var</span> <span class="num">1</span>) (<span class="conid">Var</span> <span class="num">2</span>))
<span class="conid">App</span> (<span class="conid">Var</span> <span class="num">3</span>) (<span class="conid">Var</span> <span class="num">4</span>)
<div class="empty-line"></div>
<span class="input">λ></span> <span class="varid">subst</span> <span class="listcon">[</span><span class="conid">Global</span> <span class="str">"x"</span><span class="listcon">]</span> (<span class="conid">App</span> (<span class="conid">Var</span> <span class="num">0</span>) (<span class="conid">Lam</span> (<span class="conid">Var</span> <span class="num">0</span>)))
<span class="conid">App</span> (<span class="conid">Global</span> <span class="str">"x"</span>) (<span class="conid">Lam</span> (<span class="conid">Var</span> <span class="num">0</span>))
<div class="empty-line"></div>
<span class="input">λ></span> <span class="varid">substRaiseBy</span> <span class="listcon">[</span><span class="conid">App</span> (<span class="conid">Global</span> <span class="str">"x"</span>) (<span class="conid">Var</span> <span class="num">0</span>)<span class="listcon">]</span> <span class="num">2</span> <span class="varop">$</span> <span class="conid">App</span> (<span class="conid">Lam</span> (<span class="conid">App</span> (<span class="conid">Var</span> <span class="num">1</span>) (<span class="conid">Var</span> <span class="num">0</span>))) (<span class="conid">Var</span> <span class="num">2</span>)
<span class="conid">App</span> (<span class="conid">Lam</span> (<span class="conid">App</span> (<span class="conid">App</span> (<span class="conid">Global</span> <span class="str">"x"</span>) (<span class="conid">Var</span> <span class="num">1</span>)) (<span class="conid">Var</span> <span class="num">0</span>))) (<span class="conid">Var</span> <span class="num">3</span>)
</pre><p>As a slight generalization, it can also make sense to put <tt><span class="varid">traverseExpD</span></tt> into a type class. That way we can traverse over the subexpressions inside other data types. For instance, if the language uses a separate data type for case alternatives, we might write
</p><pre class="haskell"><span class="keyword">data</span> <span class="conid">Exp</span>
<span class="keyglyph">=</span> <span class="varop">...</span>
<span class="keyglyph">|</span> <span class="conid">Case</span> <span class="listcon">[</span><span class="conid">Alt</span><span class="listcon">]</span>
<div class="empty-line"></div>
<span class="keyword">data</span> <span class="conid">Alt</span>
<span class="keyglyph">=</span> <span class="conid">Alt</span> <span class="conid">Pat</span> <span class="conid">Exp</span>
<div class="empty-line"></div>
<span class="keyword">class</span> <span class="conid">TraverseExp</span> <span class="varid">a</span> <span class="keyword">where</span>
<span class="varid">traverseExpD</span> <span class="keyglyph">::</span> <span class="conid">Applicative</span> <span class="varid">f</span> <span class="keyglyph">=></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="conid">Depth</span> <span class="keyglyph">-></span> <span class="varid">a</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="varid">a</span>)
<div class="empty-line"></div>
<span class="keyword">instance</span> <span class="conid">TraverseExp</span> <span class="varid">a</span> <span class="keyglyph">=></span> <span class="conid">TraverseExp</span> <span class="listcon">[</span><span class="varid">a</span><span class="listcon">]</span> <span class="keyword">where</span>
<span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span> <span class="keyglyph">=</span> <span class="varid">traverse</span> (<span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span>)
<div class="empty-line"></div>
<span class="keyword">instance</span> <span class="conid">TraverseExp</span> <span class="conid">Exp</span> <span class="keyword">where</span>
<span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span> <span class="varop">...</span>
<span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span> (<span class="conid">Case</span> <span class="varid">xs</span>) <span class="keyglyph">=</span> <span class="conid">Case</span> <span class="varop"><$></span> <span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span> <span class="varid">xs</span>
<div class="empty-line"></div>
<span class="keyword">instance</span> <span class="conid">TraverseExp</span> <span class="conid">Alt</span> <span class="keyword">where</span>
<span class="varid">traverseExpD</span> <span class="varid">f</span> <span class="varid">d</span> (<span class="conid">Alt</span> <span class="varid">x</span> <span class="varid">y</span>) <span class="keyglyph">=</span> <span class="conid">Alt</span> <span class="varid">x</span> <span class="varop"><$></span> <span class="varid">traverseExpD</span> <span class="varid">f</span> (<span class="varid">d</span> <span class="varop">+</span> <span class="varid">varsBoundByPat</span> <span class="varid">x</span>) <span class="varid">y</span>
</pre><p>Another variation is to track other things besides the number of bound variables. For example we might track the names and types of bound variables for better error messages. And with a type class it is possible to track different aspects of bindings as needed,
</p><pre class="haskell"><span class="keyword">class</span> <span class="conid">Env</span> <span class="varid">env</span> <span class="keyword">where</span>
<span class="varid">extend</span> <span class="keyglyph">::</span> <span class="conid">VarBinding</span> <span class="keyglyph">-></span> <span class="varid">env</span> <span class="keyglyph">-></span> <span class="varid">env</span>
<div class="empty-line"></div>
<span class="keyword">instance</span> <span class="conid">Env</span> <span class="conid">Depth</span> <span class="keyword">where</span>
<span class="varid">extend</span> <span class="varid">_</span> <span class="keyglyph">=</span> (<span class="varop">+</span><span class="num">1</span>)
<div class="empty-line"></div>
<span class="keyword">instance</span> <span class="conid">Env</span> <span class="listcon">[</span><span class="conid">VarBinding</span><span class="listcon">]</span> <span class="keyword">where</span>
<span class="varid">extend</span> <span class="keyglyph">=</span> <span class="listcon">(:)</span>
<div class="empty-line"></div>
<span class="keyword">instance</span> <span class="conid">Env</span> () <span class="keyword">where</span>
<span class="varid">extend</span> <span class="varid">_</span> <span class="varid">_</span> <span class="keyglyph">=</span> ()
<div class="empty-line"></div>
<span class="varid">traverseExpEnv</span> <span class="keyglyph">::</span> <span class="conid">Applicative</span> <span class="varid">f</span> <span class="keyglyph">=></span> (<span class="varid">env</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>) <span class="keyglyph">-></span> (<span class="varid">env</span> <span class="keyglyph">-></span> <span class="conid">Exp</span> <span class="keyglyph">-></span> <span class="varid">f</span> <span class="conid">Exp</span>)
<span class="varid">traverseExpEnv</span> <span class="varid">f</span> <span class="varid">env</span> (<span class="conid">Lam</span> <span class="varid">name</span> <span class="varid">x</span>) <span class="keyglyph">=</span> <span class="conid">Lam</span> <span class="varop"><$></span> <span class="varid">f</span> (<span class="varid">extend</span> <span class="varid">name</span> <span class="varid">env</span>) <span class="varid">x</span>
<span class="varid">traverseExpEnv</span> <span class="varid">f</span> <span class="varid">env</span> <span class="varop">...</span>
</pre><p>Overall, I have found that after writing <tt><span class="varid">traverseExpD</span></tt> once, I rarely have to look at all constructors again. I can just handle the default cases by traversing the children.
</p><p>A nice thing about this pattern is that it is very efficient. The <tt><span class="varid">traverseExpD</span></tt> function is not recursive, which means that the compiler can inline it. So after optimization, a function like <tt><span class="varid">lowerCase</span></tt> or <tt><span class="varid">varCount</span></tt> is exactly what you would have written by hand.
</p>Wed, 23 Aug 2017 00:26:00 +0000Oliver Charles: Providing an API for extensible-effects and monad transformershttp://ocharles.org.uk/blog/posts/2017-08-23-extensible-effects-and-transformers.html
http://ocharles.org.uk/blog/posts/2017-08-23-extensible-effects-and-transformers.html
<p>I was recently working on a small little project - a client API for the <a href="https://listenbrainz.org">ListenBrainz</a> project. Most of the details aren’t particularly interesting - it’s just a HTTP client library to a REST-like API with JSON. For the implementation, I let Servant and aeson do most of the heavy lifting, but I got stuck when considering what final API to give to <em>my</em> users.</p>
<p>Obviously, interacting with ListenBrainz requires some sort of IO so whatever API I will be offering has to live within some sort of monad. Currently, there are three major options:</p>
<ol style="">
<li><p><em>Supply an API targetting a concrete monad stack.</em></p>
<p>Under this option, our API would have types such as</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">submitListens ::</span> <span class="fu">...</span> <span class="ot">-></span> <span class="dt">M</span> ()
<span class="ot">getListens ::</span> <span class="fu">...</span> <span class="ot">-></span> <span class="dt">M</span> <span class="dt">Listens</span></code></pre></div>
<p>where <code>M</code> is some <em>particular</em> monad (or monad transformer).</p></li>
<li><p><em>Supply an API using type classes</em></p>
<p>This is the <code>mtl</code> approach. Rather than choosing which monad my users have to work in, my API can be polymorphic over monads that support accessing the ListenBrainz API. This means my API is more like:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">submitListens ::</span> <span class="dt">MonadListenBrainz</span> m <span class="ot">=></span> <span class="fu">...</span> <span class="ot">-></span> m ()
<span class="ot">getListens ::</span> <span class="dt">MonadListenBrainz</span> m <span class="ot">=></span> <span class="fu">...</span> <span class="ot">-></span> m <span class="dt">Listens</span></code></pre></div></li>
<li><p><em>Use an extensible effects framework.</em></p>
<p>Extensible effects are a fairly new entry, that are something of a mix of the above options. We target a family of concrete monads - <code>Eff</code> - but the extensible effects framework lets our effect (querying ListenBrainz) seamlessly compose with other effects. Using <code>freer-effects</code>, our API would be:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">submitListens ::</span> <span class="dt">Member</span> <span class="dt">ListenBrainzAPICall</span> effects <span class="ot">=></span> <span class="fu">...</span> <span class="ot">-></span> <span class="dt">Eff</span> effects ()
<span class="ot">getListens ::</span> <span class="dt">Member</span> <span class="dt">ListenBrainzAPICall</span> effects <span class="ot">=></span> <span class="fu">...</span> <span class="ot">-></span> <span class="dt">Eff</span> effects <span class="dt">Listens</span></code></pre></div></li>
</ol>
<p>So, which do we choose? Evaluating the options, I have some concerns.</p>
<p>For option one, we impose pain on all our users who want to use a different monad stack. It’s unlikely that you’re application is going to be written soley to query ListenBrainz, which means client code becomes littered with <code>lift</code>. You may write that off as syntatic, but there is another problem - we have committed to an interpretation strategy. Rather than describing API calls, my library now skips directly to prescribing how to run API calls. However, it’s entirely possible that you want to intercept these calls - maybe introducing a caching layer or additional logging. Your only option is to duplicate my API into your own project and wrap each function call and then change your program to use your API rather than mine. Essentially, the program itself is no longer a first class value that you can transform.</p>
<p>Extensible effects gives us a solution to both of the above. The use of the <code>Member</code> type class automatically reshuffles effects so that multiple effects can be combined without syntatic overhead, and we only commit to an interpretation strategy when we actually run the program. <code>Eff</code> is essentially a <em>free monad</em>, which captures the syntax tree of effects, rather than the result of their execution.</p>
<p>Sounds good, but extensible effects come with (at least) two problems that make me hesistant: they are experimental and esoteric, and it’s unclear that they are performant. By using <em>only</em> extensible effects, I am forcing an extensible effects framework on my users, and I’d rather not dictate that. Of course, extensible effects can be composed with traditional monad transformers, but I’ve still imposed an unnecessary burden on my users.</p>
<p>So, what do we do? Well, as Old El Paso has taught us: why don’t we have both?</p>
<p>It’s trivial to actually support both a monad transformer stack <em>and</em> extensible effects by using an <code>mtl</code> type class. As I argue in <a href="https://ocharles.org.uk/posts/2016-01-26-transformers-free-monads-mtl-laws.html">Monad transformers, free monads, mtl, laws and a new approach</a>, I think the best pattern for an <code>mtl</code> class is to be a monad homomorphism from a program description, and often a free monad is a fine choice to lift:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">class</span> <span class="dt">Monad</span> m <span class="ot">=></span> <span class="dt">MonadListenBrainz</span> m <span class="kw">where</span>
<span class="ot"> liftListenBrainz ::</span> <span class="dt">Free</span> f a <span class="ot">-></span> m a</code></pre></div>
<p>But what about <code>f</code>? As observed earlier, extensible effects are basically free monads, so we can actually share the same implementation. For <code>freer-effects</code>, we might describe the ListenBrainz API with a GADT such as:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">data</span> <span class="dt">ListenBrainzAPICall</span> returns <span class="kw">where</span>
<span class="dt">GetListens</span><span class="ot"> ::</span> <span class="fu">...</span> <span class="ot">-></span> <span class="dt">ListenBrainzAPICall</span> <span class="dt">Listens</span>
<span class="dt">SubmitListens</span><span class="ot"> ::</span> <span class="fu">...</span> <span class="ot">-></span> <span class="dt">ListenBrainzAPICall</span> ()</code></pre></div>
<p>However, this isn’t a functor - it’s just a normal data type. In order for <code>Free f a</code> to actually be a monad, we need <code>f</code> to be a functor. We could rewrite <code>ListenBrainzAPICall</code> into a functor, but it’s even easier to just fabricate a functor for free - and that’s exactly <a href="https://www.reddit.com/r/haskelltil/comments/4ea7er/coyoneda_is_just_the_free_functor/">what <code>Coyoneda</code> will do</a>. Thus our <code>mtl</code> type class becomes:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">class</span> <span class="dt">Monad</span> m <span class="ot">=></span> <span class="dt">MonadListenBrainz</span> m <span class="kw">where</span>
<span class="ot"> liftListenBrainz ::</span> <span class="dt">Free</span> (<span class="dt">Coyoneda</span> <span class="dt">ListenBrainzAPICall</span>) a <span class="ot">-></span> m a </code></pre></div>
<p>We can now provide an implementation in terms of a monad transformer:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">instance</span> <span class="dt">Monad</span> m <span class="ot">=></span> <span class="dt">MonadListenBrainz</span> (<span class="dt">ListenBrainzT</span> m)
liftListenBrainz f <span class="fu">=</span>
iterM (join <span class="fu">.</span> lowerCoyoneda <span class="fu">.</span> hoistCoyoneda go)
<span class="kw">where</span>
<span class="ot"> go ::</span> <span class="dt">ListenBrainzAPICall</span> a <span class="ot">-></span> <span class="dt">ListenBrainzT</span> m a</code></pre></div>
<p>or extensible effects:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">instance</span> <span class="dt">Member</span> <span class="dt">ListenBrainzAPICall</span> effs <span class="ot">=></span> <span class="dt">MonadListenBrainz</span> (<span class="dt">Eff</span> effs) <span class="kw">where</span>
liftListenBrainz f <span class="fu">=</span> iterM (join <span class="fu">.</span> lowerCoyoneda <span class="fu">.</span> hoistCoyoneda send) f </code></pre></div>
<p>or maybe directly to a free monad for later inspection:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">instance</span> <span class="dt">MonadListenBrainz</span> (<span class="dt">Free</span> (<span class="dt">Coyoneda</span> <span class="dt">ListenBrainzAPICall</span>)) <span class="kw">where</span>
liftListenBrainz <span class="fu">=</span> id</code></pre></div>
<p>For the actual implementation of performing the API call, I work with a concrete monad transformer stack:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">performAPICall ::</span> <span class="dt">Manager</span> <span class="ot">-></span> <span class="dt">ListenBrainzAPICall</span> a <span class="ot">-></span> <span class="dt">IO</span> (<span class="dt">Either</span> <span class="dt">ServantError</span> a)</code></pre></div>
<p>which both my extensible effects “run” function calls, or the <code>go</code> function in the <code>iterM</code> call for <code>ListenBrainzT</code>’s <code>MonadListenBrainz</code> instance.</p>
<p>In conclusion, I’m able to offer my users a choice of either:</p>
<ul>
<li>a traditional monad transformer approach, which doesn’t commit to a particular intepretation strategy by using an <code>mtl</code> type class</li>
<li>extensible effects</li>
</ul>
<p>All without extra syntatic burden, a complicated type class, or duplicating the implementation.</p>
<p>You can see the final implemantion of <a href="https://hackage.haskell.org/package/listenbrainz-client-1.0.1/docs/src/Web-ListenBrainz.html"><code>listenbrainz-client</code> here</a>.</p>
<h3 id="bonus---what-about-the-readert-pattern">Bonus - what about the ReaderT pattern?</h3>
<p>The <a href="https://www.fpcomplete.com/blog/2017/06/readert-design-pattern">ReaderT design pattern</a> has been mentioned recently, so where does this fit in? There are two options if we wanted to follow this pattern:</p>
<ul>
<li>We require a HTTP <code>Manager</code> in our environment, and commit to using this. This has all the problems of providing a concrete monad transformer stack - we are committing to an interpretation.</li>
<li>We require a family of functions that explain how to perform each API call. This kind of like a <a href="http://r6.ca/blog/20140210T181244Z.html">van Laarhoven free monad</a>, or really just explicit dictionary passing. I don’t see this really gaining much on abstracting with type classes.</li>
</ul>
<p>I don’t feel like the ReaderT design pattern offers anything that isn’t already dealt with above.</p>Wed, 23 Aug 2017 00:00:00 +0000Manuel M T Chakravarty: Two months back, I gave my talk “Do-It-Yourself Functional...http://justtesting.org/post/164441753196
http://justtesting.org/post/164441753196
<iframe allowfullscreen="allowfullscreen" frameborder="0" height="225" mozallowfullscreen="mozallowfullscreen" src="https://player.vimeo.com/video/222165406?title=0&amp;byline=0&amp;portrait=0" title="Do-It-Yourself Functional Reactive Programming" webkitallowfullscreen="webkitallowfullscreen" width="400"></iframe><br /><br /><p>Two months back, I gave my talk “Do-It-Yourself Functional Reactive Programming” at the <a href="http://www.sydneycocoaheads.com">Sydney CocoaHeads meetup</a>. I am explaining what FRP is all about, how to easily implement an FRP library in Swift, and how to use it in an iPhone app.</p>Mon, 21 Aug 2017 11:36:23 +0000Mark Jason Dominus: Recognizing when two arithmetic expressions are essentially the sametag:,2017:/math/24-puzzle-2
https://blog.plover.com/math/24-puzzle-2.html
<p>[ Warning: The math formatting in the RSS / Atom feed for this article is badly mutilated. I suggest you <a href="https://blog.plover.com/math/24-puzzle-2.html">read the article on my blog</a>. ]</p>
<blockquote>
<p>In this article, I discuss “twenty-four puzzles”. The puzzle «4 6 7 9 ⇒ 24»
means that one should take the numbers 4, 6, 7, and 9, and combine
them with the usual arithmetic operations of addition, subtraction,
multiplication, and division, to make the number 24. In this case the
unique solution is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%246%c2%b7%5cfrac%7b7%20%2b%209%7d%7b4%7d%24" />.</p>
<p>When the target number after the <code>⇒</code> is 24, as it often is, we omit
it and just write «4 6 7 9». Every example in this article has
target number 24.</p>
<p>This is a continuation of my previous articles on this
topic:</p>
<ul>
<li><a href="https://blog.plover.com/math/17-puzzle.html">A simple but difficult arithmetic puzzle</a> (July 2016)</li>
<li><a href="https://blog.plover.com/math/24-puzzle.html">Solving twenty-four puzzles</a> (March 2017)</li>
</ul>
</blockquote>
<p>My first cut at writing a solver for twenty-four puzzles was a straightforward
search program. It had a couple of hacks in it to cut down the search
space by recognizing that <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2bE%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24E%2ba%24" /> are the same, but other
than that there was nothing special about it and
<a href="https://blog.plover.com/math/24-puzzle.html">I've discussed it before</a>.</p>
<p>It would quickly and accurately report whether any particular twenty-four
puzzle was solvable, but as it turned out that wasn't quite good
enough. The original motivation for the program was this: Toph and I
play this game in the car. Pennsylvania license plates have three
letters and four digits, and if we see a license plate <code>FBV 2259</code> we
try to solve «2 2 5 9». Sometimes we can't find a solution and
then we wonder: it is because there isn't one, or is it because we
just didn't get it yet? So the searcher turned into a phone app,
which would tell us whether there was solution, so we'd know whether
to give up or keep searching.</p>
<p>But this wasn't quite good enough either, because after we would find
that first solution, say <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%242%c2%b7%285%20%2b%209%20%2d%202%29%24" />, we would wonder: are there
any more? And here the program was useless: it would
cheerfully report that there were three, so we would rack our brains
to find another, fail, ask the program to tell us the answer, and
discover to our disgust that the three solutions it had in mind were:</p>
<p>$$
2 \cdot (5 + (9 - 2)) \\
2 \cdot (9 + (5 - 2)) \\
2 \cdot ((5 + 9) - 2)
$$</p>
<p>The computer thinks these are different, because it uses different
data structures to represent them. It represents them with an abstract
syntax tree, which means that each expression is either a single
constant, or is a structure comprising an operator and its two operand
expressions—always exactly two. The computer understands the three
expressions above as having these structures:</p>
<table align="center" style="margin: auto;"><tbody><tr>
<td><img src="https://pic.blog.plover.com/math/24-puzzle-2/t2.svg" style="" />
</td><td><img src="https://pic.blog.plover.com/math/24-puzzle-2/t3.svg" style="" />
</td><td><img src="https://pic.blog.plover.com/math/24-puzzle-2/t1.svg" style="" />
</td></tr></tbody></table>
<p>It's not hard to imagine that the computer could be taught to
understand that the first two trees are equivalent. Getting it to
recognize that the third one is also equivalent seems somewhat more
difficult.</p>
<h2>Commutativity and associativity</h2>
<p>I would like the computer to understand that these three expressions
should be considered “the same”. But what does “the same” mean? This
problem is of a kind I particularly like: we want the computer to do
<em>something</em>, but we're not exactly sure what that something is. Some
questions are easy to ask but hard to answer, but this is the
opposite: the real problem is to decide what question we want to ask.
Fun!</p>
<p>Certainly some of the question should involve commutativity and
associativity of addition and multiplication. If the only difference
between two expressions is that one has <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20b%24" /> where the other has
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24b%20%2b%20a%24" />, they should be considered the same; similarly <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20%28b%20%2b%0ac%29%24" /> is the same expression as <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%20%2b%20b%29%20%2b%20c%24" /> and as <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28b%20%2b%20a%29%20%2b%20c%24" />
and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24b%20%2b%20%28a%20%2b%20c%29%24" /> and so forth.</p>
<p>The «2 2 5 9» example above shows that commutativity and
associativity are not limited to addition and multiplication. There
are commutative and associative properties of subtraction also! For example,
$$a+(b-c) = (a+b)-c$$ and $$(a+b)-c = (a-c)+b.$$
There ought to be names for these laws but as far as I know there aren't. (Sure, it's
just commutativity and associativity of addition in disguise, but
nobody explaining these laws to school kids <em>ever</em> seems to point out
that subtraction can enter into it. They just observe that <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%2db%29%2dc%0a%e2%89%a0%20a%2d%28b%2dc%29%24" />, say “subtraction isn't associative”, and leave it at that.)</p>
<p>Closely related to these identities are operator inversion identities
like <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2d%28b%2bc%29%20%3d%20%28a%2db%29%2dc%24" />, <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2d%28b%2dc%29%20%3d%20%28a%2db%29%2bc%24" />, and their
multiplicative analogues. I don't know names for these algebraic laws
either.</p>
<p>One way to deal with all of this would to build a complicated
comparison function for abstract syntax trees that tried to transform
one tree into another by applying these identities. A better approach
is to recognize that the data structure is over-specified. If we want
the computer to understand that <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%20%2b%20b%29%20%2b%20c%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20%28b%20%2b%20c%29%24" />
are the same expression, we are swimming upstream by using a data
structure that was specifically designed to capture the difference
between these expressions.</p>
<p>Instead, I invented a data structure, called an <em>Ezpr</em> (“Ez-pur”), that
can represent expressions, but in a somewhat more natural way than
abstract syntax trees do, and in a way that makes commutativity and
associativity transparent. </p>
<p>An Ezpr has a simplest form, called its “canonical” or “normal” form.
Two Ezprs represent essentially the same mathematical expression if
they have the same canonical form. To decide if two abstract syntax
trees are the same, the computer converts them to Ezprs, simplifies
them, and checks to see if resulting canonical forms are identical.</p>
<h2>The Ezpr</h2>
<p>Since associativity doesn't matter, we don't want to represent it.
When we (humans) think about adding up a long column of numbers, we
don't think about associativity because we don't add them
pairwise. Instead we use an addition algorithm that adds them all at
once in a big pile. We don't treat addition as a binary operation; we
normally treat it as an operator that adds up the numbers in a list.
The Ezpr makes this explicit: its addition operator is applied to a
list of subexpressions, not to a pair. Both <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20%28b%20%2b%20c%29%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%0a%2b%20b%29%20%2b%20c%24" /> are represented as the Ezpr</p>
<pre><code> SUM [ a b c - ]
</code></pre>
<p>which just says that we are adding up <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%24" />, <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24b%24" />, and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%24" />. (The
<code>-</code> sign is just punctuation; ignore it for now.)</p>
<p>Similarly the Ezpr <code>MUL [ a b c ÷ ]</code> represents the product of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%24" />,
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24b%24" />, and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%24" />. (Please ignore the <code>÷</code> sign for the time being.)</p>
<p>To handle commutativity, we want those <code>[ a b c ]</code> lists to be bags.
Perl doesn't have a built-in bag object, so instead I used arrays and
required that the array elements be in sorted order. (Exactly <em>which</em>
sorted order doesn't really matter.)</p>
<h2>Subtraction and division</h2>
<p>This doesn't yet handle subtraction and division, and the way I chose
to handle them is the only part of this that I think is at all
clever. A <code>SUM</code> object has not one but two bags, one for the positive
and one for the negative part of the expression. An expression like
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2d%20b%20%2b%20c%20%2d%20d%24" /> is represented by the Ezpr:</p>
<pre><code>SUM [ a c - b d ]
</code></pre>
<p>and this is <em>also</em> the representation of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20c%20%2d%20b%20%2d%20d%24" />, of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%20%2b%20a%0a%2d%20d%20%2d%20b%24" />, of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%20%2d%20d%2b%20a%2db%24" />, and of any other expression of the
idea that we are adding up <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%24" /> and then deducting <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24b%24" />
and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24d%24" />. The <code>-</code> sign separates the terms that are added from those
that are subtracted. </p>
<p>Either of the two bags may be empty, so for example <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20b%24" /> is just
<code>SUM [ a b - ]</code>.</p>
<p>Division is handled similarly. Here conventional mathematical
notation does a little bit better than in the sum case: <code>MUL [ a c ÷ b
d ]</code> is usually written as <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac%7bac%7d%7bbd%7d%24" />.</p>
<p>Ezprs handle the associativity and commutativity of subtraction and
division quite well. I pointed out earlier that subtraction has an
associative law <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%20%2b%20b%29%20%2d%20c%20%3d%20a%20%2b%0a%28b%20%2d%20c%29%24" /> even though it's not usually called that.
No code is required to understand that those two expressions are
equal if they are represented as Ezprs, because they are represented
by completely identical structures:</p>
<pre><code> SUM [ a b - c ]
</code></pre>
<p>Similarly there is a commutative law for subtraction: <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20b%20%2d%20c%20%3d%20a%0a%2d%20c%20%2b%20b%24" /> and once again that same Ezpr does for both.</p>
<h2>Ezpr laws</h2>
<p>Ezprs are more flexible than binary trees. A binary tree can
represent the expressions <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28a%2bb%29%2bc%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2b%28b%2bc%29%24" /> but not the
expression <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2bb%2bc%24" />. Ezprs can represent all three and it's easy to
transform between them. Just as there are rules for building
expressions out of simpler expressions, there are a few rules for
combining and manipulating Ezprs.</p>
<h3>Lifting and flattening</h3>
<p>The most important transformation is <em>lifting</em>, which is the Ezpr
version of the associative law. In the canonical form of an Ezpr, a
<code>SUM</code> node may not have subexpressions that are also <code>SUM</code> nodes. If
you have</p>
<pre><code> SUM [ a SUM [ b c - ] - … ]
</code></pre>
<p>you should lift the terms from the inner sum into the outer one:</p>
<pre><code> SUM [ a b c - … ]
</code></pre>
<p>effectively transforming <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2b%28b%2bc%29%24" /> into <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%2bb%2bc%24" />. More generally,
in</p>
<pre><code> SUM [ a SUM [ b - c ]
- d SUM [ e - f ] ]
</code></pre>
<p>we lift the terms from the inner Ezprs into the outer one:</p>
<pre><code> SUM [ a b f - c d e ]
</code></pre>
<p>This effectively transforms <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20%28b%20%2d%20c%29%20%2d%20d%20%2d%20%28e%20%2d%20f%29%29%24" /> to <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%20%2b%20b%20%2b%0af%20%2d%20c%20%2d%20d%20%2d%20e%24" />.</p>
<p>Similarly, when a <code>MUL</code> node contains another <code>MUL</code>, we can flatten
the structure.</p>
<p>Say we are converting the expression <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%247%20%c3%b7%20%283%20%c3%b7%20%286%20%c3%97%204%29%29%24" /> to an Ezpr.
The conversion function is recursive and the naïve version computes
this Ezpr:</p>
<pre><code> MUL [ 7 ÷ MUL [ 3 ÷ MUL [ 6 4 ÷ ] ] ]
</code></pre>
<p>But then at the bottom level we have a <code>MUL</code> inside a <code>MUL</code>, so the
4 and 6 in the innermost <code>MUL</code> are lifted upward:</p>
<pre><code> MUL [ 7 ÷ MUL [ 3 ÷ 6 4 ] ]
</code></pre>
<p>which represents <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac7%7b%5cfrac%7b3%7d%7b6%5ccdot%204%7d%7d%24" />.
Then again we have a <code>MUL</code> inside a <code>MUL</code>, and again the
subexpressions of the innermost <code>MUL</code> can be lifted:</p>
<pre><code> MUL [ 7 6 4 ÷ 3 ]
</code></pre>
<p>which we can imagine as <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac%7b7%c2%b76%c2%b74%7d3%24" />.</p>
<p>The lifting only occurs when the sub-node has the same type as its
parent; we may not lift terms out of a <code>MUL</code> into a <code>SUM</code> or vice
versa.</p>
<h3>Trivial nodes</h3>
<p>The Ezpr <code>SUM [ a - ]</code> says we are adding up just one thing, <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%24" />,
and so it can be eliminated and replaced with just <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%24" />. Similarly
<code>SUM [ - a ]</code> can be replaced with the constant <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%2da%24" />, if <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24a%24" /> is a
constant. <code>MUL</code> can be handled similarly.</p>
<p>An even simpler case is <code>SUM [ - ]</code> which can be replaced by the
constant 0; <code>MUL [ ÷ ]</code> can be replaced with 1. These sometimes arise
as a result of cancellation.</p>
<h3>Cancellation</h3>
<p>Consider the puzzle «3 3 4 6».
My first solver found 49 solutions to this puzzle. One is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%283%20%2d%203%29%20%2b%20%284%20%c3%97%206%29%24" />.
Another is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%284%20%2b%20%283%20%2d%203%29%29%20%c3%97%206%24" />. A third is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%20%286%20%2b%20%283%20%2d%203%29%29%24" />.</p>
<p>I think these are all the same: the solution is to multiply the 4 by
the 6, and to get rid of the threes by subtracting them to make a zero
term. The zero term can be added onto the rest of expression or to
any of its subexpressions—there are ten ways to do this—and it doesn't
really matter where.</p>
<p>This is easily explained in terms of Ezprs:
If the same subexpression appears in both of a node's bags,
we can drop it. For example,
the expression <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%284%20%2b%20%283%20%2d3%29%29%20%c3%97%206%24" />
starts out as</p>
<pre><code> MUL [ 6 SUM [ 3 4 - 3 ] ÷ ]
</code></pre>
<p>but the duplicate threes in <code>SUM [ 3 4 - 3 ]</code> can be canceled, to
leave</p>
<pre><code> MUL [ 6 SUM [ 4 - ] ÷ ]
</code></pre>
<p>The sum is now trivial, as described in the previous section, so can
be eliminated and replaced with just 4:</p>
<pre><code> MUL [ 6 4 ÷ ]
</code></pre>
<p>This Ezpr records the essential feature of each of the three solutions to
«3 3 4 6» that I mentioned:
they all are multiplying the 6 by the 4, and then doing something else
unimportant to get rid of the threes.</p>
<p>Another solution to the same puzzle is <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%286%20%c3%b7%203%29%20%c3%97%20%284%20%c3%97%203%29%24" />.
Mathematically we would write this as <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac63%c2%b74%c2%b73%24" /> and we can see
this is just <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%246%c3%974%24" /> again, with the threes gotten rid of by
multiplication and division, instead of by addition and subtraction.
When converted to an Ezpr, this expression becomes:</p>
<pre><code> MUL [ 6 4 3 ÷ 3 ]
</code></pre>
<p>and the matching threes in the two bags are cancelled, again leaving</p>
<pre><code> MUL [ 6 4 ÷ ]
</code></pre>
<p>In fact there aren't 49 solutions to this puzzle. There is only one,
with <a href="https://pic.blog.plover.com/math/24-puzzle-2/3346-solutions.txt">49 trivial variations</a>.</p>
<h3>Identity elements</h3>
<p>In the preceding example, many of the trivial variations on the
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%c3%976%24" /> solution involved multiplying some subexpression by <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac%0a33%24" />. When one of the input numbers in the puzzle is a 1, one can
similarly obtain a lot of useless variations by choosing where to
multiply the 1.</p>
<p>Consider
«1 3 3 5»:
We can make 24 from <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%243%20%c3%97%20%283%20%2b%205%29%24" />. We then have to get
rid of the 1, but we can do that by multiplying it onto any
of the five subexpressions of <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%243%20%c3%97%20%283%20%2b%205%29%24" />:</p>
<p>$$
1 × (3 × (3 + 5)) \\
(1 × 3) × (3 + 5) \\
3 × (1 × (3 + 5)) \\
3 × ((1 × 3) + 5) \\
3 × (3 + (1×5))
$$</p>
<p>These should not be considered different solutions.
Whenever we see any 1's in either of the bags of a <code>MUL</code> node,
we should eliminate them.
The first expression above, <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%241%20%c3%97%20%283%20%c3%97%20%283%20%2b%205%29%29%24" />, is converted to the Ezpr</p>
<pre><code> MUL [ 1 3 SUM [ 3 5 - ] ÷ ]
</code></pre>
<p>but then the 1 is eliminated from the <code>MUL</code> node leaving</p>
<pre><code> MUL [ 3 SUM [ 3 5 - ] :- ]
</code></pre>
<p>The fourth expression,
<img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%243%20%c3%97%20%28%281%20%c3%97%203%29%20%2b%205%29%24" />,
is initially converted to the Ezpr</p>
<pre><code> MUL [ 3 SUM [ 5 MUL [ 1 3 ÷ ] - ] ÷ ]
</code></pre>
<p>When the 1 is eliminated from the inner <code>MUL</code>, this leaves a trivial
<code>MUL [ 3 ÷ ]</code> which is then replaced with just 3, leaving:</p>
<pre><code> MUL [ 3 SUM [ 5 3 - ] ÷ ]
</code></pre>
<p>which is the same Ezpr as before.</p>
<p>Zero terms in the bags of a <code>SUM</code> node can similarly be dropped.</p>
<h3>Multiplication by zero</h3>
<p>One final case is that <code>MUL [ 0 … ÷ … ]</code> can just be simplified to 0.</p>
<p>The question about what to do when there is a zero in the denominator
is a bit of a puzzle.
In the presence of division by zero, some of our simplification rules
are questionable. For example, when we have <code>MUL [ a ÷ MUL [ b ÷ c ]
]</code>, the lifting rule says we can simplify this to <code>MUL [ a c ÷ b
]</code>—that is, that <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%5cfrac%20a%7b%5cfrac%20bc%7d%20%3d%20%5cfrac%7bac%7db%24" />. This is correct,
except that when <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24b%3d0%24" /> or <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24c%3d0%24" /> it may be nonsense, depending on what else is
going on. But since zero denominators never arise in the solution of
these puzzles, there is no issue in this application.</p>
<h2>Results</h2>
<p>The <code>Ezpr</code> module is around 200 lines of Perl code, including
everything: the function that converts abstract syntax trees to Ezprs,
functions to convert Ezprs to various notations (both <code>MUL [ 4 ÷ SUM [
3 - 2 ] ]</code> and <code>4 ÷ (3 - 2)</code>), and the two versions of the
normalization process described in the previous section. The
normalizer itself is about 35 lines.</p>
<p>Associativity is taken care of by the Ezpr structure itself, and
commutativity is not too difficult; as I mentioned, it would have been
trivial if Perl had a built-in bag structure. I find it much easier
to reason about transformations of Ezprs than abstract syntax trees.
Many operations are much simpler; for example the negation of
<code>SUM [ A - B ]</code> is simply <code>SUM [ B - A ]</code>. Pretty-printing is also easier
because the Ezpr better captures the way we write and think about
expressions.</p>
<p>It took me a while to get the normalization tuned properly, but the
results have been quite successful, at least for this problem domain.
The current puzzle-solving program reports the number of distinct
solutions to each puzzle. When it reports two different solutions,
they are really different; when it fails to support the exact solution
that Toph or I found, it reports one essentially the same. (There are
some small exceptions, <a href="http://#arguable">which I will discuss below</a>.)</p>
<p>Since there is no specification for “essentially the same” there is no
hope of automated testing. But we have been using the app for several
months looking for mistakes, and we have not found any.
If the normalizer failed to recognize that two expressions were
essentially similar, we would be very likely to notice: we would be
solving some puzzle, be unable to find the last of the solutions that
the program claimed to exist, and then when we gave up and saw what it was we
would realize that it was essentially the same as one of the solutions we
had found. I am pretty confident that there are no errors of this
type, but see <a href="http://#arguable">“Arguable points”</a> below. </p>
<p>A harder error to detect is whether the computer has erroneously
conflated two essentially dissimilar expressions. To detect this we
would have to notice that an expression was missing from the computer's solution list. I am
less confident that nothing like this has occurred, but as the months
have gone by I feel better and better about it.</p>
<p>I consider the problem of “how many solutions does this puzzle
<em>really</em> have to have?” been satisfactorily solved. There are some
edge cases, but I think we have identified them.</p>
<p><a href="https://github.com/mjdominus/24-puzzle-solver">Code for my solver is on
Github</a>. The Ezpr code
is in <a href="https://github.com/mjdominus/24-puzzle-solver/blob/master/Expr.pm">the <code>Ezpr</code> package in the <code>Expr.pm</code>
file</a>.
This code is all in the public domain.</p>
<h3>Some examples</h3>
<p>The original program claims to find 35 different solutions to «4 6 6
6». The revised program recognizes that these are of only two types:</p>
<table align="center">
<tbody><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%206%20%c3%97%206%20%c3%b7%206%24" /></td><td><tt>MUL [ 4 6 - ]</tt>
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%286%20%2d%204%29%20%c3%97%20%286%20%2b%206%29%24" /></td><td><tt>MUL [ SUM [ 6 - 4 ] SUM [ 6 6 - ] ÷ ]
</tt></td></tr></tbody></table>
<p>Some of the variant forms of the first of those include:</p>
<p>$$
6 × (4 + (6 - 6)) \\
6 + ((4 × 6) - 6) \\
(6 - 6) + (4 × 6) \\
(6 ÷ 6) × (4 × 6) \\
6 ÷ ((6 ÷ 4) ÷ 6) \\
6 ÷ (6 ÷ (4 × 6)) \\
6 × (6 × (4 ÷ 6)) \\
(6 × 6) ÷ (6 ÷ 4) \\
6 ÷ ((6 ÷ 6) ÷ 4) \\
6 × (6 - (6 - 4)) \\
6 × (6 ÷ (6 ÷ 4)) \\
\ldots <br />
$$</p>
<p>In an even more extreme case, the original program finds 80 distinct
expressions that solve
«1 1 4 6»,
all of which are trivial variations on <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%c2%b76%24" />.</p>
<p>Of the 715 puzzles, 466 (65%) have solutions; for 175 of these the
solution is unique. There are 3 puzzles with 8 solutions each
(«2 2 4 8»,
«2 3 6 9»,
and «2 4 6 8»), one with 9 solutions («2 3 4 6»), and one with 10 solutions
(«2 4 4 8»).</p>
<p>The 10 solutions for «2 4 4 8»
are as follows:</p>
<table align="center">
<tbody><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%208%20%2d%202%20%c3%97%204%20%20%20%20%24" /></td><td>SUM [ MUL [ 4 8 ÷ ] - MUL [ 2 4 ÷ ] ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%20%282%20%2b%208%20%2d%204%29%20%20%20%24" /></td><td>MUL [ 4 SUM [ 2 8 - 4 ] ÷ ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%288%20%2d%204%29%20%c3%97%20%282%20%2b%204%29%20%24" /></td><td>MUL [ SUM [ 8 - 4 ] SUM [ 2 4 - ] ÷ ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%20%284%20%2b%208%29%20%c3%b7%202%20%20%24" /></td><td>MUL [ 4 SUM [ 4 8 - ] ÷ 2 ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%284%20%2d%202%29%20%c3%97%20%284%20%2b%208%29%20%24" /></td><td>MUL [ SUM [ 4 - 2 ] SUM [ 4 8 - ] ÷ ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%248%20%c3%97%20%282%20%2b%204%2f4%29%20%20%20%20%20%24" /></td><td>MUL [ 8 SUM [ 1 2 - ] ÷ ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%242%20%c3%97%204%20%c3%97%204%20%2d%208%20%20%20%20%24" /></td><td>SUM [ MUL [ 2 4 4 ÷ ] - 8 ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%248%20%2b%202%20%c3%97%20%284%20%2b%204%29%20%20%20%24" /></td><td>SUM [ 8 MUL [ 2 SUM [ 4 4 - ] ÷ ] - ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%2b%204%20%2b%202%20%c3%97%208%20%20%20%20%20%24" /></td><td>SUM [ 4 4 MUL [ 2 8 ÷ ] - ]
</td></tr><tr><td><img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%244%20%c3%97%20%288%20%2d%204%2f2%29%20%24" /></td><td>MUL [ 4 SUM [ 8 - MUL [ 4 ÷ 2 ] ] ÷ ]
</td></tr></tbody></table>
<p><a href="https://pic.blog.plover.com/math/24-puzzle-2/Solutions-24.txt">A complete listing of every essentially different solution to every
«a b c d» puzzle is available here</a>.
There are 1,063 solutions in all.</p>
<h2>Arguable points <a name="arguable"></a></h2>
<p>There are a few places where we have not completely pinned down what
it means for two solutions to be essentially the same; I think there
is room for genuine disagreement. </p>
<ol>
<li><p>Any solution involving <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%242%c3%972%24" /> can be changed into a slightly different
solution involving <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%242%2b2%24" /> instead. These expressions are
arithmetically different but numerically equal. For example, I
mentioned earlier that
«2 2 4 8»
has 8 solutions. But two of these are <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%208%20%2b%204%20%c3%97%20%282%20%2b%202%29%24" /> and <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%0a%20%20%20%208%20%2b%204%20%c3%97%202%20%c3%97%202%24" />. I am willing to accept these as
essentially different. Toph, however, disagrees.</p></li>
<li><p>A similar but more complex situation arises in connection with «1
2 3 7». Consider <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%243%c3%977%2b3%24" />, which equals 24. To get a solution
to «1 2 3 7», we can replace either of the threes in <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%243%c3%977%2b3%24" />
with <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%281%2b2%29%24" />, obtaining <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%28%281%20%2b%202%29%20%c3%97%207%29%20%2b%203%24" /> or <img src="https://chart.apis.google.com/chart?chf=bg,s,00000000&cht=tx&chl=%24%20%283%c3%977%29%2b%281%0a%20%20%20%20%2b2%29%24" />. My program considers these to be different solutions.
Toph is unsure.</p></li>
</ol>
<p>It would be pretty easy to adjust the normalization process to handle
these the other way if the user wanted that.</p>
<h2>Some interesting puzzles</h2>
<p><a href="https://pic.blog.plover.com/math/24-puzzle-2/JJZ4631.jpg"><img src="https://pic.blog.plover.com/math/24-puzzle-2/JJZ4631-sm.png" align="right" border="0" /></a></p>
<p>«1 2 7 7»
has only one solution, quite unusual. <a href="https://pic.blog.plover.com/math/24-puzzle-2/1277.txt">(Spoiler)</a>
«2 2 6 7»
has two solutions, both somewhat unusual. <a href="https://pic.blog.plover.com/math/24-puzzle-2/2267.txt">(Spoiler)</a></p>
<p>Somewhat similar to
«1 2 7 7»
is
«3 9 9 9»
which also has an unusual solution. But it has two other solutions
that are less surprising. <a href="https://pic.blog.plover.com/math/24-puzzle-2/3999.txt">(Spoiler)</a></p>
<p>«1 3 8 9» has an easy solution but also a quite tricky solution.
<a href="https://pic.blog.plover.com/math/24-puzzle-2/1389.txt">(Spoiler)</a></p>
<p>One of my neighbors has the license plate <code>JJZ 4631</code>.
«4 6 3 1» is one of the more difficult puzzles.</p>
<h2>What took so long?</h2>
<p><a href="https://blog.plover.com/math/24-puzzle.html">Back in March, I wrote</a>:</p>
<blockquote>
<p>I have enough material for at least three or four more articles
about this that I hope to publish here in the coming weeks.</p>
<p>But <a href="https://blog.plover.com/math/17-puzzle.html">the previous article on this
subject</a> ended similarly, saying</p>
<blockquote>
<p>I hope to write a longer article about solvers in the next week or so.</p>
</blockquote>
<p>and that was in July 2016, so don't hold your breath.</p>
</blockquote>
<p>And here we are, five months later!</p>
<p>This article was a <em>huge</em> pain to write. Sometimes I sit down to write
something and all that comes out is dreck. I sat down to write this
one at least three or four times and it never worked. The tortured
Git history bears witness. In the end I had to abandon all my earlier
drafts and start over from scratch, writing a fresh outline in an
empty file.</p>
<p>But perseverance paid off! WOOOOO.</p>
<p>[ Addendum 20170825: I completely forgot that Shreevatsa R. wrote <a href="https://shreevatsa.wordpress.com/2016/07/20/a-simple-puzzle-with-a-foray-into-inequivalent-expressions/">a very interesting article on the same topic as this one</a>,
in July of last year soon after I published my first article in this
series. ]</p>
<p>[ Addendum 20170829: A previous version of this article used the notations <code>SUM [ … # … ]</code>
and <code>MUL [ … # … ]</code>, which I said I didn't like. Zellyn Hunter has
persuaded me to replace these with <code>SUM [ … - … ]</code> and <code>MUL
[ … ÷ … ]</code>. Thank you M. Hunter! ]</p>
<p>[ <a href="https://blog.plover.com/math/24-puzzle-3.html">Yet more on this topic!</a> ]</p>Mon, 21 Aug 2017 01:53:00 +0000mjd@plover.com (Mark Dominus)Neil Mitchell: Ghcid and VS Codetag:blogger.com,1999:blog-7094652.post-469918405847654096
http://neilmitchell.blogspot.com/2017/08/ghcid-and-vs-code.html
<p><em>Summary: There's now a Ghcid VS Code addin that gives you red squiggles.</em></p><p>I've been using <a href="http://hackage.haskell.org/package/ghcid">Ghcid</a> for about 3 years, and <a href="https://code.visualstudio.com/">VS Code</a> for about 6 months. Ghcid alone is able to display the errors and warnings, and update them whenever you save. In this post I'll show how VS Code users can also click on errors to jump to them, and how to get red squiggles in the text buffer for errors.</p><p><strong>Clicking on errors</strong></p><p>Using a recent VS Code, if you run <code>ghcid</code> from the terminal window, hold <code>Ctrl</code> and click the filename and it jumps to the right location in the erroneous file.</p><p><strong>Red squiggles</strong></p><p>Red squiggles are now possible using the <a href="https://marketplace.visualstudio.com/items?itemName=ndmitchell.haskell-ghcid">haskell-ghcid</a> addin. To get it working:</p><ul><li>Run <code>ghcid -o ghcid.txt</code> which will produce a file <code>ghcid.txt</code> which updates every time <code>ghcid</code> updates. Running the underlying <code>ghci</code> with <code>-ferror-spans</code> will significantly improve the errors reported.</li><li>Open <code>ghcid.txt</code> in VS Code as the active editor. Run the VS Code command (<code>Ctrl+Shift+P</code>) named "Watch Ghcid output".</li></ul><p>These steps cause the <code>ghcid</code> errors to appear in the VS Code Problems pane, and have red squiggles in the editor. Even though the errors are in the proper problems pane, I still prefer the output provided by the <code>ghcid</code> terminal, so still look at that.</p><p>The VS Code addin is not very well polished - but I'm using it on a daily basis.</p>Sun, 20 Aug 2017 20:18:00 +0000noreply@blogger.com (Neil Mitchell)Roman Cheplyaka: Understanding Asymmetric Numeral Systemshttp://ro-che.info//articles/2017-08-20-understanding-ans.html
http://ro-che.info/articles/2017-08-20-understanding-ans
<p>Apparently, <a href="https://www.reddit.com/r/programming/comments/6h08z5/google_is_currently_trying_to_patent_video/">Google is trying to patent (an application of) Asymmetric Numeral Systems</a>, so I spent some time today learning what it is.</p>
<p>In its essense lies a simple and beautiful idea.</p>
<figure>
<img src="https://dropboxtechblog.files.wordpress.com/2016/06/silicon-valley.jpg?w=650&h=380" />
</figure>
<p>ANS is a lossless compression algorithm. Its input is a list of symbols from some finite set. Its output is a positive integer. Each symbol <span class="math inline">\(s\)</span> has a fixed known probability <span class="math inline">\(p_s\)</span> of occurring in the list. The algorithm tries to assign each list a unique integer so that the more probable lists get smaller integers.</p>
<p>If we ignore the compression part (assigning smaller integers to more probable inputs), the encoding could be done as follows: convert each symbol to a number from <span class="math inline">\(0\)</span> to <span class="math inline">\(B-1\)</span> (where <span class="math inline">\(B\)</span> is the number of symbols), add a leading 1 to avoid ambiguities caused by leading zeros, and interpret the list as an integer written in a base-<span class="math inline">\(B\)</span> positional system.</p>
<p>This encoding process is an iterative/recursive algorithm:</p>
<ol type="1">
<li>Start with the number 1;</li>
<li>If the current number is <span class="math inline">\(n\)</span>, and the incoming symbol correspond to a number <span class="math inline">\(s\)</span>, update the number to be <span class="math inline">\(s + n\cdot B\)</span>.</li>
</ol>
<p>The decoding process is a corecursive algorithm:</p>
<ol type="1">
<li>Start with the number that we are decoding;</li>
<li>Split the current number <span class="math inline">\(n\)</span> into the quotient and remainder modulo <span class="math inline">\(B\)</span>;</li>
<li>Emit the remainder and continue decoding the quotient;</li>
<li>Stop when the current number reaches 1.</li>
</ol>
<p>(The decoding is LIFO: the first decoded element will be the last encoded element.)</p>
<p>This encoding scheme relies on the standard isomorphism between the sets <span class="math inline">\(\{0,\ldots,B-1\}\times \mathbb{Z}_{\geq 1}\)</span> and <span class="math inline">\(\mathbb{Z}_{\geq B}\)</span>, established by the functions</p>
<p><span class="math display">\[f(s,n) = s + n\cdot B;\]</span> <span class="math display">\[g(n) = (n \bmod B, [n/B]).\]</span></p>
<p>(The peculiar domain and codomain of this isomorphism are chosen so that we have <span class="math inline">\(\forall n,s.\;f(s,n) > n\)</span>; this ensures that the decoding process doesn’t get stuck.)</p>
<p>We can represent this in Haskell as</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="co">{-# LANGUAGE ScopedTypeVariables, TypeApplications,</span>
<span class="co"> NamedFieldPuns, AllowAmbiguousTypes #-}</span>
<span class="kw">import </span><span class="dt">Data.Ord</span>
<span class="kw">import </span><span class="dt">Data.List</span>
<span class="kw">import </span><span class="dt">Numeric.Natural</span>
<span class="kw">data</span> <span class="dt">Iso</span> a b <span class="fu">=</span> <span class="dt">Iso</span>
{<span class="ot"> to ::</span> a <span class="ot">-></span> b
,<span class="ot"> from ::</span> b <span class="ot">-></span> a
}
<span class="ot">encode ::</span> <span class="dt">Iso</span> (s, <span class="dt">Natural</span>) <span class="dt">Natural</span> <span class="ot">-></span> [s] <span class="ot">-></span> <span class="dt">Natural</span>
encode <span class="dt">Iso</span>{to} <span class="fu">=</span> foldl' (\acc s <span class="ot">-></span> to (s, acc)) <span class="dv">1</span>
<span class="ot">decode ::</span> <span class="dt">Iso</span> (s, <span class="dt">Natural</span>) <span class="dt">Natural</span> <span class="ot">-></span> <span class="dt">Natural</span> <span class="ot">-></span> [s]
decode <span class="dt">Iso</span>{from} <span class="fu">=</span> reverse <span class="fu">.</span> unfoldr
(\n <span class="ot">-></span>
<span class="kw">if</span> n <span class="fu">==</span> <span class="dv">1</span>
<span class="kw">then</span> <span class="dt">Nothing</span>
<span class="kw">else</span> <span class="dt">Just</span> <span class="fu">$</span> from n
)</code></pre></div>
<p>And the standard isomorphism which we used in the simple encoding process is</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">std_iso ::</span> forall s <span class="fu">.</span> (<span class="dt">Bounded</span> s, <span class="dt">Enum</span> s) <span class="ot">=></span> <span class="dt">Iso</span> (s, <span class="dt">Natural</span>) <span class="dt">Natural</span>
std_iso <span class="fu">=</span> <span class="dt">Iso</span> (\(s,n) <span class="ot">-></span> s2n s <span class="fu">+</span> base <span class="fu">@</span>s <span class="fu">*</span> n) (\n <span class="ot">-></span> (n2s <span class="fu">$</span> n <span class="ot">`mod`</span> base <span class="fu">@</span>s, n <span class="ot">`div`</span> base <span class="fu">@</span>s))
<span class="ot">s2n ::</span> forall s <span class="fu">.</span> (<span class="dt">Bounded</span> s, <span class="dt">Enum</span> s) <span class="ot">=></span> s <span class="ot">-></span> <span class="dt">Natural</span>
s2n s <span class="fu">=</span> fromIntegral <span class="fu">$</span>
((fromIntegral <span class="fu">.</span> fromEnum)<span class="ot"> s ::</span> <span class="dt">Integer</span>) <span class="fu">-</span>
((fromIntegral <span class="fu">.</span> fromEnum) (minBound <span class="fu">@</span>s)<span class="ot"> ::</span> <span class="dt">Integer</span>)
<span class="ot">n2s ::</span> forall s <span class="fu">.</span> (<span class="dt">Bounded</span> s, <span class="dt">Enum</span> s) <span class="ot">=></span> <span class="dt">Natural</span> <span class="ot">-></span> s
n2s n <span class="fu">=</span> toEnum <span class="fu">.</span> fromIntegral <span class="fu">$</span>
(fromIntegral n <span class="fu">+</span> (fromIntegral <span class="fu">.</span> fromEnum) (minBound <span class="fu">@</span>s)<span class="ot"> ::</span> <span class="dt">Integer</span>)
<span class="ot">base ::</span> forall s <span class="fu">.</span> (<span class="dt">Bounded</span> s, <span class="dt">Enum</span> s) <span class="ot">=></span> <span class="dt">Natural</span>
base <span class="fu">=</span> s2n (maxBound <span class="fu">@</span>s) <span class="fu">+</span> <span class="dv">1</span></code></pre></div>
<p>(The functions are more complicated than they have to be to support symbol types like <code>Int</code>. <code>Int</code> does not start at 0 and is prone to overflow.)</p>
<p>Let’s now turn to the general form of the isomorphism</p>
<p><span class="math display">\[f \colon \{0,\ldots,B-1\}\times \mathbb{Z}_{\geq 1} \to \mathbb{Z}_{\geq \beta};\]</span> <span class="math display">\[g \colon \mathbb{Z}_{\geq \beta} \to \{0,\ldots,B-1\}\times \mathbb{Z}_{\geq 1}.\]</span></p>
<p>(In general, <span class="math inline">\(\beta\)</span>, the smallest value of <span class="math inline">\(f\)</span>, does not have to equal <span class="math inline">\(B\)</span>, the number of symbols.)</p>
<p>If we know (or postulate) that the second component of <span class="math inline">\(g\)</span>, <span class="math inline">\(g_2\colon \mathbb{Z}_{\geq \beta} \to \mathbb{Z}_{\geq 1}\)</span>, is increasing, then we can recover it from the first component, <span class="math inline">\(g_1\colon \mathbb{Z}_{\geq \beta} \to \{0,\ldots,B-1\}\)</span>.</p>
<p>Indeed, for a given <span class="math inline">\(s=g_1(n)\)</span>, <span class="math inline">\(g_2\)</span> must be the unique increasing isomorphism from <span class="math display">\[A_s = \{f(s,m)\mid m\in\mathbb{Z}_{\geq 1}\} = \{n\mid n\in\mathbb{Z}_{\geq \beta}, g_1(n) = s\}\]</span> to <span class="math inline">\(\mathbb{Z}_{\geq 1}\)</span>. To find <span class="math inline">\(g_2(n)\)</span>, count the number of elements in <span class="math inline">\(A_s\)</span> that are <span class="math inline">\(\leq n\)</span>.</p>
<p>Similarly, we can recover <span class="math inline">\(f\)</span> from <span class="math inline">\(g_1\)</span>. To compute <span class="math inline">\(f(s,n)\)</span>, take <span class="math inline">\(n\)</span>th smallest number in <span class="math inline">\(A_s\)</span>.</p>
<p>In Haskell:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">ans_iso ::</span> forall s <span class="fu">.</span> <span class="dt">Eq</span> s <span class="ot">=></span> (<span class="dt">Natural</span>, <span class="dt">Natural</span> <span class="ot">-></span> s) <span class="ot">-></span> <span class="dt">Iso</span> (s, <span class="dt">Natural</span>) <span class="dt">Natural</span>
ans_iso (b, classify) <span class="fu">=</span> <span class="dt">Iso</span>{to, from} <span class="kw">where</span>
<span class="ot"> to ::</span> (s, <span class="dt">Natural</span>) <span class="ot">-></span> <span class="dt">Natural</span>
to (s, n) <span class="fu">=</span> [ k <span class="fu">|</span> k <span class="ot"><-</span> [b<span class="fu">..</span>], classify k <span class="fu">==</span> s ] <span class="ot">`genericIndex`</span> (n<span class="fu">-</span><span class="dv">1</span>)
<span class="ot"> from ::</span> <span class="dt">Natural</span> <span class="ot">-></span> (s, <span class="dt">Natural</span>)
from n <span class="fu">=</span>
<span class="kw">let</span> s <span class="fu">=</span> classify n
n' <span class="fu">=</span> genericLength [ () <span class="fu">|</span> k <span class="ot"><-</span> [b<span class="fu">..</span>n], classify k <span class="fu">==</span> s ]
<span class="kw">in</span> (s, n')</code></pre></div>
<p>For every function <span class="math inline">\(g_1\colon \mathbb{Z}_{\geq \beta} \to \{0,\ldots,B-1\}\)</span> (named <code>classify</code> in Haskell), we have a pair of encode/decode functions, provided that each of the sets <span class="math inline">\(A_s\)</span> is infinite. In particular, we can get the standard encode/decode functions (originally defined by <code>std_iso</code>) by setting <code>classify</code> to</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">classify_mod_base ::</span> forall s <span class="fu">.</span> (<span class="dt">Bounded</span> s, <span class="dt">Enum</span> s) <span class="ot">=></span> (<span class="dt">Natural</span>, <span class="dt">Natural</span> <span class="ot">-></span> s)
classify_mod_base <span class="fu">=</span> (base <span class="fu">@</span>s, \n <span class="ot">-></span> n2s (n <span class="ot">`mod`</span> base <span class="fu">@</span>s))</code></pre></div>
<p>By varying <span class="math inline">\(g_1\)</span> (and therefore the sets <span class="math inline">\(A_s\)</span>), we can control which inputs get mapped to smaller integers.</p>
<p>If <span class="math inline">\(A_s\)</span> is more dense, <span class="math inline">\(f(s,n)\)</span>, defined as <span class="math inline">\(n\)</span>th smallest number in <span class="math inline">\(A_s\)</span>, will be smaller.</p>
<p>If <span class="math inline">\(A_s\)</span> is more sparse, <span class="math inline">\(f(s,n)\)</span> will be larger.</p>
<p>The standard isomorphism makes the sets <span class="math display">\[A_s = \{ s+n\cdot B \mid n\in \mathbb Z_{\geq 1} \} \]</span> equally dense for all values of <span class="math inline">\(s\)</span>. This makes sense when all <span class="math inline">\(s\)</span> are equally probable.</p>
<p>But in general, we should make <span class="math inline">\(A_s\)</span> denser for those <span class="math inline">\(s\)</span> that are more frequent. Specifically, we want</p>
<p><span class="math display">\[
\frac{|\{k\in A_s \mid k \leq x\}|}{x} \approx p_s.
\]</span></p>
<p>Substituting <span class="math inline">\(x=f(s,n)\)</span> then gives <span class="math inline">\(\log_2 f(s,n) \approx \log_2 n + \log_2 (1/p_s)\)</span>. This means that adding a symbol <span class="math inline">\(s\)</span> costs <span class="math inline">\(\log_2 (1/p_s)\)</span> bits, which is <a href="https://en.wikipedia.org/wiki/Entropy_encoding">what we should strive for</a>.</p>
<p>Here’s a simple example of a suitable <span class="math inline">\(g_1\)</span>:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="ot">classify_prob ::</span> <span class="dt">Show</span> s <span class="ot">=></span> (<span class="dt">Bounded</span> s, <span class="dt">Enum</span> s) <span class="ot">=></span> [<span class="dt">Double</span>] <span class="ot">-></span> (<span class="dt">Natural</span>, <span class="dt">Natural</span> <span class="ot">-></span> s)
classify_prob probs <span class="fu">=</span>
<span class="kw">let</span> beta <span class="fu">=</span> <span class="dv">2</span> <span class="co">-- arbitrary number > 1</span>
t <span class="fu">=</span> genericLength l
l <span class="fu">=</span> concatMap (\(s, t) <span class="ot">-></span> replicate t s)
<span class="fu">.</span> sortBy (comparing (<span class="dt">Down</span> <span class="fu">.</span> snd))
<span class="fu">.</span> zip [minBound<span class="fu">..</span>maxBound]
<span class="fu">$</span> map (round <span class="fu">.</span> (<span class="fu">/</span> minimum probs)) probs
g1 n <span class="fu">=</span> l <span class="ot">`genericIndex`</span> ((n<span class="fu">-</span>beta) <span class="ot">`mod`</span> t)
<span class="kw">in</span> (beta, g1)</code></pre></div>
<p>This is a periodic function. It computes the number of times each symbol <span class="math inline">\(s\)</span> will appear within a single period as <span class="math inline">\(k_s=\mathrm{round}(p_s/\min \{p_s\})\)</span>. The number <span class="math inline">\(p_s/\min \{p_s\}\)</span> is chosen for its following two properties:</p>
<ol type="1">
<li>it is proportional to the probability of the symbol, <span class="math inline">\(p_s\)</span>;</li>
<li>it is <span class="math inline">\(\geq 1\)</span>, so that even the least likely symbol occurs among the values of the function.</li>
</ol>
<p>The function then works by mapping the first <span class="math inline">\(k_0\)</span> numbers to symbol <span class="math inline">\(0\)</span>, the next <span class="math inline">\(k_1\)</span> numbers to symbol <span class="math inline">\(1\)</span>, and so on, until it maps <span class="math inline">\(k_{B-1}\)</span> numbers to symbol <span class="math inline">\(B-1\)</span> and repeats itself. The period of the function is <span class="math inline">\(\sum_s k_s\approx 1/\min \{p_s\}\)</span>.</p>
<p><code>classify_prob</code> rearranges the symbols in the order of decreasing probability, which gives further advantage to the more probable symbols. This is probably the best strategy if we want to allocate integers in blocks; a better way would be to interleave the blocks in a fair or random way in order to keep the densities more uniform.</p>
<p>Another downside of this function is that its period may be too small to distinguish between similar probabilities, such as 0.4 and 0.6. The function used in rANS is better in this regard; it uses progressively larger intervals, which provide progressively better approximations.</p>
<p>But <code>classify_prob</code> is enough to demonstate the idea. Let’s encode a list of booleans where <code>True</code> is expected 90% of time.</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> iso <span class="fu">=</span> ans_iso <span class="fu">$</span> classify_prob [<span class="fl">0.1</span>,<span class="fl">0.9</span>]
<span class="fu">></span> encode iso (replicate <span class="dv">4</span> <span class="dt">True</span>)
<span class="dv">5</span>
<span class="fu">></span> encode iso (replicate <span class="dv">4</span> <span class="dt">False</span>)
<span class="dv">11111</span></code></pre></div>
<p>Four <code>True</code>s compress much better than four <code>False</code>s. Let’s also compare the number of bits in 11111 with the number of bits that the information theory predicts are needed to encode four events with probability 0.1:</p>
<div class="sourceCode"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="fu">></span> logBase <span class="dv">2</span> <span class="dv">11111</span>
<span class="fl">13.439701045971955</span>
<span class="fu">></span> <span class="dv">4</span> <span class="fu">*</span> logBase <span class="dv">2</span> (<span class="dv">1</span><span class="fu">/</span><span class="fl">0.1</span>)
<span class="fl">13.28771237954945</span></code></pre></div>
<p>Not bad.</p>
<p>The implementation of ANS in this article is terribly inefficient, especially its decoding part, mostly because the isomorphism uses brute force search instead of computation. The intention is to elucidate what the encoding scheme looks like and where it comes from. An efficient implementation of ANS and its different variants is an interesting topic in itself, but I’ll leave it for another day.</p>
<p>The full code (including tests) is available <a href="https://ro-che.info/files/2017-08-20-understanding-ans.hs">here</a>.</p>
<p>Thanks to /u/sgraf812 for <a href="https://www.reddit.com/r/haskell/comments/6v4tg1/understanding_asymmetric_numeral_systems_with/dlyjc1u/">pointing out</a> a mistake in a previous version of <code>classify_prob</code>.</p>Sun, 20 Aug 2017 20:00:00 +0000Joachim Breitner: Titelhttp://www.joachim-breitner.de/blog/729-Titel
http://www.joachim-breitner.de/blog/729-Titel
Sun, 20 Aug 2017 18:50:10 +0000mail@joachim-breitner.de (Joachim Breitner)