Friday, March 18, 2011

Clojure Macros: Wanted and Unwanted Symbol Capture

Introduction

No language is perfect. Most languages do not even come close. Some give the programmer the tools to bridge the gap between his idea of perfection and the actual language. Most of these tools are under the "meta-programming" umbrella, and macros are probably the more powerful.

Macros are dangerous tools, indeed. Most articles, books and posts on macros mention somewhere not to abuse macro usage, because they make the programs less clear (TODO: mention somewhere not to abuse macro usage; DONE).

One of the trickiest part is avoiding unwanted symbol capture. Here the the Lisp world divides: some think that symbol capture should be syntactically banned (or something like that), some believe that it is the programme's task to avoid unwanted symbol capture (with the underlying assumption that wanted symbol capture is fine).

I’m not usually happy when a feature is designed not to let me do something; well, quite depending from that something, indeed. I’m not at all concerned with higher level languages not letting me hacking directly with memory; but as far as I can tell, that is more a question of “language model” than of “designing a language not to let me do something”. In fact, Hoyte goes a long way in motivating that symbol capture is not a bug, but a feature.

I believe he is right. I do not find it hard to find situations where I want symbol capture to occur. Of course, in his book, he gives plenty of examples about that; but that’s not the point. I am limiting this to the context of clojure.

Clojure

First, clojure has a nice feature to make it easier to avoid unwanted symbol capture without actually forbidding that. I’m talking about placing a # at the end of an identifier to gensym a new symbol. This is nice to have, nice to use, nice to understand and solves brilliantly the common cases.

But the point is wanted symbol capture. In Clojure we have an interesting example: when creating a gen-class, the methods take an additional explicit parameter “this” (a-la Python, %s/this/self/gi). Like in Python, this is a conventional name, and could be called in any way; still, since Java has an implicit this reference to the current object, it is nice to maintain the convention.

This is a typical example when we explicitly introduce names. It is easy to understand, easy to write and that’s it. There is no capture involved, as we are simply using a function parameter. On the other hand, code in the proxy macro refers to the proxy being created with “this”.

It can’t be changed (as far as I know) since it is injected by the macro. In fact this is wanted symbol capture. The authors want to capture the symbol “this” in the client code and give it meaning. Looking at the code of proxy, it is rather explict.

Both approaches have their advantages: explicit this is easier to understand. Knowledge about how functions work is sufficient to read (and write the code). On the other hand, the proxy macro requires us to know something more: in fact it is a true “new” piece of language. If we did not know that proxy “magically” injects this, we could not understand/write the code. Probably we would say that the code is wrong as “this” should not be bound to anything there.

On the other hand, auto-this in proxies somewhat reduces code clutter. I have not a distinct opinion whether it is a good thing or if I’d prefer explicit this. Perhaps I’d prefer uniformity: that is to say, explicit this everywhere or implicit this everywhere. Though, that part of clojure is under active design and development.

Back to my macro abuses

This post started because of a macro I wrote. The first version is “clean” as it does not capture anything.

(defmacro let-attributes* [this state-method attrs & body]
`(let [{:keys ~attrs} @(. ~this ~state-method)] ~@body))

which would be used as:

(defn exp [this foo]
(let-attributes* this state [bar baz]
(println bar baz foo)))

Essentially it just wraps let+keys to provide a slightly terser syntax (in fact, I find it boring to use .state all the time; I have to say that it is one of the parts of clojure I like less). However, it is still rather verbose. I have to manually specify things I would rather not; in fact, in my code this is always called this and state is always called state, as I did not find a better name.

So I wrote a second version of the macro. A version that is not clean.

(defmacro let-attributes [attrs & body]
(apply list 'let-attributes* 'this 'state attrs body))

And the usage would be just:

(defn exp [this foo]
(let-attributes [bar baz]
(println bar baz foo)))

In fact, is very dirty as it depends from the context where is expanded (which makes it different from the proxy macro). It assumes that “this” is bound to an object with a .state method. But is faster to write.

Here we can also spot the other problem with macros. I believe that every clojure developer is tempted to write his own macros. Macros that may solve similar purposes, but are inherently different. Code using such macros looks different at a different level than code which just uses different libraries and functions.

This is essentially the problem with fiddling with the language itself, which is what happens when we are writing a macro. Basically every time we write and use a macro, we are developing (inc clojure).

Amazon associates

"Let Over Lambda" (Doug Hoyte)

Technorati Tags: , ,



Wednesday, March 16, 2011

Clojure: great exploratory programming

One of the major drawbacks of Java development is the almost complete lack of a REPL. All my favourite languages have a REPL, and I have to say that I've grown completely accustomed to it. I almost can't do without. The possibility to explore new libraries step by step, perhaps with some features common on IDEs (and some more, as it's runtime), is invaluable.

It is hard to understand how the lack of a REPL cripples the learning process of the language (in the first place) and of libraries without having extensively used a REPL. Java somehow compensate this with extensive IDE support; it is possible to write many small programs to try the libraries. However, it is a poor substitute for a true REPL. Java has some third party REPLs, but I did not feel at ease with them. Rather, I resorted on unit testing (which has the added benefit of documenting what you are doing).

Recently, I found out that clojure is an excellent candidate to do the kind of exploratory programming Java is not good at. Essentially, I find it easier to try the libraries with clojure than with Java. My usual setup is Emacs + swank. The close integration between the buffer and the REPL, means that I'm free to try wrapper functions in the REPL (most Java APIs are quite over-engineered) then move them to the editor and use them to build further abstractions.

For example, here I just wrapped the API to build ER and BA random graphs. It is quick to use in the REPL and then I can easily study the generated graphs directly in the REPL. Notice that this is different from both using unit testing and small programs; in both situations, should I want to run further functions on a graph, I would have to save it to disk and reload it, because a successive run on the program would yield a different graph.

This is another example:

Here I created a bunch of functions to visually present a graph. Essentially I can easily use the macro to experiment with different layouts (that is to say the algorithm placing the nodes. Notice that once again, running things step by step, means that I can try new things on the same graph and compare them.

And these were added at the end to check some standard metrics:

I have to admit that I somewhat missed Common Lisp loop, for the latter.

Technorati Tags: , , ,,

Monday, March 14, 2011

Clojure & Leiningen in a Java Project

These days I've been busy with other stuff to do (including but not limited to some logistics and fixing a couple of papers); however, these morning, I picked up my good old mixed java/clojure project to port it with leiningen.I'm pretty happy with IntelliJ build system, but I needed some more flexibility.

There are some interesting issues with the project:

  1. it has both Java and Clojure files
  2. it has both aot and regular Clojure files
  3. it depends from lots of different Jars
  4. it depends from a "private library" which has not been (and cannot be) placed under publicly available repositories right now

The idea is that: Java code does not explicitly depend on clojure anymore. It is safe to compile Java before clojure; this is a major upgrade from (this and this and made things easier). On the other hand I have some "utility" clojure files which have not to explicitly aot compiled, but are required by the aot compiled sources. And I have some wrapper clojure scripts which depend on everything and is also compiled but it does not give any trouble, apparently.

To give an idea, this is the project:

I also installed lein-javac, even though I'm not sure it is needed. However, I can compile the Java code and that is fine. That was the easy part. And using "official" jar files was also easy Moreover, since interactions from aot compiled and regular clojure code did not give me any issues that is the state:

  1. it has both Java and Clojure files
  2. it has both aot and regular Clojure files
  3. it depends from lots of different Jars
  4. it depends from a "private library" which has not been (and cannot be) placed under publicly available repositories right now

Unfortunately, it seems to me that lein does not detect dependencies among aot compiled sources. That is to say, I have to manually order the files in the correct order. I hope that I just missed some lein feature, because this is just crazy. My first idea was that I could compile all the namespaces that contained all aot compiled code (I structured the project because of that). Apparently, it works; however if the compilation does not take place in the correct order, it does not work. Please, someone tell me I'm wrong. Someone told me and today with a simpler regular expression everything worked. Basically I'm not excluding files which are not required to be aot compiled from aot compilation. I just compile everything and that's it.

The only "problem" was dependency from a proprietary jar file we are developing. I found out that simply placing it in the deps was not enough as the directory is often removed and recreated. I put up a kludge:

Now everything kind of works. I'm not sure it's an improvement over the IntelliJ variant. I can use Emacs+swank (which is good), but I have a tricky "manual" step if I want to add aot compiled files. Besides, since there is a lot of setup to run the project, I'm not even sure that swank is a huge advantage, as I could only use it to run snippets "outside" the core.

So essentially it boils down to whether I prefer Emacs or IntelliJ for this project. Being heavily framework based, exploratory programming is rather hard to do, which removes some appeal from slime, as LaClojure REPL is adequate. Moreover, I prefer IntelliJ on pure Java (and I have some) and it seems to better autocomplete Java stuff (e.g., in the :import sections). So perhaps the trick is IntelliJ+LaClojure+Leiningen.

Technorati Tags:,, ,

Wednesday, March 2, 2011

Practical Clojure

There are many different kind of technology books. Some books introduce the reader to a specific technology or to a concept. Some books are references, some teach the basics of programming (or some other elementary concept). Some books do more things at once.

Some books last. When a book does not teach a specific technology (that is transient, by its own nature, and changes) but a broad concept, they are more likely to last (perhaps with minor modifications). The books I love most are part of the second category.

A particular category of technology books are the ones teaching new programming languages. If the programming language is a good one, the books tend to lean towards the second category, the one of books teaching broader concepts, even though perhaps they need more frequent updates, as languages evolve.

Perlis said that: "A language that doesn't affect the way you think about programming, is not worth knowing." And I could not agree more. Some languages improve your programming in every other language. Lisp is one of such languages. Nice dynamic OO languages don't do that automatically, but in the hands of a careful programmer, they have the same effect.

So, if the language is good and the book teaches you how to actually think in the language, how to use its proper abstractions, then it is teaching you something noteworthy. Unfortunately, sometimes there is so much language to teach, that there is no more space for anything else (C++?). Moreover, some languages just teach programmers to thing the wrong way (C++, Basic).

But this does not really concern us, as we are going to talk about Clojure. Clojure is a wonderful language (even though I actually favour Common Lisp).

Right at the moment, there are two book on Clojure. I already reviewed Programming Clojure in a previous post. The "other" book about clojure is "Practical Clojure".

From a typographical point of view, Practical Clojure sucks. I just hate the way APress books are typeset. I hate the fonts. I hate the spacings. It looks to me as it was paged with word. On the other hand, I rather like the Practical Bookshelf paging, indeed. Still, I find I prefer more the typesetting of some more traditional editors (e.g. On Lisp, most stuff from MIT Press, or even some newer stuff such as NoStarch Press Land of Lisp). But of course it is content that matters.

Practical Clojure uses a newer version of clojure and that is an advantage, of course. A part from this, it is another introductory book on clojure. There is a wide intersection area on the subjects, but the two books have some differences.

Practical Clojure introduces Java interoperability at the end of the book and that means that most examples are "pure clojure" and some central chapters can be used as a reference. Consequently I sometimes read through Practical Clojure even now. I also find some explanations more thorough.

On the minus side, there are fewer long length examples. Consequently, it is harder to figure out Clojure pragmatics. I believe that we really need a different breed of books. Practical and Programming Clojure basically show us the basic Clojure constructs and how to use them. They are introductory books on Clojure (and we need these books as well).

I believe that we need books that build on those two to go deeper. Some books on Lisp are old, but still teach the abstract principles of Lisp programming. Consider for example Graham's books, or PAIP. We need something like that for Clojure. In fact, I'm looking forward to read Joy of Clojure.

Amazon Associates Link

Technorati Tags: , ,