Does Object-Oriented Programming really suck?

I recently read Armstrong essay on how much OOP sucks. While such opinions would probably have considered pure blasphemy a few years ago, nowadays they are becoming more popular. So, what’s the matter with OOP?

The first thing that comes to mind is that OOP promised too much. As already occurred to other paradigms and ideas (e.g., Logic Programming and Artificial Intelligence) sometimes researchers promise too much and then they (or the industry) cannot stay true to their promises.

OOP promised better modularity, better code-reuse [0], better “whatever”. Problem is, bad developers write crap with every single programming paradigm out there. With crap I really mean “utter crap”. A good programmer using structured programming and a bad programmer using OOP, is that the good programmer’s structured program would probably look “more object oriented” than the other one.

Another problem occurred in the OOP community: OOP “done right” was about message-passing. Information Hiding, Encapsulation and the other buzz-words are a consequence of the approach (because object communicate with messages, their internal state is opaque) not a goal of OOP. Objects are naturally “dynamic” and the way we reason about code is separating concerns in objects having related concerns and having the objects communicate in order to achieve the task at hand. And I’m a bit afraid of talking about “OOP done right”: I really just have my vision. OOP is very under-specified, so that it becomes very hard to criticize (or defend) the approach.

However, OOP becomes “data + functions”. To me data is simply *not* part of OOP. It’s an implementation detail of how objects maintain state. As a consequence, I do not really see data-driven applications as good candidates to OOP. Once again, OOP was sold as universal. It is not. Consider the successes that functional programming is achieving (performance and concept-wise) in this area.

This “data + functions” comes from C++ being the first commercially successful OOP platform. The great advantage was that the transition from structured programming to OOP was quite more easier, that programs could be far more efficient (read “less runtime”) than message passing dynamic OOP systems, at least back in the day. However, there was so much missing from the original idea!

Since back then classes were considered good (and C++ had — and has — no separate concept of interface) + static typing and a relatively verbose language with no introspection, it became rather natural to focus on inheritance. Which later was proved to be a bad strategy. Consequently, there is less experience in building OO software than one would think, considering that for a large part of its history OOP was dominated by sub-optimal practices.

So, what is really bad with OOP? Following Armstrong:

Data structure and functions should not be bound together

Agreed! But I believe that Data Structures are not *truly* part of OOP. Let me explain.

Data is laid out in some way. There is a “physical layout”, for example. You can express that very precisely in C with extensions, to the point of exactly specifying the offsets of the various datas. A phyisical layout also exists in high level languages such as Python. However, it is not part of the Python model. Of course with the struct module or ctypes you can fiddle with it (but they are libraries), however, for the most part, it is just outside the conceptual language used to describe the problem.

Data is also laid out in logical ways. Computer Science defined many data structures and algorithms over them. You can use OOP to express such structures and such algorithms. Still they are not “part” of OOP, they are just expressed in an object oriented way (or maybe not, and remain at structured programming level).

One very good practice in OOP is not using together in the same body of code objects reasoning at different levels. So, for example, you have your low level business logic code that is expressed in terms of data structures, high level business logic expressed in terms of low level business code… and that’s it. Functions is not really together with data. You don’t have “data”. You have some layers of objects.

The point is that OOP is about creating languages at semantic level (you usually do not get to change the syntax). That’s it. If the language is good, well… good. If the language is bad, ok, we’ve got a problem.

Is this suboptimal? In a way, yes. All the indirection may be very expensive. And since abstractions, well… abstract, you may find yourself with a language that is not expressive enough (at least not at the expenses of additional performance costs). Still functions and data structure is not bound together. You just have objects and messages. No “functions” and no “data”. Please notice: this may not be the right thing to do. Still, the problem is not with data + functions, is just related to applying OOP to a specific domain that is ill suited for the approach. OOP is not for every task in the world. But the same objections applies to every system that is built as a stack of layered abstractions.

Please also consider the Qc Na story! Objects are not necessarily opposed to functional programming (you can see them as closures that make different actions available). Objects are just about state + behavior + identity.

Everything has to be an object

And this is something that I don’t think is a problem. It is like saying that Scheme is bad because “everything is a list” (which, strictly speaking, is not true, there are atoms and lambdas etc). If you do not want to program OO, then don’t. If you want, you probably want that everything is an object.

The only issue with everything is an object is, sometimes, performance. From a variety of points of view, such strategy may kill performance. Objects usually have indirections (read pointer). 64 bit pointer for every integer is bad. So we have hybrid stuff like Java and then we have to deal with boxing (either manually in the past or automatically right now). Performance issues can be somewhat “fixed” using proper JIT systems or other optimization techniques. Or providing libraries that do the right thing (see Numpy, for example).

Other than that, I prefer objections in the line of “this thing that should be an object is something else” than those requesting that something that is an object really is. So, I can favorably consider an objection that says that, in Python, “if” is not an object (true). Even though I’m convinced that having if as a statement is not bad either.

And the whole objections with “time”… well, time (and dates) are a bitch to handle. But I find that Python datetime module gets as close to perfection as humanely possible. I really can’t see describing time with a bunch of enums + some structures as an improvement. On the other hand, it looks to me as one of the cases where it is *easy* to see that the OO approach works better.

Consider the related problem of representing a date in a locale. If you introduce a new representation of time, you need either to modify the original function or create a new function (maybe one that works with both things). Creating an interface, on the other hand makes it very easy to introduce new representations of time.

Objection 3 - In an OOPL data type definitions are spread out all over the place

And once again, in OOPL languages data type definitions are not spread out all over the place. They are in C++ and perhaps in Java. And even in Java, if things are done properly, you reason in terms of interface, i.e., in terms of typed messages, not in terms of data structure.
“”“In an OOPL I have to choose some base object in which I will define the ubiquitous data structure, all other objects that want to use this data structure must inherit this object. Suppose now I want to create some “time” object, where does this belong and in which object…
””“
Here it is pretty clear what he has in mind. I think that it is clear that, conceptually, time is an interface and that there is really no need for it to define “data structure”. And if your language is duck typed, the interface is implicit and you have finished before you started.

Objection 4 - Objects have private state

And here I have to agree: state is the root of all evil. Problem is that entirely stateless systems are simply unpractical. We reason in terms of state. We can describe lots of things as stateless, but somewhat we have state. We have files. We have documents. We want them saved and retrieved.
So, we have to deal with state. Agreed. But: 1. imperative programming is about state2. object oriented programming is not, strictly speaking
You can use a “functional” style in OOP, for example. Most of the times, you don’t do it because the code becomes harder to write. But in many situations, on the contrary, you do it because it makes it easier. Often I write immutable objects that cannot be modified: they are not more stateful than a parameter in a function.
Sometimes this is not practical. Ok. And surely using a very stateful style of programming is bad. State in OOP is expressed as behavior. That’s it. Sometimes is done properly, sometimes it is not. Some strategy to deal with state are extremely nice (STM, as an example). Others are not. Also consider how “modern” functional languages really merge concepts from OOP (without being OOP) and how “modern” OOP languages merge concepts from FP (without being functional).
Examples: Python from the OOP side, using generators, lazy evaluation, list comprehensions, closures, etc etc etc.Clojure from the FP side… and all the features it has to express what resembles interfaces, STM and so on…

——

[0] someone once told me they feel lucky when they can actually use the code effectively once, let alone reusing it.

Java and Clojure - How data structures influence language usage

One of the essential points is that we are using object for at least two different things: i) data-types and ii) domain objects. Data-types are usually general entities that can make sense in every program and are not particularly related to a specific application. Examples of these objects are numbers and strings, but also vectors, lists, maps. On the other hand domain objects are domain specific and depend on the specific context. Both a payroll system and a web-server have probably some use for a vector class, however web-server probably does not have an employee class.

The logic level of the objects of the two different kinds is different and code using both should not be written. Essentially, the idea is that low-level code should be encapsulated in methods or functions that work at an higher level, thus making the whole source more details because low level stuff does not clutter the flow of information. Also notice that most standard library code uses low-level abstractions, because it shall be rather domain independent. Thus better data structures mean better code.

So what are these data-types? Languages differ greatly in their choice of datatypes. C has only integers and arrays, for example (ok, it is not OO, but does not matter right now). C++ inherit this. Strings are a library function, so are more well behaved collections. The interesting thing is that the different parts of the STL do not depend much on each other. So for example streams do not play really well with strings (e.g., file names cannot be strings, but in fact streams do not play well with about everything else) and std::string does not have a method such as

vector<string> string::split(string sep);

or, better:

some_range<String> string::split(string sep);

In Java, we have Strings, but collections were retro-fitted. So for example String#split returns an array of strings. In my opinion, one of the essential problems of many "static" programming languages such as C++ and Java is right here: they have the wrong primitives. Writing low level code, is very cumbersome and unnecessarily imperative (C++ STL somewhat makes an exception as it has a somewhat functional flavor the rest of the language lack).

I think that having entities that fit a similar role but have very different usage places a heavy conceptual burden. In Java arrays are really different from collections: they have a nice ugly syntax (but at least they have one) and cannot be used in the same ways collections can. Have different methods, names, facilities. On the other hand, collections lack a literal syntax, which is something that makes them feel "first-class". C++ has basically the very same problem; however, the STL does a great job in normalizing access patterns between STL collections and regular arrays.

Back to the main subject, I feel that Java actually lacks algorithmic code that eases manipulation of collections. In fact, the only "user-friendly" feature is the "for-each" statement. Other than that, code is very imperative and comparable with C code manipulating the same structures. Plus, Maps are really ugly because a first class tuple data-type is missing.

Clojure, on the other hand, has plenty of functions to manipulate low level stuff. This is rather relevant as there is a lot of code that works at this level. Being able to write it quicky is very important. Moreover, using the correct abstractions, avoid bugs. Consider for example the map functions? In a col like

(map f coll)

there can be only one single point of failure: what f does on the single elements. There is no explicit looping, no offset problems, nothing. Consider for example the number of things that can go wrong imperatively writing a loop that calls a function on pairs of consecutive elements in a collection. Compare it with its functional equivalent:

(let [coll (range 10)] (map + coll (rest coll)))

Here nothing can go really wrong. It is easier to write and easier to understand (provided you know a bit of clojure). It is worth noting that real high level object oriented languages have almost this kind of abstraction. But hey, they also have FOF and LOL (first order functions and let ovel lambda -- actually, they do not have let, but the name doesn't matter, does it?). Besides, that kind of code may also be easier to parallelize (in the sense that the compiler could do it).

Then comes class specific code. Object are great, fine. However, sometimes they are just used to group data together. In a sense, it makes sense. In Java there are no tuples, maps are ugly as hell and have the stupid requirement of type uniformity. Which means that either we abuse Object or we can't use them at all.

As a consequence, many specific classes are created. They have no true purpose, but carrying around pieces of data together. And that is just the task for maps, tuples, vectors. Maps, tuples and vectors have a uniform interface, "do the right thing" regardless of the types (in a decently typed language -- new definition… both static, e.g., Haskell, and dynamic, e.g., Clojure, Ruby, Python, languages can qualify! ). And there are plenty of functions to manipulate such stuff.

I think that one of the clearest examples is the Python tuple. A tuple is not magic. Is not clever. It is simple. Still, while in Python functions return only single values, tuples in fact allow to simulate multiple return values. And that is something which is overly useful in many different situations. In Clojure vectors can be used with a similar meaning (in fact, Clojure vectors are quite similar to Python tuples). And the destructuring syntax is amazing (once again, both in Python and in Clojure).

for k, v in some_map:
    do_something_with(k, v)

or

(for [[k v] some-map] (do-something-with k v))
(doseq [[k v] some-map] (do-something-with k v))

Amazingly enough, the set of orthogonal language features used in both languages is roughly the same. Consider that with using a Set[Map.Entry[K,V]] Map#getEntrySet method... here we have this new Entry class. And perhaps somewhere else we have a Pair class which basically does the same thing, but it is a separate type, with different methods and different usage patterns. In dynamic languages at least, as long as the signature of the methods is compatible, something could be saved... but in statically typed object oriented languages, good luck with that.

I think these are some of the reasons I find especially pleasing writing algorithmic code in Python or Clojure and rather despise the very same thing in Java.

Does Clojure fix "Fundamental problems with the Common Lisp language (citation)"?

I was looking for some Common Lisp libraries to implement an idea of mine. For lots of reasons I was considering not using Python or Clojure and going directly with Common Lisp (in fact, I think it is going to be Scheme... but it is hard to tell).

As it often happens when following semi-random links on Google, I stumbled on something quite interesting:
Fundamental problems with the Common Lisp language

Nothing extremely new, indeed. The only point that really surprised me was the "hard to compile efficiently" thing. I would have said that in general SBCL is a pretty fast environment. Not as fast as C++, perhaps not even as Java (but I believe that this depends from the specific benchmarks used), but still fast.

However I was mostly interested in the other claimed problems:

1. Too many concepts; irregular
2. Hard to compile efficiently
3. Too much memory
4. Unimportant features that are hard to implement
5. Not portable
6. Archaic naming
7. Not uniformly object-oriented

May seem like a lame argument... but I think that clojure actually addresses all the problems but number 2 and 3. Please notice that I'm not claiming that clojure is a memory hog or that it is slow. Simply put, right now my impression is that common lisp is still faster than clojure. I believe that this is due to Clojure being an additional layer over Java. Java is itself probably marginally faster than SBCL (though your mileage may vary). With marginally faster I mean that really depends on what you are doing and one or the other may result faster.

Alioth benchmarks are, like all benchmarks, not extremely relevant. Still, this is my general impression. SBCL does a wonderful job, in that CL is much higher level than Java and there are many more engineers optimizing the JVM. However, Clojure takes its toll, in that, according to my tests (and to alioth as well) there is quite a lot of work to do to make it catch up with Java.

Probably for high level stuff or specific problems (where in Java there would be essentially some part of clojure runtime/libraries to be re-implemented) they are on par.

About the memory, my feeling is that JVM is a rather memory intensive business, and Clojure can't do much about it. E.g., Python programs usually run using much less memory when confronted with similar data-sets.

I'm not saying that we should drop CL and switch to clojure. However, I believe that clojure addressed some of the problem that many (some?) in the CL community feel CL has.

Erlang and OTP in Action (review)

First time I got into Erlang, it was with "Programming Erlang: Software for a Concurrent World" (Joe Armstrong). It is a very nice book, in my opinion, and I enjoyed immensely reading it. That was like 4 years ago or something. Back then, the functional revolution was just at the beginning: no widespread Scala, no Clojure at all, essentially no F#. Back then it looked like OO was going to rule the industry for years and years, with no contender of sort. Rails was fresh, Django was fresher (back then the APress book was just being released).

I bought the book because I wanted to see this "brand new" technology (20 year old, but just going to make it through in the circles I did frequent). And really, the language looked like 20 years old. Full of Prolog legacy, Unicode who's that guy? and so on. However, it no other piece of software I knew could as easily. Massive concurrency, hot swapping code. Wow.

The language, I did not especially like. The runtime… WOW! As a language, I love Python or Clojure because the way apparently distant functionalities work together and create something even more beautiful. Erlang does not have that at language level. It has at a framework level.

Think about hot swapping code in an object oriented software. First, I somewhat believe that object oriented modules are somewhat more tangled that functional equivalents. Partly because of the object reuse OO promotes (that can really be against you if you want to swap code). Then, there is the whole problem of references vs. addresses. Addresses have an additional level of indirection that makes it far easier to swap a process than it is to swap an object.

But the very idea that state is in the function parameters and that process linking and easy restarting thing are at the very root of how easy it is to swap code in Erlang. But then… back to the books.

The essential problem was that after seeing a bit of Erlang, I thought OTP was not a big deal. Yes, it is easier to use. But also plain Erlang is. And I convinced myself that should I need Erlang, I could just use plain Erlang. Than things changed: I read some OTP using code and I understood not so much of it. That was this year. What I understood, was that it could be helpful. I had to write a concurrent prototype and I welcomed the idea not to write as much code as possible.

In the meantime, I forgot many things about Erlang. In this situations, instead of reading the same book twice, I buy another book to gain perspective. So I decided to buy another book. One of the candidates was "ERLANG Programming" (Francesco Cesarini, Simon Thompson). It also had excellent reviews. Essentially I believe it contains more material than Armstrong's book and is also a bit more recent. However, as far as I understand, is still a bit terse on OTP.

As a consequence, I bought "Erlang and OTP in Action" (Martin Logan, Eric Merritt, Richard Carlsson) instead. And I'm very happy of this choice. It complements Armstrong's book well and extensively covers OTP. In fact, I also believe that the approach is very interesting. Introduce OTP first and learn to use it, then when you know what it can do, you are going just to use that. Then, learn plain Erlang in order to extend OTP when your use case is not covered. And a nice plus was a detailed description of JInterface, which I could need as well.

In fact, I do think that as it may make sense to introduce objects as early as possible in a book on an object oriented language, starting with OTP is a very big plus from a learning perspective. Then perhaps the point is that I did not need to get into a functional mindset (which I think Armstrong book does with more attention.

If the question is however just "learn to think functionally" I believe that "The Joy of Clojure: Thinking the Clojure Way" (Michael Fogus, Chris Houser), LYHFGG or Land of Lisp are probably better alternatives. Another interesting one is "Functional Programming for Java Developers: Tools for Better Concurrency, Abstraction, and Agility" (Dean Wampler), even if it has a whole different perspective.

Too many cool languages (Python, Clojure, ...)

There is plenty of languages I like. I'm naming just a bunch of them here... so a non exclusive list may be:

1. Python
2. Common Lisp
3. Scheme
4. Clojure
5. Erlang
6. Haskell
7. Prolog
8. Ruby

The list is quite incomplete... for example I left out languages from the ML family. I do not have anything against them, I just think Haskell nowadays is way cooler (cooler != better) and also the Haskell platform is a definitive advantage. I did not mention languages such as Io, because although I'd really like to do something with it, it just does do not feel enough motivation to try and do it in Io instead than in one of the languages above.

Same thing for Javascript. It's not that I do not like it... it is just that if I can, I would rather use one of the languages I mentioned (which at least in case of client side JS most of the times is a no go). I also left out things like CoffeeScript and a whole lot of languages that did not even jump to my mind right now.

The list is quite representative of my tastes in programming languages. All the languages are very high level languages (I enjoy plain old C, but I tend to use it to extend the above languages and/or for problems which really need C points of strength). Most of them are declarative languages, almost every one of them has a strong functional flavor (basically just Prolog hasn't, the others are either functional languages or received a strong influence from functional programming).

Some of the languages are better for certain tasks, because they are built for them or have widely tested libraries/frameworks. And this somewhat makes it a difficult choice which one to chose. For example, every time a network service has to be written, I consider Python (Twisted/Stackless), Erlang and recently Clojure. If the system has to be highly concurrent, probably Erlang has an edge, but also Haskell (with STM and lightweight threads) and Scheme (Gambit/Termite) are strong contenders.

This is essentially about personal projects. In fact, for "serious" stuff, I am still using Python or Java (and C/C++ if more muscle is needed). This is basically because I have far more experience with those technologies. Clojure is probably a viable choice as well, as I can easily interact with Java and I have been using that quite intensively in the past two years (I also used it to extend one of the "serious" projects, though I had to remove it because I had other experimental components and it was tiresome to find out whether bugs were due to the other components or my relative inexperience with clojure). Besides, I have still to find winning debugging strategies for clojure: probably I'm missing something obvious here. I'm also trying to increase my expertise with Erlang.

These are some random thought about developing in the environments.

Editing Tools

The first issue, is to find the right tools. It is not exactly easy to discover the right tools to develop in non mainstream languages. For example, with Java any major IDE is a good enough tool. For Python, both PyCharm and WingIDE are excellent and easy to use tools. Moreover, vim is an excellent tool with only minor tweaking. Oddly enough, I'm still struggling to find a good setup for Emacs.

On the contrary, Emacs is quite easily one of the best environments for Scheme/Common Lisp/Clojure (thanks to slime), Haskell, Erlang and Prolog (thanks to the improved modes). Still, after years of switching back and forth from Emacs to vim and vice versa, I think I'm more comfortable with vim. Unfortunately, it is not always the case that customizing vim for these languages is as easy as it is with Emacs. For example, I found an easy to use Erlang plugin, but I have still to try using advanced plugins for Haskell and Clojure (I know the plugins, still I did not finish the setup). For Clojure it's just easier to fire LaClojure or Emacs+Slime. For Haskell, I'm not serious enough to know my own needs. I wrote software for my bachelor's in Prolog with vim, and I have to say that Emacs is quite superior (especially considering Sicstus custom mode).

Back in the day I used TextMate for Ruby. Right now I guess I'd probably go with RubyMine or vim.

Platform

Now this is a serious point. Many of my "difficulties" with Scheme lie here: chose one interpreter/compiler with libraries to do most things I want to do and some tool to ease the installation of other libraries (I should probably just stick with Racket). For example with Python it is easy: pick the default CPython interpreter, use pip/easy_install for additional libraties.

Common Lisp is also going well (SBCL + Quicklisp here). I was pleasantly surprised by Haskell: the Haskell platform is wonderful. Some years ago it was not completely clear which interpreter to use and external tools (like Haskell) were not completely immediate. Now with the Haskell platform you just have everything under control.

About clojure, I feel rather satisfied. For project oriented stuff there is leiningen, which works fine. My main objection is that most of the time I'm not *starting* with project oriented stuff... which is something which so much remembers me of the Java world, where first you start a project (as opposed to unix, where you just start with a file). Luckily enough in the clojure community there are also guys like me, and so there is cljr which just fills the area of "before" something becomes a project.

Cljr is very nice with emacs/swank/slime. Unfortunately I've not yet found a way to make it interact with vim.

Erlang already comes bundled with an interesting platform and rebar does lots of nice stuff too. Erlang/OTP tends to be a bit too project oriented but it quite makes sense. Basically all of the times you are setting up systems with many processes and all and it just makes sense to do it that way.

Prolog is essentially a complete mess. I advise just using SWI-Prolog and forget the rest even exists, unless you need some of the advanced stuff Sicstus offers. In this case you pay and enjoy.

Clojure Macros: Wanted and Unwanted Symbol Capture

Introduction

No language is perfect. Most languages do not even come close. Some give the programmer the tools to bridge the gap between his idea of perfection and the actual language. Most of these tools are under the "meta-programming" umbrella, and macros are probably the more powerful.

Macros are dangerous tools, indeed. Most articles, books and posts on macros mention somewhere not to abuse macro usage, because they make the programs less clear (TODO: mention somewhere not to abuse macro usage; DONE).

One of the trickiest part is avoiding unwanted symbol capture. Here the the Lisp world divides: some think that symbol capture should be syntactically banned (or something like that), some believe that it is the programme's task to avoid unwanted symbol capture (with the underlying assumption that wanted symbol capture is fine).

I’m not usually happy when a feature is designed not to let me do something; well, quite depending from that something, indeed. I’m not at all concerned with higher level languages not letting me hacking directly with memory; but as far as I can tell, that is more a question of “language model” than of “designing a language not to let me do something”. In fact, Hoyte goes a long way in motivating that symbol capture is not a bug, but a feature.

I believe he is right. I do not find it hard to find situations where I want symbol capture to occur. Of course, in his book, he gives plenty of examples about that; but that’s not the point. I am limiting this to the context of clojure.

Clojure

First, clojure has a nice feature to make it easier to avoid unwanted symbol capture without actually forbidding that. I’m talking about placing a # at the end of an identifier to gensym a new symbol. This is nice to have, nice to use, nice to understand and solves brilliantly the common cases.

But the point is wanted symbol capture. In Clojure we have an interesting example: when creating a gen-class, the methods take an additional explicit parameter “this” (a-la Python, %s/this/self/gi). Like in Python, this is a conventional name, and could be called in any way; still, since Java has an implicit this reference to the current object, it is nice to maintain the convention.

This is a typical example when we explicitly introduce names. It is easy to understand, easy to write and that’s it. There is no capture involved, as we are simply using a function parameter. On the other hand, code in the proxy macro refers to the proxy being created with “this”.

It can’t be changed (as far as I know) since it is injected by the macro. In fact this is wanted symbol capture. The authors want to capture the symbol “this” in the client code and give it meaning. Looking at the code of proxy, it is rather explict.

Both approaches have their advantages: explicit this is easier to understand. Knowledge about how functions work is sufficient to read (and write the code). On the other hand, the proxy macro requires us to know something more: in fact it is a true “new” piece of language. If we did not know that proxy “magically” injects this, we could not understand/write the code. Probably we would say that the code is wrong as “this” should not be bound to anything there.

On the other hand, auto-this in proxies somewhat reduces code clutter. I have not a distinct opinion whether it is a good thing or if I’d prefer explicit this. Perhaps I’d prefer uniformity: that is to say, explicit this everywhere or implicit this everywhere. Though, that part of clojure is under active design and development.

Back to my macro abuses

This post started because of a macro I wrote. The first version is “clean” as it does not capture anything.

(defmacro let-attributes* [this state-method attrs & body]
    `(let [{:keys ~attrs} @(. ~this  ~state-method)] ~@body))

which would be used as:

(defn exp [this foo]
  (let-attributes* this state [bar baz]
    (println bar baz foo)))

Essentially it just wraps let+keys to provide a slightly terser syntax (in fact, I find it boring to use .state all the time; I have to say that it is one of the parts of clojure I like less). However, it is still rather verbose. I have to manually specify things I would rather not; in fact, in my code this is always called this and state is always called state, as I did not find a better name.

So I wrote a second version of the macro. A version that is not clean.

(defmacro let-attributes [attrs & body]
   (apply list 'let-attributes* 'this 'state attrs body))

And the usage would be just:

(defn exp [this foo]
  (let-attributes [bar baz]
    (println bar baz foo)))

In fact, is very dirty as it depends from the context where is expanded (which makes it different from the proxy macro). It assumes that “this” is bound to an object with a .state method. But is faster to write.

Here we can also spot the other problem with macros. I believe that every clojure developer is tempted to write his own macros. Macros that may solve similar purposes, but are inherently different. Code using such macros looks different at a different level than code which just uses different libraries and functions.

This is essentially the problem with fiddling with the language itself, which is what happens when we are writing a macro. Basically every time we write and use a macro, we are developing (inc clojure).

Amazon associates

"Let Over Lambda" (Doug Hoyte)

Technorati Tags: Clojure, Lisp, Programming

Clojure: great exploratory programming

One of the major drawbacks of Java development is the almost complete lack of a REPL. All my favourite languages have a REPL, and I have to say that I've grown completely accustomed to it. I almost can't do without. The possibility to explore new libraries step by step, perhaps with some features common on IDEs (and some more, as it's runtime), is invaluable.

It is hard to understand how the lack of a REPL cripples the learning process of the language (in the first place) and of libraries without having extensively used a REPL. Java somehow compensate this with extensive IDE support; it is possible to write many small programs to try the libraries. However, it is a poor substitute for a true REPL. Java has some third party REPLs, but I did not feel at ease with them. Rather, I resorted on unit testing (which has the added benefit of documenting what you are doing).

Recently, I found out that clojure is an excellent candidate to do the kind of exploratory programming Java is not good at. Essentially, I find it easier to try the libraries with clojure than with Java. My usual setup is Emacs + swank. The close integration between the buffer and the REPL, means that I'm free to try wrapper functions in the REPL (most Java APIs are quite over-engineered) then move them to the editor and use them to build further abstractions.

For example, here I just wrapped the API to build ER and BA random graphs. It is quick to use in the REPL and then I can easily study the generated graphs directly in the REPL. Notice that this is different from both using unit testing and small programs; in both situations, should I want to run further functions on a graph, I would have to save it to disk and reload it, because a successive run on the program would yield a different graph.

This is another example:

Here I created a bunch of functions to visually present a graph. Essentially I can easily use the macro to experiment with different layouts (that is to say the algorithm placing the nodes. Notice that once again, running things step by step, means that I can try new things on the same graph and compare them.

And these were added at the end to check some standard metrics:

I have to admit that I somewhat missed Common Lisp loop, for the latter.

Technorati Tags: Clojure, Java, Programming,Lisp,Common Lisp

call/cc! I yield! Bah, lazy-seq.

Modifications

1. 03-01-2011: an additional scheme variant has been added.
2. 03-02-2011: minor modifications

Imperative Iterators

The archetypal implementation of imperative iterators are Java's subclasses of the Iterator interface. Everyone who has ever implemented an Iterator should acknowledge the pain that is to explicitly maintain state between calls. The essential problem is that the state has to be explicitly maintained. In particular there are two different concepts of state: the iterator "low level" state and the iteration state. In the iterator we have to save "what we have done" and explicitly do so.

Then every time next() or hasNext() are called, we have to do some more computation, save our internal state and return the control explicitly to our caller. This is the implementation of a Javish iterator in clojure to pre-order visit a tree like structure.

I decided to use Clojure instead of Java because the example structure was easier to write, the example could be just run in a REPL, i prefer Clojure to Java and other irrelevant details. However, that is essentially the way iterators are written in object oriented languages (included nice ones, like Python) or languages that have to interoperate with OO languages.

The principal issues is that the "value returning" logic has to be mingled with the iterator bookkeeping logic; I believe that this (although more versatile) is far more complicated that simply adding all the values in a (flat) structure and returning the structure. The reason why this is not the dominant approach is that it is less flexible and less efficient. We also notice that iterators can be infinite.

Ah, if you like OO mumbo-jumbo, this is of course the Iterator pattern.

Yield Statement

In this section we briefly examine Python yield statement. The corresponding yield expression (which essentially allows limited coroutines) is not considered here. If a Python function contains the yield keyword, its semantic changes.

The yield statement may only be used inside functions. A function that contains a yield statement is called a generator function.

[...]

When a generator function is called, the actual arguments are bound to function-local formal argument names in the usual way, but no code in the body of the function is executed. Instead a generator-iterator object is returned; this conforms to the iterator protocol[6], so in particular can be used in for-loops in a natural way.

[...]

Each time the .next() method of a generator-iterator is invoked, the code in the body of the generator-function is executed until a yield or return statement (see below) is encountered, or until the end of the body is reached.

If a yield statement is encountered, the state of the function is frozen, and the value of expression_list is returned to .next()'s caller. By "frozen" we mean that all local state is retained, including the current bindings of local variables, the instruction pointer, and the internal evaluation stack: enough information is saved so that the next time .next() is invoked, the function can proceed exactly as if the yield statement were just another external call.

Essentially instead of using the structure of the previous example, we could have written:

The idea is that when yield is encountered the generator "returns" the value and when next() is subsequently invoked, it resumes execution from that spot. Essentially it is a neat syntactical facility that "saves" somewhere the continuation of the computation in order to be ready to resume from that point.

call/cc

Ok. I said it. Continuation. That is. When the goal is to experiment with control, scheme comes to mind. This is our representation of the tree:

Now I provide an easy example for doing the preorder "functionally". The idea is to provide a function taking a callback function that is called on each element of the tree when is visited. The function also returns the return values of the callback function (in reverse order); anyway, use display as a callback function and we have our familiar example:

Here we did not use continuations. In the following example, I use continuations to save the state of the computation where to resume (after the yield) and to jump out (simulating python yield statement) with another continuation. I think this code should be written more clearly; I'm no expert with continuations (I usually find them too general), so suggestions are welcome.

When generator is called, it returns a function that every time that is called returns the successive element. It behaves like an almost like an iterator and has the structure of Python generator functions. Boilerplate code could be avoided using a macro. Essentially, lines 2-6 are boilerplate and similarly are lines 11-13 and 18. The difference with a "java like" iterator is that when there is no way to test if there is a "next" element (no hasNext method, closures are objects with just one method! ;)) and it returns null forever. So the implicit protocol is "do not put null objects in the tree" and if you get null, then you are done. Other strategies could be used, e.g., instead of returning the value, a list containing the value is returned; empty list means that the iteration completed; list containing an empty list meant that the tree contained a null valued node.

We could have used continuation passing style and that would perhaps have been more similar (even though far more complex to understand and write) than the corresponding Clojure way to do it.

In the comments a different scheme continuation based solution has been suggested by Marco. It is far more elegant than my original solution, but is structurally different from the Python code of the sections above. In fact, it does not use an explicit stack to manage the tree traversal (as both the Python version and my scheme version do), but relies on more recursive calls, essentially using the implicit call stack.

What about lazy-seq?

Essentially we specified that iterators are needed because they are efficient and more general. Indeed in languages such as Python, generators are an easier way to write iterators. What about Clojure? The example at the very beginning is not the way things should be done in Clojure. On the other hand the iterator in scheme would be a good API for Clojure as well. In fact it is the very same pattern that we have with map (Haskell programmers here may want to extend the discussion on "mapping" on different algebraic types, such as Trees, and not only Lists).

Imperative iterators are just a design pattern. Call/cc is a feature so powerful you can use it build almost every other control abstraction. Yield can be used to implement coroutines (which could also be implemented with call/cc). And what about lazy-seq? Lazy-seq is nothing like that. Many Lisp books show a simple example where macros are used to implement lazy evaluation (or at least list lazy evaluation, which is what matters most). lazy-seq basically does that. The difference is that other Lisps are usually not built around the idea of lazy sequences the way clojure is.

Consequently, most libraries and APIs do not work (or do not play well) with lazy evaluation. They could be made to work, but it is simply outside Common Lisp pragmatics, and there is not anything wrong with that.

But back to lazy-seq... why it is here? Consider the code below.

I want to point out the structure of the tree-seq-aux function. Indeed, the function is not recursive. It simply return a lazy-seq. The second parameter of the cons cell we put in the lazy-seq, is another call to tree-seq-aux. However, this is not recursion: when lazy-seq is evaluated, we already returned from the previous call.

But thing are even more interesting than this. That cons cell is somewhat similar to the iterator we saw at the very beginning. Its car (first) is the current value. It is "next". And what is its cdr (rest)? Suppose that our iterator was not stateful. Suppose instead that calling next() returns both the current value and another iterator which will provide the successive values. Well... basically it is our cons cell.

Pair<Value, ConstIterator<Value>> v = it.next();

That is essentially our cons cell. But in fact we somehow have the same pattern we had in the continuation based example. We "save" what to do next (akin to the jump continuation) in the cdr of the cons cell; however, we do not have to evaluate it right now because of the lazy-eval. As I already said, if we used continuation passing style, we would have placed the recursive call which we put in the lazy-seq cons cdr in the explicit continuation.

The advantage is that it is easier to write, easier to understand, easier to use. Unfortunately is much more specific and powerful. But I don't mind practical tools, when they have the elegance lazy-seq has.

Technorati Tags: Clojure, Lisp,Scheme, Python, Programming