Geek automation: Maven, Ant and other things

About automation, I'm definitely a geek. And I totally agree with the famous graph:

And I was working to a Java project with an unusual deployment. Part of the problem is that the actual application for which I was writing the plugin wasn't available as a maven repository. Moreover, running the whole thing needed some fiddling with long command line options, plus a non-trivial class-path.

I started appreciating maven with leiningen, and then by itself. Moreover, it solves one of the worse problems I have with IDEs. I hate to version IDE files (after all, people may use a different environment, something many eclipse users do not even consider). And I hate to recreate them every time (especially when they are not trivial). Eventually, I hate to depend from a GUI program to actually build a project (where a command line should suffice).

Then there is another problem related to libraries, i.e., I find it annoying all the solution regarding jars:

1. place the jars in a system wide directory akin to /usr/local/lib
2. download the jars every time and manually add them to the project
3. create a script that downloads the jars (and curse the first time you develop on windows)
4. do not use external dependencies

Maven solves all these problems together. I just name the jars in the pom and everything is taken care of. I loved the thing so much with Clojure and Leiningen that even with the despicable xml syntax things work smoothly.

But I had this very important jar that ain't versioned nowhere. So maven was out of question, I thought: moreover. The idea of manually downloading the jar and then adding it to maven's repository (and doing it on the 2-3-4 computer I use) really looked unacceptable.

So I dig into ant most advanced stuff and built an ant script that actually downloaded the jar for me and put it in a lib directory inside the project (packaging jars with the project is not acceptable). But I still wanted to use maven… so I learned the terribly documented ant plugin to get the dependencies from maven… and since that was not installed by default, I also wrote the code to go and get the jar.

So I basically had this ant script that did everything for me… and was some hundred lines long. Eventually, I also wrote some targets to generate scripts to execute the project (I told about complex command lines).

Of course, this also meant that maven was under-used. But then I realized… I have maven. I have a plugin that creates the scripts for me (not part of maven, but maven can easily get it). I only need ant to get the ant-maven bridge (which I wouldn't need anymore) and to get the missing jar. So I'm maintaining a very complex system for basically nothing: it is so much simpler to wget the jar, install it in the local maven repository and go on.

Fun that in order not to write to lines a couple of times I wrote a one hundred lines script.

Ubuntu 11.10

Yesterday I decided to update my netbook to the last incarnation of the Ubuntu distribution. In the past days I read something about bad bugs in the kernel regarding power saving and similar stuff; however, nothing happened here (yet).

I was quite amazed to find a Linux 3.0 kernel. It is the single event that made me think how little I keep myself informed about Linux specific stuff. I still find it hard to study all the languages, technologies and theory which has to do with my work (which is also responsible for me posting very little, lately) and my generic interests (functional programming, for example). I simply dropped most stuff regarding system administration and even platforms. In fact, I haven't upgraded to Lion either.

So I don't know a thing about this Linux 3.0 kernel. On the other hand I remember I followed very closely the new features of the 2.2, 2.4 and 2.6 kernels. In fact, I have also some memories of stuff before that (I used mkLinux, though). I think this means I'm just growing older.

Anyway... I still don't like Unity. However, I'm one of those guys who basically open up a shell a fire some stuff from there (editor, interpreters and compilers). Or occasionally I open IntelliJ/PyCharm. So really, I'm not entitled to talk about that. I simply noticed that the colours moved from that tiring orange to shades of green which are just easier on my eyes. Nothing important. I just find it nice not to have the gnome menus, since I have a very very tiny screen. In this sense, my iPad screen feels just larger (even though it is not). I think it simply depends on the way applications are designed.

I just found out that for some reason I do not have Java properly installed, even if before the update I did have it. It felt akward to run clojure and get a "command not found: java" error; especially considering that up to 5 minutes before that I was debugging a Java project inside IntelliJ. Though I think it has something to do with my current installation of OpenJDK 7.

About the "good" things... Now leiningen is packaged (and decently recen, as far as I can tell); moreover, also clojure 1.2 (and contrib) are installable packages. Perhapsit could be enough... Waiting for Clojure 1.3, by the way.

Too many cool languages (Python, Clojure, ...)

There is plenty of languages I like. I'm naming just a bunch of them here... so a non exclusive list may be:

1. Python
2. Common Lisp
3. Scheme
4. Clojure
5. Erlang
6. Haskell
7. Prolog
8. Ruby

The list is quite incomplete... for example I left out languages from the ML family. I do not have anything against them, I just think Haskell nowadays is way cooler (cooler != better) and also the Haskell platform is a definitive advantage. I did not mention languages such as Io, because although I'd really like to do something with it, it just does do not feel enough motivation to try and do it in Io instead than in one of the languages above.

Same thing for Javascript. It's not that I do not like it... it is just that if I can, I would rather use one of the languages I mentioned (which at least in case of client side JS most of the times is a no go). I also left out things like CoffeeScript and a whole lot of languages that did not even jump to my mind right now.

The list is quite representative of my tastes in programming languages. All the languages are very high level languages (I enjoy plain old C, but I tend to use it to extend the above languages and/or for problems which really need C points of strength). Most of them are declarative languages, almost every one of them has a strong functional flavor (basically just Prolog hasn't, the others are either functional languages or received a strong influence from functional programming).

Some of the languages are better for certain tasks, because they are built for them or have widely tested libraries/frameworks. And this somewhat makes it a difficult choice which one to chose. For example, every time a network service has to be written, I consider Python (Twisted/Stackless), Erlang and recently Clojure. If the system has to be highly concurrent, probably Erlang has an edge, but also Haskell (with STM and lightweight threads) and Scheme (Gambit/Termite) are strong contenders.

This is essentially about personal projects. In fact, for "serious" stuff, I am still using Python or Java (and C/C++ if more muscle is needed). This is basically because I have far more experience with those technologies. Clojure is probably a viable choice as well, as I can easily interact with Java and I have been using that quite intensively in the past two years (I also used it to extend one of the "serious" projects, though I had to remove it because I had other experimental components and it was tiresome to find out whether bugs were due to the other components or my relative inexperience with clojure). Besides, I have still to find winning debugging strategies for clojure: probably I'm missing something obvious here. I'm also trying to increase my expertise with Erlang.

These are some random thought about developing in the environments.

Editing Tools

The first issue, is to find the right tools. It is not exactly easy to discover the right tools to develop in non mainstream languages. For example, with Java any major IDE is a good enough tool. For Python, both PyCharm and WingIDE are excellent and easy to use tools. Moreover, vim is an excellent tool with only minor tweaking. Oddly enough, I'm still struggling to find a good setup for Emacs.

On the contrary, Emacs is quite easily one of the best environments for Scheme/Common Lisp/Clojure (thanks to slime), Haskell, Erlang and Prolog (thanks to the improved modes). Still, after years of switching back and forth from Emacs to vim and vice versa, I think I'm more comfortable with vim. Unfortunately, it is not always the case that customizing vim for these languages is as easy as it is with Emacs. For example, I found an easy to use Erlang plugin, but I have still to try using advanced plugins for Haskell and Clojure (I know the plugins, still I did not finish the setup). For Clojure it's just easier to fire LaClojure or Emacs+Slime. For Haskell, I'm not serious enough to know my own needs. I wrote software for my bachelor's in Prolog with vim, and I have to say that Emacs is quite superior (especially considering Sicstus custom mode).

Back in the day I used TextMate for Ruby. Right now I guess I'd probably go with RubyMine or vim.

Platform

Now this is a serious point. Many of my "difficulties" with Scheme lie here: chose one interpreter/compiler with libraries to do most things I want to do and some tool to ease the installation of other libraries (I should probably just stick with Racket). For example with Python it is easy: pick the default CPython interpreter, use pip/easy_install for additional libraties.

Common Lisp is also going well (SBCL + Quicklisp here). I was pleasantly surprised by Haskell: the Haskell platform is wonderful. Some years ago it was not completely clear which interpreter to use and external tools (like Haskell) were not completely immediate. Now with the Haskell platform you just have everything under control.

About clojure, I feel rather satisfied. For project oriented stuff there is leiningen, which works fine. My main objection is that most of the time I'm not *starting* with project oriented stuff... which is something which so much remembers me of the Java world, where first you start a project (as opposed to unix, where you just start with a file). Luckily enough in the clojure community there are also guys like me, and so there is cljr which just fills the area of "before" something becomes a project.

Cljr is very nice with emacs/swank/slime. Unfortunately I've not yet found a way to make it interact with vim.

Erlang already comes bundled with an interesting platform and rebar does lots of nice stuff too. Erlang/OTP tends to be a bit too project oriented but it quite makes sense. Basically all of the times you are setting up systems with many processes and all and it just makes sense to do it that way.

Prolog is essentially a complete mess. I advise just using SWI-Prolog and forget the rest even exists, unless you need some of the advanced stuff Sicstus offers. In this case you pay and enjoy.

Clojure & Leiningen in a Java Project

These days I've been busy with other stuff to do (including but not limited to some logistics and fixing a couple of papers); however, these morning, I picked up my good old mixed java/clojure project to port it with leiningen.I'm pretty happy with IntelliJ build system, but I needed some more flexibility.

There are some interesting issues with the project:

1. it has both Java and Clojure files
2. it has both aot and regular Clojure files
3. it depends from lots of different Jars
4. it depends from a "private library" which has not been (and cannot be) placed under publicly available repositories right now

The idea is that: Java code does not explicitly depend on clojure anymore. It is safe to compile Java before clojure; this is a major upgrade from (this and this and made things easier). On the other hand I have some "utility" clojure files which have not to explicitly aot compiled, but are required by the aot compiled sources. And I have some wrapper clojure scripts which depend on everything and is also compiled but it does not give any trouble, apparently.

To give an idea, this is the project:

I also installed lein-javac, even though I'm not sure it is needed. However, I can compile the Java code and that is fine. That was the easy part. And using "official" jar files was also easy Moreover, since interactions from aot compiled and regular clojure code did not give me any issues that is the state:

1. it has both Java and Clojure files
2. it has both aot and regular Clojure files
3. it depends from lots of different Jars
4. it depends from a "private library" which has not been (and cannot be) placed under publicly available repositories right now

Unfortunately, it seems to me that lein does not detect dependencies among aot compiled sources. That is to say, I have to manually order the files in the correct order. I hope that I just missed some lein feature, because this is just crazy. My first idea was that I could compile all the namespaces that contained all aot compiled code (I structured the project because of that). Apparently, it works; however if the compilation does not take place in the correct order, it does not work. Please, someone tell me I'm wrong. Someone told me and today with a simpler regular expression everything worked. Basically I'm not excluding files which are not required to be aot compiled from aot compilation. I just compile everything and that's it.

The only "problem" was dependency from a proprietary jar file we are developing. I found out that simply placing it in the deps was not enough as the directory is often removed and recreated. I put up a kludge:

Now everything kind of works. I'm not sure it's an improvement over the IntelliJ variant. I can use Emacs+swank (which is good), but I have a tricky "manual" step if I want to add aot compiled files. Besides, since there is a lot of setup to run the project, I'm not even sure that swank is a huge advantage, as I could only use it to run snippets "outside" the core.

So essentially it boils down to whether I prefer Emacs or IntelliJ for this project. Being heavily framework based, exploratory programming is rather hard to do, which removes some appeal from slime, as LaClojure REPL is adequate. Moreover, I prefer IntelliJ on pure Java (and I have some) and it seems to better autocomplete Java stuff (e.g., in the :import sections). So perhaps the trick is IntelliJ+LaClojure+Leiningen.

Technorati Tags:Clojure,IntelliJ, Leiningen, Programming

'(leiningen slime Emacs) are the way.

Leiningen seems to be really the easiest way to create a clojure developing environment.

Up to now, I worked mainly with IntelliJ + LaClojure. The rest of the project is Java (so having IntelliJ is a huge plus) and is very "project oriented" stuff. In fact, it is not even possible to think about individual scripts or so, things are meant to work in an environment and the environment needs to be started beforehand. As a consequence, a solution which does not work well "one file at a time" is not a big issue right now.

Although LaClojure plugin is very good, I feel more at home with Emacs. Not that I'm been an assiduous Emacs user (I use and used pretty much everything), it is just that Emacs has been optimized in last "don't even know how many years" to work with Lisp. Emacs indentation is beautiful and all. Moreover, leiningen should be a good choice even for my main project.

As a consequence, I decided to test leiningen with smaller projects.  I started a one file project as an example. The first step was just lein new'ing, then I had to manually add the :dev-dependencies [[swank-clojure "1.2.1"]].

$ lein deps

Then it is just a matter of lein swank and the server is there, waiting for my Emacs to connect. I had already installed slime (and swank) for my common lisp development. I installed swank clojure and set some configuration variables in my emacs configuration:

 
(require 'clojure-mode)
(add-to-list 'load-path
             "~/.emacs.d/swank-clojure/src/emacs")

(setq swank-clojure-jar-path "/usr/local/Cellar/clojure/1.2.0/clojure.jar"
      swank-clojure-extra-classpaths (list
                                      "~/.emacs.d/swank-clojure/src/main/clojure"
                                      "/usr/local/Cellar/clojure-contrib/1.2.0/clojure-contrib.jar"))

and that is the rest of the slime related configuration

(eval-after-load "slime" 
  '(progn (slime-setup '(slime-repl))))


(setq load-path 
      (cons "~/.emacs.d/slime" load-path))
(cond
 ((macosx-p)
  (setq inferior-lisp-program "/usr/local/bin/sbcl --noinform"))
 ((win-p)
  (setq inferior-lisp-program "C:/Program Files/Steel Bank Common Lisp/1.0.37/sbcl.exe --noinform")))

; (slime-setup)
(require 'slime)
(require 'slime-autoloads)
(add-hook 'slime-repl-mode-hook 'split-window-vertically)

I think that I should improve the configuration, as it is still sub-optimal. However, with M-x slime I just run a new sbcl slime server, with sbcl client. Then I can use it just to hack with common lisp. On the other hand, M-x slime-connect asks me which server to connect. With default values, if I already run lein swank in my project, it connects to that one (but ports and servers are completely configurable).

At that point I have a familiar lisp developing environment. I can easily compile and load with C-c C-k, reload every function with C-M-x and everything that is standard in slime.

It seems that functions in src/projectname/core.clj can be used directly when they are loaded with C-M-k, but not with C-c C-k. However, I find more useful to do the latter most of the times. Moreover, I believe that files outside src do not load directly in user. Besides, I prefer not to. As a consequence I switch namespace in slime.

(in-ns 'org.enrico_franchi.paip.simplegrammar.grammar)

I believe that this should make clojure.core functions not accessible; however, I have the habit of importing the "core" libraries (at least, I did that in common lisp packages, as recommended). Consequently my namespace declaration is, for example:

(ns org.enrico_franchi.paip.simplegrammar.grammar
  (:use
   clojure.core
   clojure.contrib.combinatorics))

And slime is in my namespace, so I can use the functions as I want (user is not modified) and I can also call clojure.core and whatever I need.

Technorati Tags: Clojure, Lisp, Programming, Leiningen