Unix lover: snippet 1.

I love Unix so much... I had this latex file and I wrote the names of the files of the intended chapters to be included in the mainfile. Then I realized that I had to basically write all the filenames or do some crappy copy and paste.

However...

cat <<EOF | sed -e's/\\input{\(.*\)}/\1.tex/' | xargs touch
\input{intro}
\input{cha1}          
\input{cha2} 
\input{cha3}
\input{cha4}  
\input{cha5}
\input{conclusions}
EOF

Spoilt by Python?

To my horror and dismay I found out that my bash-fu (ok, zsh-fu) dropped to the point of seriously sucking. The question was simple... I made a judgment error and put a git repository on my dropbox are and I also used it actively from multiple computers (while I should have just use it to synchronize stuff and edit two clones).

So I basically found out some files like “./.git/index (enrico-eee's conflicted copy 2010-10-16)” and my commits disappeared. Dropbox renamed the “right” file with that name and used an older file. It is not clear why this happened. However, it was not pleasant.

Given my modest knowledge of git (modest, but not non-existent) I supposed that everything could be solved renaming the files to their rightful name. And in fact it did. The point was renaming the files.

My idea was using find. Ok. I tried some solutions and I “almost” got there. In fact a couple of cut and pastes could have finished it. Though I understood that:

1. given my present knowledge of shell scripting, I would have done better using a for loop in the first place
2. considering that I had to start from scratch, I would have finished that earlier if I used Python
3. Lazyness wins over thirst of knowledge if the specific knowledge is dull and uninteresting

I eventually put down a bunch of python lines doing the trick:

   1 import os
   2 import re
   3 import shutil
   4 from os import path
   5 
   6 rex = re.compile('(?P<name>.*) \(enrico-eee\'s conflicted copy .*\)(?P<ext>.*)')
   7 
   8 for root, dirs, files in os.walk(os.getcwd()):
   9     for filename in files:
  10         match = rex.match(filename)
  11         if match:
  12             new_name = "%(name)s%(ext)s" % match.groupdict()
  13             shutil.move(
  14                 path.join(root, filename),
  15                 path.join(root, new_name)
  16             )

I think that my shell scripting abilities are going to decrease. Almost every-time I have a problem I solve it with Python, since I’m far more skilled in Python scripting. Not even sure it is a bad thing.

Ubuntu 10.10: so far, so good

On October 11th I decided to upgrade to the new ubuntu. I wanted to try the update procedure, as:

1. It's not a big issue to reinstall afterwards (right now Linux is not critical for my job)
2. I was really curious (and scared, from what I read on the web... but well, if OS X manages to just update, why Ubuntu should not?)

First I updated the system to the latest version as recomended. Then, I started the proper upgrade process. The installation process took *hours*. Not only because of the download process (here at the University we've got a pretty fast connection ;) ). It was just the setup/configure process. It took somethink like five or six bloody hours (I don't know, since I left the netbook on and went home). Besides, having question asked during the installation process (question blocking the process) means you have to be in front of the monitor for the whole process. So the day after I found some nice blocking panes asking me silly questions[0]. Then it also stopped responding to input. I inadvertently touched the shut down key and the system ignored my tentatives to press the cancel button. However, Ubuntu rebooted the half installed system fine (I think that it was just setting up unimportant applicative packages when I interrupted the procedure). dpkg --configure -a and now *everything* is fine. Perhaps I have been lucky, but I think they are doing a great job. I was expecting an epic failure (my fault, by the way). Usage impressions another day. Maybe. Or maybe not.

Interesting DenyHosts...

http://think.random-stuff.org/posts/denyhosts-on-mac-os-x
http://ptone.com/dablog/2009/03/setting-up-denyhosts-to-block-ssh-attacks-on-leopard/

Python example on atomic files

This is a very simple example on how to write atomically to a file in Python. We already dealt with the subject here.

The code here essentially works only if you plan to rewrite the file from scratch. Append and similar stuff is not supported. Reading is not supported (as it does not make sense, since it has no troubles with letting the filesystem dirty). Notice that this code does not work on windows with the intended semantics, since rename is not guaranteed to be atomic (as far as I know).

'''Simple module containing a file-like object whose behavior is
to actually save the file only when closed. Before that, data is
stored in a temporary file: thus when creating a file, if another file
with the same name exists that is never substituted with a partial
file.'''

import os
import tempfile

class AtomicFile(object):
    '''A file-like object where writes are committed only when closed.

    A temporary named file is used in order to store the results.
    On close, the temporary is renamed according to the name parameter.

    AtomicFiles are meant to be used with context managers.'''

    # Unfortunately the TemporaryFile interface uses a dir parameter
    # pylint: disable-msg=R0913
    # pylint: disable-msg=W0622
    def __init__(self, name, bufsize=-1, suffix='', prefix='', dir=None):
        mode = 'r+b'
        self.name = name
        self.tempfile = tempfile.NamedTemporaryFile(
            mode=mode, bufsize=bufsize, suffix=suffix, prefix=prefix,
            dir=dir, delete=False)

    def __enter__(self):
        return self

    def __exit__(self, _exc_type, _exc_val, _exc_tb):
        self.swap()
        return False

    def swap(self):
        'Explicitly closes and renames the temporary file.'
        self.tempfile.close()
        os.rename(self.tempfile.name, self.name)
    close = swap


    def write(self, what):
        'Writes a string to the file'
        self.tempfile.write(what)

And now a bit of unit-testing (and example usage):

import unittest
import tempfile
import atomic_file

import os
from os import path

class TestAtomicFile(unittest.TestCase):
    def setUp(self):
        self.directory = path.dirname(__file__)
        self.name = tempfile.mktemp()

    def tearDown(self):
        try:
            os.remove(self.name)
        except OSError:
            pass

    def testCreate(self):
        af = atomic_file.AtomicFile(name=self.name, dir=self.directory)
        self.assert_(not path.exists(self.name))
        self.assert_(path.exists(path.join(self.directory,
                                           af.tempfile.name)))
        af.swap()
        self.assert_(path.exists(self.name))
        self.assert_(not path.exists(path.join(self.directory,
                                           af.tempfile.name)))
    def testClose(self):
        af = atomic_file.AtomicFile(name=self.name, dir=self.directory)
        self.assert_(not path.exists(self.name))
        self.assert_(path.exists(path.join(self.directory,
                                           af.tempfile.name)))
        af.close()
        self.assert_(path.exists(self.name))
        self.assert_(not path.exists(path.join(self.directory,
                                           af.tempfile.name)))

    def testContext(self):
        with atomic_file.AtomicFile(name=self.name, dir=self.directory) as af:
            self.assert_(not path.exists(self.name))
            self.assert_(path.exists(path.join(self.directory,                                               af.tempfile.name)))
        self.assert_(path.exists(self.name))
        self.assert_(not path.exists(path.join(self.directory,
                                           af.tempfile.name)))

    def testWrite(self):
        with atomic_file.AtomicFile(name=self.name, dir=self.directory) as af:
            text = 'THE TEXT\n'
            af.write(text)
        self.assertEqual(file(self.name).read(), text)

    def testMoreWrite(self):
        with atomic_file.AtomicFile(name=self.name, dir=self.directory) as af:
            lines = ['THE TEXT', 'MORE TEXT', 'AGAIN!']
            for line in lines:
                print >> af, line
        self.assertEqual(file(self.name).read(), '\n'.join(lines) + '\n')

    def hasExplosion(self):
        with atomic_file.AtomicFile(name=self.name, dir=self.directory) as af:
            raise RuntimeError()
        self.assert_(not path.exists(self.name))
        self.assert_(not path.exists(path.join(self.directory,
                                               af.tempfile.name)))
    def testBoom(self):
        self.assertRaises(RuntimeError, self.hasExplosion)

Atomicity and files

We all greatly value atomic operations. Either they fully succeed or they completely fail. Even better, if they fail, it is like they never occurred, so that it is not necessary to clean things up.

The term itself is widely used in the database community (the A of ACID stands for Atomicity) and in the parallel computing community. The meaning is essentially very similar, though the scope is different. Moreover, the idea is pervasive in computer science.

Writing exception safe code in C++[0] is essentially a strive for atomicity. C++ code is strongly exception safe if it has rollback semantics. That is to say failed operations have no side effects and consequently leave all data as it never happened. More details here [1]. It is usually considered extremely expensive to write all the code to be strongly exception safe.

In languages such as Clojure this comes for free. Memory is transactional[2], that is to say, it support the ACI (without D) properties of a relational database. Of course durability is not an issue: we are dealing with memory, after all. This is not extremely expensive as, by default, variables in Clojure are not modifiable. Essentially they are but names of const object. If a variable needs to change its value, vars, refs or atoms (but they are an entirely different story) must be used. And they are governed by a software transactional memory system [3].

Another famous source of atomic operations, is the POSIX standard. Some system calls are guaranteed to be atomic. In the following, we are mostly concerned with open (2)[4] and rename (2)[5].

In the open(2) system call, the check for the existence of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_EXCL and O_CREAT set.
For this reason, open can be used to implement concurrency structures such as locks; the link (2)[6] system call is often used as well. Moreover, the open function shall be used in Easier to Ask Forgiveness (EAFP) strategies [7], which are fare more secure and effective because open is atomic.

Unfortunately for us, the write(2) [8] system call is only partially atomic. That is to say if the requested write is “small enough” (that is to say the number of bytes are < PIPE_BUF), then the single write action is atomic. If this is not the case, the write is not atomic. Moreover, multiple write actions are, of course, not atomic.

To make things even worse, the whole open-write-close thing is not atomic at all. And is destructive. The typical problem is that when you open a file with O_TRUNC (or with something which in turns calls an open with O_TRUNC set) you destroy the content of the file. However, you may have successive errors which prevents the correct and complete writing of the new file. The typical solution is using temporaries.

References

[0] “Strive for exception safe code”, Effective C++, 3rd ed, S. Meyers, Addison-Wesley, pp. 127-134
[1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/1997/N1077.asc
[2] http://clojure.org/refs
[3] http://en.wikipedia.org/wiki/Software_transactional_memory
[4] http://www.opengroup.org/onlinepubs/009695399/functions/open.html
[5] http://www.opengroup.org/onlinepubs/009695399/functions/rename.html
[6] http://www.opengroup.org/onlinepubs/009695399/functions/link.html
[7] Python in a Nutshell, 2nd ed., A. Martelli, O’Reilly Media, pp. 134-136
[8] http://www.opengroup.org/onlinepubs/009695399/functions/write.html

Pydistutil Configuration

Something I rather hate is mixing up my development environment with my ordinary system environment. I like having things I installed by myself separated from system stuff (and I'm not the only one... think about the convention of /usr/local).

My development machine are essentially single user machines. I am the only user. Thus, global installations are simply an overkill. And I hate pervasive package management systems. For the MacOS, for example, I chose brew. I love it. It does the minimum to automate tedious stuff, but it leaves me complete freedom.

Of course, since I want my package manager to do the minimum necessary stuff, I don't want it to touch my python modules.

Don't touch my Python!

 Well... anyway. As a consequence I heavily rely on python own module manager, that is to say pip/easy_install. You can tell distutils the default base install directory. For example, on Linux, I have in my home directory this .pydistutil.cfg file:

% cat .pydistutils.cfg  
[install]
prefix = ~/.local
install_scripts = ~/bin
install_lib = ~/.local/lib/python$py_version_short/site-packages

which basically makes all the installation in my home-directory.
More information here and here. I use this for modules I plan to use in multiple projects.
Otherwise, virtualenv.

ssh-copy-id

I develop a lot on unix remote machines. I have CVS/SVN accounts and shell access. In fact it's quite tedious to retype the password everytime.
For this reason I started using ssh keys to automate this project. You find a lot of tutorials and informations on this subject, I won't spend more time on this. Basically you have to generate the key and then you have to upload it with scp or similar on the remote machine. However, I find scp syntax quite boring to type... :P
There is a nifty script that does the job for you. You call something like (with you correct remote credentials), it asks you password and you're done.
ssh-copy-id -i ~/.ssh/id_rsa.pub rik0@shell.berlios.de
This script is not in my open-ssh distribution and I had to find it on CVS. The direct download is . However, the script may be upgraded. So if you read this post after some time it has been posted, you may want to find a newer version.