Accessing Elements

Let’s say, for example, that we have a vector v with 1500 elements and we want to get the one at index 1101. There are three different ways to find an element in a vector by index: the nth function, the get function, and using the vector as a function. They differ in how they handle the vector being nil, the index being out of range, and whether they support a “not found” argument. For this discussion we’ll use nth; it returns nil if the vector is nil, throws an exception, if the index is out of range, but you can pass an optional argument that is returned, when the index is not found.

(nth v 1101)

Internally, this function is mapped to a call to the nth method in PersistentVector:

public Object nth(int i){
    Object[] node = arrayFor(i);
    return node[i & 0x01f];
}

The arrayFor method handles the lookup in the internal structure and returns on of the 32-element arrays where the elements are stored — either from one of the leaf nodes or from the tail (line 2). The index in that array is calculated by applying a bit-mask to the index passed into the method (line 3). For our example that local index is 13.

public Object[] arrayFor(int i){
    if(i >= 0 && i < cnt)
        {
        if(i >= tailoff())
            return tail;
        Node node = root;
        for(int level = shift; level > 0; level -= 5)
            node = (Node) node.array[(i >>> level) & 0x01f];
        return node.array;
        }
    throw new IndexOutOfBoundsException();
}

The arrayFor method checks that the index is valid and throws an exception if it is not (lines 2, 11). If the requested index is greater than the number of values in the tree, the tail array is returned (lines 4,5). In our example, this is not the case, we need to look into the tree.

As a tree with two layers (root and leaves) can hold 1024 elements, the tree in for a vector with 1500 elements needs three layers. The index we’re looking for is in the second subtree, as it’s greater than 1024. The field shift which holds a multiple of 5 proportional to the height of the tree is 10 in our case, so that we enter the loop in line 7 with 10 as level.

Inside the loop, we find the subtree holding our value by bit-shifting the index by the current level and applying a bit-mask to get it into the 32-element frame. The index for the subtree in our example is 1, as expected. In the second iteration of the loop, we look at the leaves, so that level is decremented to 5. This time, we find the node with index 2. The loop terminates here, as there is not level further down. The method returns the array attached to the node we found.

Summarizing all the index-offsets, we have:

• the second subtree of the root (the first subtree contains 1024 elements),
• the third leaf of that subtree (the leaves contain 32 elements each), and
• the fourteenth element of the leaf (calculated in nth).

Like the cons operation we saw last time, the number of steps necessary to find the nth element depends on the height of the tree, so the complexity here is once again O(log32 N).

Deleting Elements

public PersistentVector pop(){
    if(cnt == 0)
        throw new IllegalStateException("Can't pop empty vector");
    if(cnt == 1)
        return EMPTY.withMeta(meta());
    if(cnt-tailoff() > 1)
        {
        Object[] newTail = new Object[tail.length - 1];
        System.arraycopy(tail, 0, newTail, 0, newTail.length);
        return new PersistentVector(meta(), cnt - 1, shift, root, newTail);
        }
    Object[] newtail = arrayFor(cnt - 2);
    Node newroot = popTail(shift, root);
    int newshift = shift;
    if(newroot == null)
        {
        newroot = EMPTY_NODE;
        }
    if(shift > 5 && newroot.array[1] == null)
        {
        newroot = (Node) newroot.array[0];
        newshift -= 5;
        }
    return new PersistentVector(meta(), cnt - 1, newshift, newroot, newtail);
}

First, the edge-cases are handled: pop on an empty vector does not work (lines 2,3), and pop on a one-element vector returns an empty vector (lines 4,5). Then we look at the easy case that the tail holds more than one element (line 6); we just copy all tail elements except for the last one into a new array and use it to create a new vector that shares the tree with the original (lines 8-10).

The condition for the simple case is phrased so that we don’t use it on a tail with only one element. This is because we need to change the tree when the tail run empty; the last node becomes the new tail in that case.

Shrinking the tree starts with getting the array from the last node by calling the arrayFor method (line 12). Next, we call popTail to get a new root node for a tree without the last node (line 13). This leaves us with three cases:

1. The node we removed was the only node, so that our return vector gets an empty node as root (lines 15-18).
2. The tree has intermediate levels between the root and the leaves and the removed node was the only leaf in the second subtree, so that we can remove one layer (lines 19-23).
3. Otherwise the new tree still has the same height as the old tree.

Summary

• Vectors are implemented based on a tree and a tail array.
• All arrays (both in the nodes of the tree and in the tail) have up to 32 elements.
• The leaves always have exactly 32 elements in them.
• Elements can added and removed efficiently at the end of the vector.
• Addition, removal, and index lookup are O(log32 N), which is essentially constant time in practice.

This post is part of the immutable data-structure canon. While the implementation of immutable lists is relatively simple, immutable persistent vectors require quite a bit of work. I decided to split this topic into two parts: this post covers how vectors are created and how elements; we’ll also learn about the structure used to store the values. The second part to be posted next week, covers how vectors are accessed and how elements are modified and deleted.

Before we dive in, I need to correct an error in the previous parts of the series. Until now, I used “sequence” and “seq” synonymously as a generic term for data-structures that can hold multiple values. I realized now that the correct generic term is collection. Sequence in the Clojure context is a collection that a series of values without reordering them where values may or may not exist yet. seq specifically refers to an API for using collections with the funtions first and rest. Henceforth I shall use the right terminology. And now: vectors.

To achieve both immutability and performance, Clojure only stores limited segments of a vector in a row in memory. The overall structure used to represent the vector behind the scenes is a tree. Each of the tree’s nodes holds an array that contains a segment of the vector. Functions that “modify” a vector return a new object that shares all the nodes not affected by the operation with the original. Only the segments that are different get stored in separate node objects.

The implementation of vectors is in the Java class clojure.lang.PersistentVector. Instances of that class provide the API through which vectors are used and they hold a reference to the root of the tree. The nodes are implemented in the internal class Node.

Creating Vectors

static public PersistentVector create(List items){
    TransientVector ret = EMPTY.asTransient();
    for(Object item : items)
        ret = ret.conj(item);
    return ret.persistent();
}

The actual work here is done by TransientVector, an internal class used for efficiently modifying vectors. The method starts by creating an empty TransientVector (line 2). The elements are inserted into it by one by calling conj (lines 3, 4). Finally, the resulting transient is converted to a persistent representation (line 5).

Transients are a feature of Clojure that allows you (and the language’s internals) to use mutable data-structures in performance critical parts. Transients are thread-save and they cannot be shared with other code. As we’re looking at immutable data-structures here, we won’t go into the details of the implementation here. The vector that we get from the call to persistent has the same structure we would get by starting with an empty persistent vector and adding the values with immutable operations. So let’s look at that.

Adding Elements

public PersistentVector cons(Object val){
    int i = cnt;
    //room in tail?
    if(cnt - tailoff() < 32)
        {
        Object[] newTail = new Object[tail.length + 1];
        System.arraycopy(tail, 0, newTail, 0, tail.length);
        newTail[tail.length] = val;
        return new PersistentVector(meta(), cnt + 1, shift, root, newTail);
        }
    //full tail, push into tree
    Node newroot;
    Node tailnode = new Node(root.edit,tail);
    int newshift = shift;
    //overflow root?
    if((cnt >>> 5) > (1 << shift))
        {
        newroot = new Node(root.edit);
        newroot.array[0] = root;
        newroot.array[1] = newPath(root.edit,shift, tailnode);
        newshift += 5;
        }
    else
        newroot = pushTail(shift, root, tailnode);
    return new PersistentVector(meta(), cnt + 1, newshift, newroot, new Object[]{val});
}

For some reason, the method starts with initializing a variable i that is never used (line 2). After that, we are confronted with the details of structure used to store the values.

Trees don’t have tails — not even the strange computer science kind that grows downward. The tail here is not part of the tree, it’s an array holding values that are not part of the tree. Up to 32 values end up in the tail, the rest goes into the tree. As the name suggests, the last elements of the vector get stored in the tail.

32 is a magic number for vectors: the leaves of the tree have exactly 32 values, the other nodes have up to 32 children, and the aforementioned 32-element tail. The tailoff method called in line 4 returns the greatest multiple of 32 that’s less than the current length, i.e. the number of elements in stored in the tree. The field cnt is the length of the vector.

When there is still room in the tail, the tail array is copied into a new array (lines 6, 7) and the new value is appended (line 8). The return value is a new PersistentVector instance with the new tail array, an incremented count, and otherwise the same properties as the original vector (line 9). In this case adding an element is a constant time operation.

Only when the vector to which we’re adding has exactly 32*n elements (with n being a positive integer), we need to deal with the tree. In that case, a new tree node to hold the original vector’s tail is created (line 13). Then we create a tree with the new node in it (lines 14-24), and we return a new instance with the updated tree, the incremented count, and a new tail array that only contains the added element (line 25).

When constructing the tree, we have two cases: if the tree is completely full at the current height, the resulting tree needs a new level (lines 18-21). Otherwise, we only need to find the right place for the new node (line 24).

The field shift is the height of the tree multiplied by 5. The height is represented in this form so that it can be used with the count and the capacity in bit-shift operations for additional efficiency. When we add the 33rd element, cnt is 32, as that is the value of the original vector. shift is 5, the initial value when there is only an empty root node. As a consequence, the root of the tree for the returned vector is constructed by calling pushTail.

private Node pushTail(int level, Node parent, Node tailnode){
    int subidx = ((cnt - 1) >>> level) & 0x01f;
    Node ret = new Node(parent.edit, parent.array.clone());
    Node nodeToInsert;
    if(level == 5)
        {
        nodeToInsert = tailnode;
        }
    else
        {
        Node child = (Node) parent.array[subidx];
        nodeToInsert = (child != null)?
                        pushTail(level-5,child, tailnode)
                        :newPath(root.edit,level-5, tailnode);
        }
    ret.array[subidx] = nodeToInsert;
    return ret;
}

The method starts by calculating the index where to add the new node depending on the length of the original vector and the level of the tree where we are looking to insert the node (line 2). In the example of adding the 33rd value, the index of the new node is 0, as this is the first node to be pushed into the root. When we add the 65th element — thus having filled up the tail for the second time — we get the index 1.

The return value of pushTail is a new node that clones the references to all the children of the original parent node, so that the child nodes are shared between the old and the new vector (line 3). If we are at the level where the child nodes are leaves, we directly insert the node we’re supposed to add at the calculated index of the returned node (lines 7, 16, 17).

When the tree is already higher, we cannot add the leaf node directly to the parent node. Instead we find the child node at the calculated index (line 11) and use the recursive nature of trees to call pushTree again for that subtree with a lower level (line 13). If no child node in the calculated place exists, we call newPath to create a new subtree of the appropriate hight that contains nodes with exactly one child down to our newly added leaf node (line 14). These operations only copy the subtree where the new value is added. Because of the 32-children condition these are at most O(log32 N) steps.

This was the part of the insertion process, when we can insert into the tree without adding another level. Back in the cons method, we still need to look at the case where we grow the tree in height (lines 18-21). Here we create an all new root for the returned vector (line 18) that gets the original vector’s root as the first child (line 19) and a new subtree of the right height with the new leaf node created by newPath as the second child (line 20). Finally, the field shift for the new vector is incremented by 5 to reflect the new height (line 21). As the number of steps for this operation is bounded by the height of the tree, the complexity here is also O(log32 N), which is consequently the complexity of the whole algorithm.

Summary

I want to close this post with an illustration of the structure using a before/after example. Let us assume we have a vector with 1056 elements and want to append another value calling cons.

The old vector (shown in blue) has a root node with 32 children each holding 32 values, and a tail with 32 more values. This means the tree is full at is current level. The new vector (shown in red) has another layer in its tree. The first child of the new root is the old root, thus sharing the structure of the old vector without copying. The second child of the new root is a node that has one leaf as child which contains a reference to the old tail — again, sharing not copying. The tail of the new vector contains a single value, the one we added.

This post is a slight detour in the course of the immutable data-structure canon. In the previous installation, I described how some operations on lists return lazy sequences to retain immutability and performance characteristics. As this behavior is not specific to lists, and indeed fundamental to all Clojure data-structures, I decided that lazy seqs deserve a closer look.

The Rationale of Lazy Evaluation

Laziness also allows for an interesting special case: infinite sequences. You can specify the computation of elements for a series (e.g. Fibonacci numbers) and use parts of that series without problems, as long as you do not try to traverse the entire sequence.

Postponing Calculations

(c) iStockphoto.com

In a language with general support for lazy evaluation like Haskell, the compiler takes care that nothing gets evaluated before it is needed. In Clojure, however, eager evaluation is the norm. That means that when you call a function with the result of another function-call as a parameter, that second function gets evaluated immediately. In a lazy language, the second function would only be evaluated when the first function needs the respective parameter.

For this reason, lazy-seq (which as we saw last time is used to define lazy sequences), cannot be implemented as a function — the body of what we pass to it must not be evaluated at the time of definition. The solution is a construct often cited as the most powerful feature in Lisp: macros.

Macros are the extension mechanism of Clojure (and Lisp in general); they allow you as a programmer to add features to the language. This makes it possible to have a very small core language, but still provide for pleasant programming. Unlike functions, macros do not evaluate their arguments immediately. When Clojure comes across code that uses a macro it first expands the macro and replaces the code with the result before the compilation proceeds normally.

The definition of the lazy-seq macro in core.clj is surprisingly short:

(defmacro lazy-seq
  [& body]
  (list 'new 'clojure.lang.LazySeq
    (list* '^{:once true} fn* [] body)))

As macro expansion and the corresponding escaping rules are beyond the scope here, let it suffice to say that (lazy-seq BODY) would expand to

(new clojure.lang.LazySeq (fn* [] BODY))

The body that defines the seq is wrapped into an anonymous function and passed to the constructor of the LazySeq class. (new is a special form for Java interoperability that allows us to construct a Java object.) The main part of the implementation of lazy seqs is in Java, more specifically in the aforementioned class clojure.lang.LazySeq.

LazySeq has three fields:

• fn: a function object to store the definition body,
• sv: a generic object to store the computed value, and
• s: a representation of the current view of the sequence.

For performance reasons, LazySeq does not simply call the function object and return its result, when the contents are accessed. Instead the other two fields are used to do some caching.

Caching

public Object first(){
    seq();
    if(s == null)
        return null;
    return s.first();
}

The seq method that is called in line 2, is the key to how access to lazy seqs works. It is also called from all the other access methods (count, more, cons, etc.).

final synchronized public ISeq seq(){
    sval();
    if(sv != null)
        {
        Object ls = sv;
        sv = null;
        while(ls instanceof LazySeq)
            {
            ls = ((LazySeq)ls).sval();
            }
        s = RT.seq(ls);
        }
    return s;
}

The sval method called from line 2 executes the function object and stores the result in the sv field. sval also takes care that the function is not executed more than once. In line 5 the calculated value is assigned to the temporary variable ls, and the field for the calculated value is set to null in line 6. Lines 7-10 resolve nested lazy seqs until we get a value of a different type assigned to the temporary variable ls.

The return value is computed and assigned to the field s in line 11. RT is the runtime class that provides the fundamental Clojure functions for the Java code implemented as static methods. The seq method in RT turns its parameter into a sequence if it implements  the Iterable Java interface.

This computation happens only once. In subsequent calls to seq the cached value in field s is returned, as sv is set to null (line 6)  so that the condition in line 3 is false and the block in lines 4-12 is not executed. The sval method does not set sv again in later invokations either.

The concat Example

(defn concat
  [x y]
    (lazy-seq
      (let [s (seq x)]
        (if s
          (cons (first s) (concat (rest s) y)))
          y))))

What happens when we take the seq resulting from a concatenation and call first on it? Via the first method we get to seq which in turn calls sval, so that the body gets executed.  The body of the lazy-seq use in concat is the call to cons (line 6). The result of cons is an instance of clojure.lang.Cons — an abstraction with list semantics, i.e. it has a head and a tail.  The Cons is assigned to the sv field and — in the seq method – to the temporary variable ls (line 5). The loop for nested lazy seqs (lines 7-10) is skipped, because a Cons is not an instance of LazySeq.

In line 11, the Cons gets assigned to the field s (RT.seq does not do anything to it). Back to first. s is not null, so we call first on the Cons and return the result. The first element of the Cons is the first element of the concatenated seq, just as expected.

Accessing the tail of the seq, happens analogously; the tail of the Cons is another lazy seq due to the recursive definition of concat which we discussed last time.

Summary

• Lazy sequences allow computations to be postponed. Even infinite sequences are possible
• Lazy seqs are created using the lazy-seq macro
• Using a macro avoids immediately evaluating the definition of the seq
• The implementation of lazy seqs is in the Java class clojure.lang.LazySeq
• Lazy seqs cache the results of the computation

Fred Brooks, “The Design of Design”. Page x

Fredrick Brooks, the author of the “Mythical Man Month ” and the project manager of theIBM System/360 (the most successful mainframe computer series back in the day), has written a new book called “The Design of Design”. Like “The Mythical Man Month”, it is a collection of essays, but in this one focus is on how good design can be achieved.

While computer-related projects are a main source of examples in the book, Brooks also draws from experience in designing buildings and other projects. The book is aimed at designers and project managers of many kinds, but computer science is clearly Brooks’s home discipline.

The essays are by no means how-tos trying to teach you a particular design process. Nor are there any singularly novel ideas you have never heard before, if you have some background in designing software. The value of the book is that it makes you think about the different aspects of design. The texts are well written and opinionated, inviting you to look at the issues presented in them from angles that are different from what you might be used to. In the rest of this post, I summarize some ideas that resonated with me, but I really recommend reading the book.

Fred Brooks (c) sd&m. Licensed under CC-BY-SA-3.0

The book is organized in six parts, I follow that structure for my notes here (not all parts are mentioned, however).

Models of Designing

The hardest part of design is deciding what to design

Page 22

In the first part of the book, Brooks explains that he rejects the waterfall model, because it is impossible to know enough about requirements and other factors influencing the design up front.

The Rational Model, in any of its forms [the waterfall model being one of them], leads us to demand up-front statements of design requirements. It leads us to believe that such can be formulated. [...]

The Waterfall Model is wrong and harmful; we must outgrow it.

Pages 33,34

This is old news to anyone sympathizing with agile styles of design, but it is interesting to hear it from someone who worked on mainframes for IBM in the 50ies and 60ies.

The rejection of the waterfall model raises two questions: (1) why are project plans still often based on it and (2) what is a better model?

There are two plausible answers to the first question. It seems obvious, clean and strait-forward. But more importantly, most projects involve some sort of contract between the designer and the customer, where the customer would not be to happy about an open-ended, pay by the hour, it’s-done-when-it’s-done deal.

Brooks proposes decoupling the contracts for design from the contracts for implementation, but concedes that this is not a complete solution especially in software, where the boundaries between design and construction are blurry.

So what about a better model? The purpose of a model for design is to serve as a means of teaching and as map to answer the question “where are we?” in the course of a project.

Brooks has no definitive and final answer to this question, but he is in favor of something based on the spiral model proposed by Barry Boehm. In this model, development repeatedly go through stages of planning, determining requirements and contraints, prototyping, risk analysis, and verification.

Collaboration and Telecollaboration

In the second part, Brooks questions the notion that collaboration and design in teams are good per se.

It is generally assumed that collaboration is, in and of itself, a “good thing.” “Plays well with others” is high praise from kindergarten onward. “All of us are smarter than any of us.” “The more participation in design, the better.” Now, these attractive propositions are far from self-evident. I will argue that they are not universally true.

Page 64

Although there are good reasons from collaborative design such as technological complexity and time contraints, the challenge is conceptual integrity. As a consequence, Brooks argues that a design team should always have one system architect who calls shots in favor of a consistent concept.

Design Perspectives

The third part of the book contains essays with ideas that are similar to those of agile development.

Brooks starts with an interesting perspective on design where he compares different approaches to the philosophical schools of empiricism and rationalism. Rationalists believe that given sufficient experience and careful consideration, you can come up with a perfect design. The empiricist view on the other hand is that anything you design, no matter how well considered, will be flawed. The flaws need to be fixed by trial and error.

Brooks is clearly in the empiricist camp:

Can I, by sufficient thought alone, design a complex object correctly? No; testing and iteration are in practice necessary. But careful thought helps.

Page 109

An important factor for the design is the model you have of your product’s user. While it is impractical to know everything about the users, Brooks argues that you should explicitly state the assumptions you make in great detail, even if some of them are wrong.

Wrong explicit assumptions are much better than vague ones. Wrong ones will perhaps be questioned; vague ones won’t.

Page 117

Reminding me of 37signals he supports the notion that constraints are your friends.

Since constraints are the designer’s friend, if the task originally seems unconstrained, first think harder about what is really desired, about the user and use models, and you will probably find some narrowing constraints, to the benefit of both designer and user.

Page 135

Brooks than goes on to examine esthetics and style in technical design. In his analysis “logical beauty” comes from parsimony, structural clarity, and consistency. Parsimony roughly means “accomplishing a great deal with few elements”. Parsimony alone, however, is not enough. Without any redundancy, as design can be cryptic and infective.

According to Brooks, “consistency underlies all principles of quality.”

A good architecture is consistent in the sense that, given partial knowledge of the system, one can predict the remainder.

Page 143

Three design principles are important for consistency: orthogonality (do not link what is independent), propriety (do not introduce what is immaterial), and generality (do not restrict what is inherent).

Great Designers

In the fifth part, Brooks examines the value of design processes, individual designers, and education in design. He maintains that “great designs come from great designers, not from great design processes.”

I believe that standard corporate product design processes do indeed work against truly great and innovative design.

Page 233

This does not mean, however, that he is rejects processes entirely. The key is some middle ground where processes prevent grave errors, but are not followed to the letter stifling creativity.

First and foremost, the top leader of the organization must passionately want innovative products with great designs

Page 238

Concerning education, the most important idea in the book is that you should learn by critiqued practice, instead of getting taught the basics before starting to make stuff. Another component in learning is teaching to others. A third part in the education of a designer should be studying exemplars — designs by other designers.

Let us review the definition of list in general. A list is either empty or has a head and a tail. The head contains a data element and the tail is another list. A simple recursive definition. The list (1 2 3) for example has 1 as its head and the list (2 3) as tail. The head of that list is 2 and the tail is the list (3). Finally, that list has 3 as head and the empty list as tail.

This structure is implemented as clojure.lang.PersistentList in the Java part of Clojure.

We can create a list using the list function

(def l1 (list 1 2 3))

We also saved the list in a binding named l1. The general convention (at least in Lisps) is that adding elements happens at the front of the list. To add an element, we use cons.

(def l2 (cons 42 l1))
l2
=> (42 1 2 3)
l1
=> (1 2 3)

cons returns a list with the added element, but l still contains only 1 2 3. This property is called persistence. Conceptually this is easy. We create a new list with 42 as the head and what we called l as the tail. l still refers to the same list as before.

Because the addition does not have to go through the existing list, this operation takes constant time.

Given the recursive definition of list, immutably and persistently removing elements from the front of the list is simple as well. Things get trickier when we concatenate two lists.

(def ls (concat l1 l2))
l1
=> (1 2 3)
l2
=> (42 1 2 3)
ls
=> (1 2 3 42 1 2 3)

In a mutable implementation, we could simply change the last element of the first list to refer to the second list as tail, but that would destroy the persistence of l1.

What is the performance expectation here? You might be tempted to say, that it is constant, because we only need to change one pointer. But remember, that we are talking about a singly-linked list. That means, we can only find the last element of l1 by traversing the whole list. Thus the operation takes linear time.

Back to our original question, how do we concatenate two lists without modifying them?

For Clojure the answer is both yes and no.

Clojure’s solution here could be described as “copying light”. We create a new data-structure that has new entries for all the elements, but each entry is only really created when it is used. This is called a lazy sequence.

A lazy sequence behaves like a normal collection. The special power of lazy seqs is that they don’t contain all the data elements, they only know how they are generated. This means that the values in the seq are only created when they are accessed, allowing expensive calculations to be postponed. Lazy seqs can even represent infinite data sets — as long as the data-structure is not traversed completely. We will look into this construct in greater depth in the next part of this series.

Back to our immutable lists. How do lazy seqs help with concatenating two lists? The concat function does not actually return a lists, it returns a lazy seq. When we go through the list, a new list entry is created. Thus for the whole list, we get a newly created entry of each element of both lists, resulting in linear complexity. The data elements, however, are not copied as we can be sure that they are also values and thus won’t change.

Here is the implementation of concat in core.clj:

(defn concat
  "Returns a lazy seq representing the concatenation of the elements in the supplied colls."
  ([] (lazy-seq nil))
  ([x] (lazy-seq x))
  ([x y]
    (lazy-seq
      (let [s (seq x)]
        (if s
          (if (chunked-seq? s)
            (chunk-cons (chunk-first s) (concat (chunk-rest s) y))
            (cons (first s) (concat (rest s) y)))
          y))))

(I’ve omitted the implementation of concatenating more than two lists, as that is not essential to this discussion.)

Let us walk through this code. There are three definitions of of the function here, the dispatch depends on arity (i.e. with how many parameters this function is called). Lines 3 and 4 handle the cases that concat is called with zero or one parameters the obvious way, by just returning a lazy seq with the same contents that were passed as parameters.

Starting in line 5, we have the “real” case for concat: two parameters. Line 6 tells us that the result is a lazy seq. Lines 7-12 are only executed when an element of the list is read — that’s the laziness.

Line 7 makes sure the rest of the function is working with a sequence type, and line 8 makes sure that the first list to concatenate is not nil, otherwise the second list is returned in line 12.

Chunked seqs are an optimization that we’ll ignore here, so we can ignore lines 9 and 10. That leaves us with line 11. Here, we take the head of the first list, and add it (using cons) to the result of a recursive call to concat passing the tail of the first list and the second list.

When we go through the above example of concatenating (1 2 3) and (42 1 2 3), we get a lazy seq back immediately, without the function looking at any of the elements. Only when we go through that result seq, lines 7-12 get invoked. For the first element, we get an entry that contains 1 as head and another lazy seq — the result of (concat (list 2 3) (list 42 1 2 3)) — as the tail. Getting the second happens the same way,  as we resolve the second lazy seq. The third, ditto. After the third element, the first list passed to the inner call to concatis empty, so that the else part of the if in line 8 kicks in, and the result of the call is simply the second list (line 12).

In summary

• Lists in clojure are immutable
• By default, elements are added to the front of the list
• The linked structure makes it easy to implement immutable adding of elements
• Concatenating lists results in a lazy sequence where new list entries are created “on-demand”
• The data elements are not copied, as they are immutable values

(c) iStockphoto.com

Clojure

Immutability is important for clean multithreaded programs, because it allows you to separate values from identities. Rich Hickey, Clojure’s creator, defines these terms as follows:

By identity I mean a stable logical entity associated with a series of different values over time. Models need identity for the same reasons humans need identity — to represent the world. How could it work if identities like ‘today’ or ‘America’ had to represent a single constant value for all time? Note that by identities I don’t mean names (I call my mother Mom, but you wouldn’t).

So, for this discussion, an identity is an entity that has a state, which is its value at a point in time. And a value is something that doesn’t change. 42 doesn’t change. June 29th 2008 doesn’t change. Points don’t move, dates don’t change, no matter what some bad class libraries may cause you to believe. Even aggregates are values. The set of my favorite foods doesn’t change, i.e. if I prefer different foods in the future, that will be a different set.

In Clojure, an identity can be in different states — i.e. have different values — over time. Identities are implemented as atomic references to values using so called Refs and Agents. But these constructs do not concern us in this series, I mentioned them only to give a background on the rationale for using immutable data-structures to model values.

Abstractly speaking, new values are functions of old values. For example, let us define a list and bind it to the name l1.

(def l1 (list 1 2 3))

When we want to add another element, we do not modify the existing list, but we call the function cons create new list with one more element.

(def l2 (cons 42 l1))
l2
=> (42 1 2 3)
l1
=> (1 2 3)

If we had passed l1 to a function in another thread, it could happily work on the initial three elements without being bothered by our addition of 42.

Your initial impression is probably that there is a lot of copying involved in the implementation of the immutable data-structures, but that ain’t so. Clojure’s data-structures maintain close to the same performance guarantees that their mutable counterparts make.

In the following posts, we look at the implementation of lists, vectors, and maps one by one.

• A survey of video on the web shows that 26% of videos are available through HTML5 <video> tags, up from 10% in January. (via the Register)
• An overview of which browsers support which features of HTML5 and CSS3
• A visualization of Molecules using the <canvas> tag

(c) iStockphoto.com/skodonnell

For the first exercise, I’m going to take a look at a simple bit of library code: reading a file into a string. I choose this example, because it turns out that reading a file with Java is more involved than I though it should be.

Coming from Ruby I’m used to being able to read the contents of a file into a string using one line of code like this:

str = File::read "bla.txt"

The simplest solution I found for Java is this

    String str = "";
    try {
        BufferedReader in = new BufferedReader(new FileReader("bla.txt"));
        String line;
        while((line = in.readLine()) != null) {
            str += line + "\n";
        }
        in.close();
    } catch(IOException e) {
    }
    

Let us ignore the missing error handling and the inefficiency of concatenating strings for the moment. The number of lines is not of interest either — this not some “Java sucks”-rant, there are more than enough of that already.

The difficulty of this trivial task illustrates a concept I have been thinking about regularly lately: the Radius of Comprehension which I wrote about here. It is a property of a codebase defined as follows

If you are looking at a given fragment of code, how far away from that bit of the code do you need to have in your mind at that time in order to understand the fragment at hand?

I’m going to make a naive attempt at quantifying the radius of comprehension here. The Ruby solution tentatively gets a score of three: two classes and one method each with meaningful names. I’ll talk about the merits of this quantification at the end of the post, but first let’s get the comparison with Java.

In the Java version, we have two classes, BufferedReader and FileReader, and one method, readLine. Having to loop through the file and concatenate all the lines makes the code more verbose and thus harder to read, but I wouldn’t say it makes it harder to understand. A source of confusion in this code is the indirect relationship between the FileReader and the actual reading. FileReader is a self-explanatory name, but BufferedReader? Having one reader and passing it to another reader from which you can actually read strings in your program increases the radius of comprehension by more than an obvious inheritance. As a consequence I assign the Java code a radius four: one for the method, one for the FileReader, and two for BufferedReader.

Error Handling

In the Java code there is an additional complication: an error could occur when creating the readers or in the call to readLine. In the latter case, we should to close the reader to avoid leaking resources. In Ruby we don’t have to worry about closing anything. Understanding this additional error case in Java adds to the radius. Thus, with error handling we end up with: Ruby: 4, Java: 6.

Again, the intention of this post is not to prove that Java stinks and Ruby is great. I’m looking at a very limited use case here and actively ignore scenarios where the added complexity in Java’s library might be useful. The point is understand more about the radius of comprehension.

What I like about this metric is that the difference between the examples is a factor of 1.5 while the more straight forward metric of lines of code differs by a factor of 10. This score feels about right. The Java code is more complicated, but not hugely so.

Why Quantify the Radius of Comprehension?

Mike Taylor, who invented the term, writes

I’m talking about a human issue here (and therefore, sadly, an all but impossible one to measure, though we know it when we see it)

Code always has two audiences: the compiler or interpreter and other programmers who have to use or maintain it. While we can reliably and immediately assess how well the machine understands our code, feedback on how well others understand it is rare and fuzzy. If we had a way to determine the radius of comprehension, we could judge how well other programmers can understand what we wrote — and take steps to improve it.

– Jason Fried and David Heinemeier Hanson “Planning is guessing” from Rework, page 19

Estimating how long a software project will take is hard. Everyone working in software can agree with this. That’s however where the agreement ends. Is it even possible to make realistic estimates? Is it worth the hassle? Do you estimate big blocks of tasks or do you break the project into small detailed ones before you estimate? How do you deal with changing requirements? Do you estimate the calender time required for a given sub-project or the pure time spend on working on it?

(c) iStockphoto.com/pidjoe

The article quoted above is not specifically about planning software projects, but I think these views reflect what many programmers think of planning projects. However, as few projects are implemented in splendid isolation, coordination with colleagues and partners and communication with bosses and customers usually confronts us with the need to present estimates.

Using the past to inform decisions about the future, is not generally seen as negative as Hanson and Fried do. Evidence-based scheduling is a method for using data about estimates and actual work times of past projects to create more accurate estimates for a future project.

The problem that you don’t know enough to make a realistic estimate at the beginning of the project is sometimes called “Cone of Uncertainty“. The range of uncertainty is enormously wide at the beginning and narrows as the project moves forward.

Cone of Uncertainty (Source: McConnell, Software Estimation, page 28)

The Cone narrows only as you make decisions that eliminate variability.[..] Defining the product vision (including committing to what you will not do) reduces variability. Defining requirements — again, including what you are not going to do — eliminates variability further. Designing the user interface helps to reduce the risk of variability arising from misunderstood requirements. Of course, if the product isn’t really defined, or if the product definition gets redefined later, the Cone will widen, and estimation accuracy will be poorer.

– Steve McConnell “The Cone of Uncertainty” from Software Estimation

Maybe you think the Pontiac Aztek was the greatest car ever made, belong to the Flat Earth Society, and make a pilgrimage to the alien landing site at Roswell, New Mexico, every four years. If you do, go ahead and believe that requirements won’t change on your projects.

– Steve McConnell “The Myth of Stable Requirements” from Code Complete, page 32

When the requirements change, developers or project managers need to update their estimates and fight to get the stakeholders accept that the project will take longer than initially estimated (requirements chages that cause the project to finish earlier are a theoretical possibility — so is life on other planets). Getting this across has a lot more to do with communication than with project estimation.

A major source of communication problems is the confusion of estimates and targets. The estimate is what you think when the work will be done. The often-cited trade show where the finished project is supposed to be introduced is a target. When you’re asked to change your estimate so that you finish in time for the trade show, you’re in a spot of communication bother.

Briefly summarized, the problems Taylor has with libraries are:

• bad quality,
• documentation that fails to explain what the library is about, and
• too libraries many to choose from with no guidance about their strengths and weaknesses.

The best part of the article is that Taylor offers ideas on how to deal with this problem instead of just complaining. And the approach is not some unrealistic, back-to-the-roots suggestion that we all write our own search functions in machine language. His suggestions are formulated as what he calls a “two-prong manifesto”.

Prong 1: Simpler, clearer, shorter documentation. I have long felt that the common Javadoc (or RDoc, or doxygen, or whatever) format is suboptimal for understanding how to use a library, as it doesn’t tell you where to begin. Taylor suggests that a library should have a one-page overview. If a library cannot be summarized in a single page, it is to complex and should be redesigned.

Prong 2: Minimize the radius of comprehension. This one is more abstract, but possibly even more important. Taylor defines the radius of comprehension of a codebase as follows:

if you are looking at a given fragment of code, how far away from that bit of the code do you need to have in your mind at that time in order to understand the fragment at hand? It is a sort of a measurement of how good encapsulation is across the whole codebase, although when I say “encapsulation” here, I am using that term in a broad sense that means more than just technical issues such as what proportion of data members are marked private. I’m talking about a human issue here (and therefore, sadly, an all but impossible one to measure, though we know it when we see it).