pl patterns

Blub by Convention (In Defense of Arc)

The entire Arc language is just a few Scheme macros. …Or so the criticism says.

I can’t use Common Lisp. mapcar, lambda, destructuring-bind, etc. These are things I use all the time, yet they have some of the longest, most unnecessary names.

Scheme is the Java of Lisps. define, begin, list-ref, etc. It was seemingly designed to be as descriptive as possible, and in doing so, also became as verbose as possible.

Yes, it’s true that I can redefine all those things. But at that point I’ve created Arc. So if you see the pain of using mapcar and list-ref, then in a nutshell, you see the need for Arc. Period.

In the same way that Perl has functional programming features, but the community and conventions don’t use it that way, which means the libraries aren’t designed for functional code, hindering your use of it, so too are Common Lisp and Scheme hindering. The conventions hinder the use of its full potential.

Lisp is potential. Untapped potential. Arc is an attempt at tapping it.

The blub paradox implies that once you truly get the features of Lisp — first-class continuations, condition-signals and restarts, macros, serialization of code to syntactically valid code — all the imitation features like function objects, exceptions, Ruby’s catch/throw, the C-preprocessor, and serialization to XML are… disappointing at best. Atrociously unusable at worst. This is why Lisp is more powerful than non-Lisps.

But Arc is better than other Lisps because Lisp by definition is flexible. And flexibility is a double-edged sword. You can define anything to be anything — both in a good way and a bad way. Arc was specifically designed to pull out common patterns and be as concise as possible. If it’s true that Arc is just a bunch of macros, then these macros that I would have had to define myself in Common Lisp or Scheme are part of the core language or conventions of Arc. So instead of only me knowing about those macros, all Arc programmers know about them.

Code that does not follow a convention is more difficult to integrate with code that does, so much so, that the fact that libraries exist using a convention is a huge incentive to use the same convention. Additionally, if I write code that does not follow the convention, people who maintain my code inevitably re-write it in their own image. It happens slowly over time perhaps, but inevitably. Every one of those names (mapcar, lambda, list-ref), and dozens more in the core language and standard libraries, is a kludge. But if I redefined them, the first thing someone else touching my code would do is look at it, say “wtf!”, and delete my redefinitions and change everything back to the canonical use. In Arc, on the other hand, instead of my code degrading to the conventions of Common Lisp over time as I move on and others take over my code, it holds the high caliber of the conventions of Arc.

However, if you actually succeed at changing all those kludge names in Common Lisp, you will have improved the language by an order of magnitude. And when you improve something by an order of magnitude, you haven’t made something better, you’ve made something new. (Hence, Arc is genuinely new.)

Basically, non-Lisps are blub compared to Lisp because they limit what you can abstract. However, once you have a dialect of Lisp, there are so few visible barriers to abstraction. Instead, non-Arc Lisps are blub compared to Arc because the conventions adopted by the community — and lack of a ability to change those conventions — create barriers to flexibility, removing one of the very features that makes Lisp so great.

Put another way, non-Arc Lisps are blub by convention. The macros that make Arc Arc are a big deal because they are an attempt to change the convention.

What I am getting at is:

1. Many languages could be better than they are if they used different conventions.
2. It’s damn-near impossible to go against the conventions.
3. Conventions are difficult to change, more of a way of thinking of the community.
4. It’s worthwhile to push good conventions into the core language/libraries, and if you do that, you’re essentially making something new.

This has little to do with Arc, except that Arc is an example where someone (Paul Graham) tried to do #4 because of 1 through 3.

Macro Patterns

I was talking about Lisp macros with Kyle and we decided to write down the patterns of different macros we’ve seen and what they allow you to do, with the intent of teaching others.

Interesting things you can do with macros:

• resource management (e.g. with-open-file)
• behavior injection à la AOP (e.g. memoization, debug printing)
• alternate control-flow constructs
• alternate looping constructs
• alternate binding constructs (e.g. destructuring-bind, cut)
• lazy evaluation
• DSLs
• code generation

Things that macros actually give you that you can’t do otherwise:

• programmatic binding
• non-hygienic binding
• controlled evaluation
• reifying the (calling) source code

If you’ve seen other types of macros or know of good examples of the above, please comment. I will try to update this list as we come up with more.

(Non-)Total Functional Programming

I’ve been getting back into language design again. (I haven’t written anything about it since August!)

Not too long ago I was introduced to Total Functional Programming. The paper is very interesting, and at first, the concept sounds great. Increase the ease of proofs of properties of your code at the expense of Turing-completeness. Who needs Turing-completeness anyway.

But after reading some comments on Lambda the Ultimate, something occurred to me. Maybe removing infinite looping, in an attempt to remove a certain kind of partial function — those that never return — is acceptable. But removing partial functions completely is not acceptable (to me).

To implement division, for example, you’d have to modify the definition of it, or change the type to make it impossible to type-check with a zero divisor. But in general, the compiler can’t prove that a runtime value is non-zero, so this forces the programmer to explicitly handle the case all the time. Maybe that’s desirable in some places, but is it always? I’m not sure.

What I take away from this is that, (1) partial functions won’t go away completely and (2) we have to accept some kind of undefined exceptional value.

Both could be eliminated by making your types über-complicated. But I think that’s the wrong way to go. An intuitive type for division is int -> int -> int or perhaps Num n => n -> n -> n. But not something artificial just to make the function total like Num n, NonZero nz => n -> nz -> n.

Code Maintenance Tool Wish-List

At my day-job, I am forced to maintain code that I did not write, that I am not familiar with, and whose authors no longer work at the company. Did you say “documentation”? Hah! No, my friends… it is just me and the editor.

I am a designer. A prototyping programmer. I do my best work when given vague requirements, lots of freedom, and a blank editor. And in my experience, unlike conventional wisdom, people do not know what they want… That’s a whole post in itself, so I’ll leave it at that.

Not only do I dislike maintenance, I am actually not that great at being a maintenance programmer whose job is to modify or add to an existing (usually unfamiliar) code-base. However, as much as I dislike it, I’m reluctant to say that I’ve actually learned a lot from the experience. I hate it so much that my mind is constantly looking for ways to make it easier. Here is what I wish I had.

Specifying and seeing scope more precisely. For example, I should be able to say these 2 functions can call this, or just the class Foo and this class’s testing module, or this is a project entry-point. Actually, what would be better is to have intended scope, and then have your tool tell you what actually refers to it. (To this end, it would help if dynamic lookup were restrictable. Perhaps whether a function can be looked up dynamically should be a metadata tag on the function.) Additionally, I don’t want to have to search every time I want to see the references to a code object. I want to be able to see this in real-time as I move my cursor, click, or even mouseover something in my editor. The way it is now, finding references feels expensive because you have to search and it takes a long time. As a result, I only do it when I feel like I really have to, which is not that often. But if it were cheap, I would think of it differently. Perhaps I would use it to learn how things are related; I’m really not sure. But I know I would use it differently.

Comments and function names should be indistinguishable from log statements, or at least toggle-able. My editor should allow me to toggle logging of function arguments whenever the function is called. I should never have to type log("in myFunction x=" + x + ", y=" + y), yet somehow I always do. It should be as simple as log(x, y), and this could be solved with something as simple as macros that output the arguments that were evaluated, in addition to the values of those arguments.

Inline, implicit functions instead of code “paragraphs” which are merely lines of code separated by blank lines. These implicit functions would show which variables the block is actually touching as parameters to the function (which can be inferred). These new kinds of function-paragraphs could easily be pulled out into actual functions, with the function application automatically generated in that spot of the code.

Viewing code fully inlined, highlighting patterns. Long functions, as opposed to many short functions, are actually easier to read if you’re not familiar with the code. For one, it is fewer levels of indirection. But also, you can think of functions in a module as a mini domain-specific language. For domain experts, it is infinitely easier to use this terminology. However, when you’re new to the field, you can’t understand anyone because they’re all using what seems like cryptic lingo.

Before the year 2000, who in the world knew what a hanging chad was. But the few experts who coined the term must have found it useful. The first time you heard “hanging chad”, you probably asked, “What the hell is a hanging chad? Pregnant chad??” Likewise, a maintenance programmer should be able to view any function calls he pleases   inlined, basically saying, “Instead of using this term, tell me what you mean in the standard terminology that I already know.” Of course, the editor won’t literally inline the code by copying and pasting; it only displays it that way. After all, code is just data. We can view the code as if it were inlined without throwing away the fact that there is a pattern of shared code. The editor could use something akin to the implicit function-paragraphs I described to keep the patterns already created by the original programmer, and at the same time, slowly teach the maintenance programmer the domain lingo.

Oftentimes when a few classes have commonality, they are refactored into an abstract base class containing the common code and several subclasses which implement or re-implement dynamically-dispatched methods. When you understand the flow of all the classes, this is a great way to factor out patterns and reuse code. However, when you’re unfamiliar with the code, it’s damn-near impossible to look at the flow to try to modify it. A method in the base class is possibly used by the derived classes, but not necessarily. Some might use it, but some might not. Others could use it but also add to it by overriding and explicitly calling the overridden method. There is no adequate tool that I know of that will show me the collapsed view of code for a derived class, to see inherited and overridden methods and all — the final result of all the abstraction and reuse. Firebug does this for inspecting CSS.

Firebug shows you the rules that cascade together to style each element. Rules are sorted in the order of precedence, and properties that have been overridden are stricken out. Each rule has a link back to the file where it came from which you can click to jump to the line.

Any web developer will tell you this is the most useful thing in the world; why not do this for methods in an OO programming language…? With a function, at least you can apply it to get its result in specific cases. With a macro, you can expand it. But with abstract classes — abstractions over classes — you’re stuck having to imagine it all yourself. Sure, you can instantiate the class and then call the methods, but you’d have to do that for each method. Depending on your language, sometimes it’s not even clear what all the methods of a given object are, let alone the source code that created them.

Currently, in order to reuse a piece of code, you have to pull it out (of context) and name it so that the 2 places you want to use it can refer to it (whether it’s a class, a function, a variable, etc.). But pulling something out and having 2 things be semantically linked are separate things. The fact that our tools do not allow us to do one without the other (when sometimes I really do only want one) is an indication of inadequate tools.

There is a general pattern of solving programming problems with an extra level of indirection. However, too much indirection creates more problems, especially when working with code you aren’t familiar with. Or put another way, when someone else works with code you wrote. It is a limitation of the human mind that we must take into consideration.

Programs are written by people, and they must also be read by people. For practical reasons, simplicity and clarity — which amount to human readability — are the first things to be sacrificed when the only benchmark for code being shipped is whether it executes correctly. Like rushing the composition of an essay, which results in prose that can be interpreted by a reader but is not necessarily well-written, code that is rushed is often similarly poorly-written.

It has been said that programs should be written primarily for people, and secondarily for computers to execute. The more I learn and the more I experience, the more I agree with this. And our tools should help this cause.