Tuesday, August 14, 2007

The Imminent Rise of the Metaprogrammer


Caveat Lector: I shall indulge here in that most futile of all exercises, predicting the future. Allow me to put on the cloak of Merlin (not to mention his infamous pointed hat), look at the crystal ball, and solemnly proclaim, announce and declare
The Imminent Rise of the Metaprogrammer.

You have guessed right: a metaprogrammer is someone who develops metaprogramming code (metacode), i.e. code that creates and/or modifies programs (included, perhaps, itself).

Metaprogramming is by no means a new game: folks in functional programming (remember LISP?) have been writing metacode for many years. However, it is only quite recently that metaprogramming paradigms have percolated down to the world of mainstream programming (I am not counting in compilers, debuggers, etc.). Languages such as Python and Ruby, and to a lesser extent the new Java with annotations, have built-in metaprogramming capabilities.

Now, what kind of metacode can one write? The possibilities are, of course, endless. Nevertheless, for the sake of simplicity, we can classify metaprograms into two main categories, static and dynamic (for reference, check this synthetic presentation by Aussman and Kessler), somewhat mirroring the distinction compile time versus run-time: in the static category a metaprogram creates, edits. inspects some code, whereas in the dynamic model the same happens while the underlying program runs.

Rails is a successful example of static metaprogramming: it generates a template web project out of thin air (incidentally, I note in passing that there is a Java version of Rails, known as Grails. I shall talk on my experience with Grails in a future posting). There is no question in my mind that static metaprogramming will make further strides in the very near future.

However, my interest and hopes lean more toward the second category. I dream of a new generation of enterprise programs that modify themselves at run-time, to accommodate the ever-changing, often erratic business needs. Those programs will use both pillars of reflection, introspection, to see themselves, and introcession, to adapt and evolve.

In spite of the promises, and the fact that languages like Ruby amply provide the plumbing for sophisticated dynamic metaprogramming, so far (at least to my knowledge) we have only whetted our toes.

So, what do we need to get there? My gut feeling is that, aside the obvious step of educating mainstream programmers to the benefits (and potential marvels) of metaprogramming, two more things are necessary:
  • a single rock-solid example of applied dynamic metaprogramming

  • tools and environments to ease the work of prospective metaprogrammers

Some brave visionary soul must rise up to the challenge: any paladin out there to pick up the glove?


PS Conjecture: the two bullets above will pop up together in a single breath-taking sweep

3 comments:

Mimmo Cosenza said...

Hi Polymath, as you probably remember 20 years ago I've been a Lisp metaprogrammer (list, defmacro, quote and backquote were my best friends). I agree with your prediction about metaprogrammer and I'm very interested in your experience with grails. Last month I dedicated a sabbatical week in aquiring knowledge in RoR (I really don't like python) and it was the first time in the last 20 years I saw I programming language with an expression power closer to lisp (i.e. clos) than others. I still think lisp is superior, but the net it's putting a lot of efforts on RoR (and VCs too). If you are interested in a very good paper on this topic, here is a good link: www.paulgraham.com/icad.html.

I'll be waiting for your report on grails.

Mimmo

Polymathicus said...

Hi Mimmo, I certainly do remember your past life as a working metaprogrammer! LISP and other functional languages (such as HASKELL) are indeed superior as far as flexibility goes, but the fact is, we do not live (alas!) on Vulcan (I visited that great planet a couple of times, and found out that they teach untyped lambda calculus in middle high school, whereas in college they use a language based on categorical archetypes known as Archeus. More on that in a later posting).

I am no fun of Python either(I simply cannot stand the idea of using indentation as braces), although it has a great community of supporters & a lot of good scientific libraries.

Ruby? It rocks. But, till someone shows me how to use its mighty power for some true dynamic metaprogramming, I''ll stick
to my cup of old fashion Java.

Best
Polymathicus

PS1 The report on Grails will show up in less than a month. Meanwhile, do your company a favor: ask one of your engineers to download it and play a bit: a little investment with some possible return.

PS2 Thanx for the ref: really cool stuff

Anonymous said...

I am in data archiving for the long term. Is data safe in an archive? Maybe not very safe. A digital object should not rely passively on its environment to stay safe. A digital object with surviving capabilities should (i) protect itself and (ii) actively look for places where it is safe (iii) update its search strategies in changing environments. Now here is scope for metaprogramming. And then one step further: imagine this surviving digital object is not just some data, but it is a collection that makes up your digital self: the photos, music, films, documents, databases that make up your footprint on the internet. I think the notion of survival combined with digital self is enough incentive to come up with a metaprogrammatic killer app.