Posted on Wed Dec 23 00:00:00 -0500 2009

Introducing the IokeLexer

It seems I just can’t stop writing syntax highlighters for Ioke. I started with adding support for TextMate, and then basic support for Geshi. Now, as a present for its P release¹ I’ve added Pygments to the list:

Pygments IokeLexer

One of the things that has bugged me since Ioke’s birth is that its incubator, GitHub has not been able to do the honourable thing and highlight it appropriately. Instead, one of the most exciting languages around at the moment has been treated like dirty plain text.

Well, hopefully this can change soon. GitHub happens to use Pygments to highlight its source. Therefore I just need to throw my IokeLexer at the Pocoo chaps, hope they include it in the next version of Pygments and then wait for GitHub to upgrade. At least someone doing the hard work of writing the lexer won’t be the blocker any more…

Just to show its glory², here’s the introduction example from http://ioke.org

#!/usr/bin/ioke

Ioke = LanguageExperiment with(
  goal: :expressiveness,
  data: as(code),
  code: as(data),
  features: [
    :dynamic,
    :object_oriented,
    :prototype_based,
    :homoiconic,
    :macros
  ],
  runtimes:(JVM, CLR),
  inspirations: set(Io, Smalltalk, Ruby, Lisp)
)

hello = method("Every example needs a hello world!",
  name,
  "hello, #{name}!" println)

Ioke inspirations select(
  features include?(:object_oriented)
) each(x, hello(x name))

Lessons Learned

OK, so apart from a fairly high number of hours battling with Python regular expressions and Ioke syntax, here’s a summary of the lessons I learned from this experience:

• You can get pretty far with plain regexps.
• You can get a long way with regexps and stackable contexts.
• Writing approximately correct lexers with regexps can be fun.
• Writing accurate lexers with regexps is for masochists, seriously.
• Look ahead and behind regexp matchers are truly wonderful.
• Writing a regexp lexer really helps you to hone your regexp skills.
• Pygments is nice software.
• Python people can be a little curt yet are very helpful nevertheless.
• Nothing beats a real lexer for lexing.

Finally, there really should be a standard syntax for this kind of thing so when the next syntax highlighting engine comes around we don’t have to continue to re-invent the wheel.