Lisp's IDE problem - and a proposed solution

Update on 07/06/2008 Edited the paragraph on LispWorks and Allegro to clarify that my criticism is directed at the limitations of the free editions, not the full versions

Lisp is an amazing programming language, with no other single language that I know of combining all of its capabilities. It is a surprise, then, that Lisp isn't as widespread and successful as expected of a language of such power. Many have written about the reasons for the language's lack of adoption, citing a lot of explanations from lack of libraries or parenthesis-heavy syntax to the attitudes of some Lisp advocates.

It is ironic how so many geniuses miss the point!

Here's my (highly personal, insufficiently researched) opinion as to the main reason of this lack of adoption, and a proposed solution to this problem.

While the aforementioned factors certainly contribute to Lisp's lack of adoption, I think they are all solvable problems. Lisp's syntax is unfamiliar but actually easy to learn. Lack of libraries becomes an obstacle later in the Lisp adoption cycle so while they are indeed a problem they are not currently the most significant problem, in my opinion.

As for the community issue: I don't have a lot of experience with the Lisp community but it's hard to believe they're the main obstacle to Lisp's adoption. Linux, for example, survived many years of zealous fans who answer any question by blaming new users for not reading the manual or trying to convince them they don't need any feature that Linux doesn't provide. This did slow down the adoption of Linux but it didn't stop it from having an increasingly growing user base.

It's the IDE!

Lisp needs a good IDE. Almost all Lisp users suggest using Emacs+SLIME for this purpose. That might do the job for a seasoned Lisper, but these tools get in the way of converting new users.

I know Emacs is really, really powerful. I know that SLIME is a very innovative tool that has the potential to lead the way to how all future IDEs are designed. Still, convincing a programmer to use Emacs is a totally separate task from convincing him to use Lisp. Both are not easy tasks and forcing them together doubles the burden of the Lisp advocate and provides a barrier to entry that would chop off a significant portion of potential future Lisp programmers.

Commercial tools

If you don't want to use Emacs, you have commercial IDEs like Allegro and LispWorks. Another barrier to entry: these tools cost too much. The cheapest edition of LispWorks costs $1500 and Allegro costs $599 for academic use (I tried to find the cost of a commercial Allegro license but didn't find a quote on their pricing website).

Both have free starter editions but they don't solve the problem.

I tried the free editions of both of them in mid-2007 while giving a series of Lisp teaching sessions in my faculty. Allegro was a frustration because installing the 'free express edition' required online activation and it wasn't practical to distribute it to students to put on their computers on the first session day since we didn't have a reliable internet connection in the session hall.
I abandoned Allegro and used LispWorks. It's actually a good IDE. Not good by Microsoft or Borland standards, but good in the sense that if you ignore the outdated interface and the lack of features in the personal edition, you could actually write and demonstrate Lisp programs with it. It wasn't frustration free, though.

In the end, while the students liked Lisp as a language and understood its power, many of them didn't want to write Lisp code themselves. I got the feeling that they think the IDE would get in the way and that they would have a bad experience trying to get the programs to run. To encourage them to work in Lisp projects I threw away all the commercial tools and hacked together a toy IDE with C# combining bits from Cusp, SBCL, IKVM and Scintilla.

The tools disappointed me. I don't think that Franz or LispWorks are bad companies or that their tools are inadequate. Their IDE's are very advanced tools on the forefront of compiler and language processing technology. The full versions come complete with libraries for GUI design, web servers, databases and so on. They probably pay for themselves multi-fold and they have lots of customers to prove it.

My problem isn't with the general quality of the tools, but in the fact that they cost too much and that the freeware personal editions are intentionally crippled and give the wrong image to new users. This means that those commercial tools fail to remove the 'lack of free/cheap IDE' as a barrier to entry for programmers who want to use Lisp.

Why did these companies surround the free editions of their tools with the software equivalent of barbed wire? Are they afraid of people actually being interested in their products?

Compare that to Microsoft's freeware Visual Studio Express: It's a very high quality piece of software. It's licensed for both commercial and non-commercial use and they promote the life out of it. Yet it doesn't seem to cannibalize on the sales of the many professional editions sold by MS.

Approaching a solution

One of the best approaches to creating a Lisp IDE is Cusp, the Eclipse plugin for Lisp. It provides an editor with color syntax highlighting and function parameter tips, a debugger, and project management capabilities. More importantly it installs with a few clicks using Eclipse's auto update facility. It's based on SBCL and the Swank part of SLIME so it doesn't sacrifice power either.

Still, it doesn't go far enough in my opinion. For a start, it's heavy since it requires the Eclipse platform to run. Also, creating a "hello world" project with it isn't trivial. The new project wizard creates a package and an assortment of files with no clear indicator of where to begin typing your code. Finally, when you outgrow the REPL and the simple examples you still have to manually install libraries using ASDF (and suffer for it).

The solution

We need a good, intuitive and capable Lisp IDE that is on the same level with professional IDEs like Delphi or Visual Studio. It doesn't need to have the same power as these tools right away, but could initially be simpler and lighter. Still its developers need to have the strategic goal of eventually competing with the aforementioned tools in both power and usability. Here's my personal sketch of such a tool, in detail:

* It needs to be open source or at least freeware. Basing it on something like SBCL wouldn't hurt, either.

* It needs to adopt traditional IDE sensibilities. Users should be able to copy and paste with the familiar Windows/Macintosh keyboard shortcuts. It should have a visual debugger with breakpoints and watches. It should have a REPL but not force developers' workflow through it and instead offer a traditional edit/run cycle. A tool like DrScheme is a good starting point for such a design.

* It needs to have a simple and authoritative name. I suggest OpenLisp, LispDevelop or Lisp Kit. (LispDevelop was the name for the toy IDE I created and which I intend to develop further, but I would happily drop the name from my program if someone wants to use it and I like the direction of his or her IDE).

* It needs to be readily installed on any platform simply by extracting files from an archive. It must avoid all need for compilation, pulling dependencies or using system definition tools like ASDF or cl-build. Installation should include a variety of useful libraries, possibly cherry-picked from existing open source projects.

* Autoupdating libraries should not be a concern for the normal user: upgrading libraries would be done as part of releasing new versions of the IDE itself. These libraries would be tested, packaged and supported by the person/organization that's developing the IDE.

If a user likes to use the bleeding edge version of some favorite library, it's his or her own responsibility to install it; Normal users should not be forced to follow that path. Library developers whose libraries are included as standard with the IDE would need to make their libraries as complete and stable as possible, and not rely on users regularly updating to the latest version to resolve bugs.

If a library is half complete, it should not be included. If a library is half complete but really cool and worthy of inclusion, the entity developing the IDE would offer a nice, big donation to the library developer if he releases a finished version of the library within a specified deadline, complete with documentation, test cases, and easy integration with the IDE.

* The IDE should come with an excellent GUI library and (possibly in a later version) a visual GUI designer. It could be developed from scratch or be based on an existing library. It should showcase Lisp's declarative power and not intimidate the user.

A lot of Lisp GUI libraries like to force the programmer to create lots of abstract objects just to make a simple program and this hides the real power of Lisp. It's certainly a good thing if the GUI toolkit could include layout managers and other powerful abstractions, but the programmer doesn't have to be forced to use them. Instead, something like that should be perfectly doable by the library:

;;
(setf form1
(Form :text "Test Form"
(Button
:x 100 :y 100
:text "Click Me"
:onclick (lambda (ev) (ShowMsg "Hello World")))))

(Show form1)
;;

* The people developing the new IDE have to be persistent! The web is full of half done attempts to "save" Lisp and very few of them worked out.
While I think the project would be best served by being open source and free, if lack of money prevents people from working on the project a business plan could be formed to bring money from the IDE (while not commercializing things too much; such a company should be similar in spirit to Redhat or Mozilla Corp).

* Company or no company, in no way should the IDE be designed by committee. It needs standardization, responsiveness and decisions made based on utility and not consensus. It wouldn't hurt to have a benevolent dictator who's a good leader and has a strong sense of design.

* Usability doesn't mean weak or dumbed down tools. To succeed such an IDE needs to appeal to both beginners and power users and to be suitable for real world projects. Also it should be customizable (yes, even supporting emacs and vi keybindings). It should also be extensible, preferably with Common Lisp itself to avoid forcing users to learn a new language to customize their IDEs. If many users want Emacs Lisp support then perhaps we could go all the way and design a unified extension API to extend the IDE with any language including Emacs Lisp, JavaScript...etc.

This doesn't need to be done all at once, of course. A basic IDE with a focus on productivity would be the first priority, and after a few iterations the real power of the project would show up. We could take a hint from the Firefox or Gnome teams who focused on throwing away all the clutter and bloat from their codebases then built up power on top of a simple and lightweight foundation.

There is no contradiction between having an powerful IDE that's also easy. A large chunk of Visual Studio users don't know the real power of the tool they're using or how really modular, versatile or extensible it is. A lot of Eclipse users probably don't know that the whole Eclipse IDE is exposed as a set of objects for all plugins to see. They simply see Eclipse as good editor/compiler/refactoring tool, while the power users go crazy developing add-ins.

* Finally, the HyperSpec is outdated and looks ugly. From what I understand, there are copyright restrictions that prevent people from freely modifying it and publishing their modifications. An open Lisp documentation project needs to be created and released either in the public domain or with a sufficiently open license. This documentation needs a standard format and a way for plugging additional documentation for libraries...etc, with the ability to index or manipulate the aggregated documentation as a whole with any external tool, including of course our proposed IDE.

If such an IDE were developed, I believe it would be a large step forward in making Lisp more attractive to new developers. It might not make Lisp replace (say) Java or C# overnight and there are still numerous unsolved problems in the language itself, the libraries and so on but a powerful IDE would, in my opinion, give Lisp a fighting chance alongside the Pythons and Rubies of the development world.