David's Computer Stuff Journal

I've been pondering this subject for a while, and I think I'm finally ready to write about it.

Tcl was, for many years, my go-to language of choice.  It's still near and dear to me in many ways, and I even worked on a portion of a book about it ( http://journal.dedasys.com/2009/09/15/tcl-and-the-tk-toolkit-2nd-edition ).

However, examining what "went wrong" is quite interesting, if one attempts, as much as possible, a dispassionate, analytical approach that aims to gain knowledge, rather than assign blame or paper over real defects with a rose-colored vision of things.  It has made me consider, and learn, about a variety of aspects of the software industry, such as economics and marketing, that I had not previously been interested in. Indeed, my thesis is that Tcl and Tk's problems primarily stem from economic and marketing (human) factors, rather than any serious defects with the technology itself.

Before we go further, I want to say that Tcl is not "dying".  It is still a very widely used language, with a lot of code in production, and, importantly, a healthy, diverse, and highly talented core team that is dedicated to maintaining and improving the code.  That said, since its "heyday" in the late 90ies, it has not ... "thrived", I guess we can say.  I would also like to state that "hindsight is 20-20" - it's easy to criticize after the fact, and not nearly so easy to do the right thing in the right moment.  This was one reason why I was reticent to write this article.  Let me repeat that I am writing it not out of malice or frustration (I went through a "frustrated" phase, but that's in the past), but because at this point I think it's a genuinely interesting "case history" of a the rise and gentle decline of a widely used software system, and that there is a lot to be learned.

At the height of its popularity, Tcl was up there with Perl, which was the scripting language in those days.  Perl, Tcl, and Python were often mentioned together.  Ruby existed, but was virtually unknown outside of Japan.  PHP was on the rise, but still hadn't really come into its own.  Lua hadn't really carved out a niche for itself yet, either.  Tcl is no longer one of the popular languages, these days, so to say it hasn't had problems is to bury one's head in the sand: it has fallen in terms of popularity.

To examine what went wrong, we should probably start off with what went right:

• Tk.  This was probably the biggest draw.  Easy, cross platform GUI's were, and are, a huge reason for the interest in Tcl.  Tk is actually a separate bit of code, but since many of the widgets are scripted in Tcl, the two are joined at the hip.  Still, though, Tk is compelling enough that it's utilized as the default GUI library for half a dozen other languages.
• A simple, powerful language.  Tcl is easy to understand and get started with.  It borrows from many languages, but is not an esoteric creation from the CS department that is inaccessible to average programmers.
• Easily embeddable/extendable.  Remember that Tcl was created in the late 80ies, when computers were orders of magnitude less powerful than today.  This meant that fewer tasks could be accomplished via scripting languages, but a scripting language that let you write routines in C, or, conversely, let the main C program execute bits of script code from time to time was a very sensible idea.  Tcl still has one of the best, and most extensive C API's in the game.
• An event loop.  Lately, systems like Python's "twisted", and node.js have made event-driven programming popular again, but Tcl has had it for years.
• BSD license.  This meant that you could integrate Tcl in your proprietary code without worrying about the GPL or any other legal issues.

These features led to Tcl being widely used, from Cisco routers to advanced television graphics generation programs to the AOLserver web server, which was busy serving out large quantities of dynamic web pages when many of us were still fiddling around with comparatively slow and clunky CGI programs in Perl.  Note also that there have been a lot of cool things to have gone into Tcl in the mean time.  It has a lot of impressive features; many more than most people realize, and has had a lot of them for a while.  Check out http://www.tcl.tk to learn more about the "good stuff".  But that's not the point of this article...

There was a healthy, active community of developers producing lots of interesting add-ons for the language, and working on the language itself.  This culminated in its adoption by Sun Microsystems, which hired the language's creator, Dr. John Ousterhout, and a team of people, who added a lot of great features to the language.  Quoting from Ousterhout's history of Tcl page:

The additional resources provided by Sun allowed us to make major improvements to Tcl and Tk. Scott Stanton and Ray Johnson ported Tcl and Tk to Windows and the Macintosh, so that Tcl became an outstanding cross-platform development environment; Windows quickly came to be the most common platform. Jacob Levy and Scott Stanton overhauled the I/O system and added socket support, so that Tcl could easily be used for a variety of network applications. Brian Lewis built a bytecode compiler for Tcl scripts, which provided speedups of as much as a factor of 10x. Jacob Levy implemented Safe-Tcl, a powerful security model that allows untrusted scripts to be evaluated safely. Jacob Levy and Laurent Demailly built a Tcl plugin, so that Tcl scripts can be evaluated in a Web browser, and we created Jacl and TclBlend, which allow Tcl and Java to work closely together. We added many other smaller improvements, such as dynamic loading, namespaces, time and date support, binary I/O, additional file manipulation commands, and an improved font mechanism.

Unfortunately, after several years, Sun decided that they wanted to promote one and only one language.  And that language was Java.  So Ousterhout and many people from his team decamped to a startup that Ousterhout founded, called Scriptics, where the Tcl and Tk innovations continued:

In 1998, Scriptics made several patch releases for Tcl 8.0 to fix bugs and add small new features, such as a better support for the [incr Tcl] extension. In April 1999, Scriptics made its first major open source release, Tcl/Tk 8.1. This release added Unicode support (for internationalization), thread safety (for multi-threaded server applications), and an all-new regular expression package by Henry Spencer that included many new features as well as Unicode support.

However, as many a company based around open source was to find later, it's a tough space to be in.  Scriptics changed its name to Ajuba, and was eventually sold (at a healthy profit, apparently, making it a relative dot com success story, all in all) to Interwoven, for the "B2B" technology that Ajuba had developed.  Interwoven was not interested in Tcl, particularly, so, to create a system for the ongoing development and governance of the language, the "Tcl Core Team" was created.

This was something of a blow to Tcl, but certainly not fatal: Perl, Python, Ruby, PHP, Lua have all had some paid corporate support, but it has by no means been constant, or included large teams.

At the same time in the late 90ies, open source was really starting to take off in general.  Programmers were making all kinds of progress, and had begun to make Linux into what is today the world's most widely used server platform, and laying the groundwork for the KDE and Gnome desktops.  While these may still not be widely used, they are for the most part very polished systems, and leaps and bounds better than what passed for the 'Unix desktop' experience in the 90ies.

One of the key bits of work that was added to Tk was to make it look pretty good on Microsoft Windows systems.  This was in an time when the "enterprisey" folks were turning away from Unix in the form of AIX, Solaris, HPUX, et al. and taking up NT as the platform of choice, so it was in some ways sensible to make Tk work well there, and in any case as a cross platform GUI toolkit, it ought to work well there in any case.

And, on the Unix side, Tk emulated the expensive, professional Motif look and feel that serious Unix programmers used.  What could go wrong?

As Gnome and KDE continued to mature, though, what would become one of Tk's major (in my opinion) marketing blunders took root.  I have it on good authority, from someone who was there in the office, that the Scriptics guys working on Tcl and Tk viewed Gnome and KDE (and the Gtk and Qt toolkits) as not really worth their while.  To be fair, since Tk has always been under a liberal BSD style license, the Qt toolkit has always been "off limits".  Still, though, the attitude was that Tk was a standalone system, and since it ran on pretty much any Unix system, it didn't need to bother with Gnome or KDE.  Gradually, though, as more and more people used Gnome and KDE exclusively on Linux, the Tk look and feel began to look more and more antiquated, a relic from the 1990ies when Motif (which has since pretty much disappeared) was king.  Tk applications started to really stand out by not looking at all like the rest of the operating system.  And, while Linux may not be responsible for a vast portion of the world's desktops, it is widely used by developers, who were turned off by the increasingly creaky look of the Tk applications they saw.

Tk is and was actually a fairly flexible system, and it would have been possible to tweak the look and feel to make it look a bit better on Linux, without even doing any major work.  Maybe not perfect, but certainly better looking.  Nothing happened, though.

Another problem was that Tk and Tcl made it so easy to create GUIs that anyone could, and did, despite, in many cases, a complete lack of design skills.  You can't particularly blame the tools for how they're used, but there was certainly a cultural problem: if you read most of the early Tcl and Tk books, and even many of the modern ones, there are hundreds of pages dedicated to exactly how to use Tk, but few to none explaining even basic user interface concepts, or even admonitions to the user to seek out that knowledge prior to attempting a serious GUI program.

The end result is that a lot of Tk programs, besides just looking "old fashioned" had fairly poor user interfaces because they were made by programmers who did not have a UI/design culture.

Contrast that with Gnome and KDE, which have made a point of focusing on how to make a good GUI for their systems, complete with guidelines about how applications should behave.  It may have taken them some time to get things right, but they have done a lot to try and instill a culture of creating high quality, well-integrated GUI's that are consistent with the system where they run.

Lately, there has been a lot of work to update the Tk look and feel, and it has finally started to bear fruit.  However, in terms of marketing, the damage has already been done: the image of "old, crufty Tk" has been firmly planted in countless developers' minds, and no amount of facts are going to displace it in the near future.

Another problem Tcl faced, as it grew, was the tug-of-war between those wishing to see it small, light, and easy to distribute embedded within some other program, and those wishing it to become a "full-fledged" programming language, with lots of tools for solving every day programs.  Unfortunately, that tug of war seems to have left it somewhere in the middle.  Lua is probably more popular these days as an embedded language, because it is very small, and very fast, and doesn't have as much "baggage" as Tcl.  Meaning, of course, that it doesn't do as much as Tcl either, but for a system where one merely wishes to embed a scripting language, without much 'extra stuff', Tcl's extra functionality is perhaps a burden rather than a bonus.  On the other hand, while Perl was happily chugging along with their CPAN system for distributing code, giving users easy access to a huge array of add-on functionality, and Python was building up a "batteries included" distribution, that included a lot of very useful software straight out of the box.  Tcl, on the other hand, chose to keep the core distribution smallish, and only lately has got some semblance of a packaging and distribution system, that is, however, run by ActiveState, and is (at least according to a cursory glance at the Tcl'ers wiki), not even fully open source.  The lack of a good distribution mechanism, and, in the meantime, eschewing a larger, batteries-included main distribution left Tcl users with a language that, out of the box, did significantly less than the competition.  Technically, a Python style "big" distribution would not have been all that difficult, so once again, I think this is a marketing problem: a failure of the Tcl Core Team to observe the "market", assess what users needed, and act on it in a timely manner.

Somewhat related to the large Tcl vs small Tcl issue was one particular extension, or extensions to the language that was noticeably absent: a system for writing "object oriented" code.  Tcl, at heart, will never be an OO language through and through, like Ruby or Smalltalk, but that doesn't mean that an OO system for it is not a useful way of organizing larger Tcl systems.  Indeed, Tcl's syntax is flexible enough that it's possible to write an OO system in Tcl itself, or, optimizing for speed, utilizing the extensive C API in order to create new commands.  Over the years, a number of such systems have arisen, the most well-known being "Incr Tcl" (a play on the incr command, which is akin to += 1 in languages like C).  However, none of these extensions was ever included with the standard Tcl distribution or somehow "blessed" as the official OO system for Tcl.  This meant that a newcomer to Tcl wishing to organize their code according to OO principles had to pick a system to use from several competing options.  And of course, newcomers are the least able to judge a complex feature like that in a language, making it a doubly stressful choice.  Furthermore, even experienced Tcl programmers who wanted to share their code could not utilize an OO system if they wanted their code to work with just standard Tcl.  Also, if their code had a dependancy on some OO system, it would require the user to download not only the extension in question, but the OO system it was built on, which, naturally, might conflict with whatever OO system the user had already selected!  As of Tcl 8.6, thanks to the work of Donal Fellows, Tcl is finally integrating the building blocks of an OO system in the core itself, but this effort is probably something like 10 years too late.

Some other more or less minor things that have taken too long to get integrated in the Tcl or Tk distributions include the PNG format (still not there in a production release of Tcl), a readline-based command line (Tkcon is nice, but not a replacement for simply being able to type "tclsh" and get a nice, functional shell like Python, Ruby and most other languages have.  This could easily lead to a bad first experience for someone trying out Tcl).  Tcl also took too long to integrate a first-class hash type (accessed with the 'dict' command), which only appeared in 8.5.  Its "arrays" aren't bad, but don't quite have the full power of a hash table as dict implements them.  Once again, the code to do these things was/is out there, it has just been a matter of integrating it into Tcl and Tk, which has been a slow process.

One actual technical problem that Tcl faces is the concept that all values must be representable as a string.  This is more or less ok for things like lists, hash tables or numbers, but is problematic when the user wishes to represent a value that simply isn't a string.  A basic example is a file handle, which at the C API level is a FILE*.  How does Tcl get around this?  It keeps an internal hash table with a Tcl-script accessible string, such as "file5" that points to a FILE * value that is used internally by file commands.  This works pretty well, but there is a big "but": since you can compose a string like "file5" at any time, that must be able to access the actual file pointer, you can't do any sort of "garbage collection" to determine when to clean things up automatically.  Other languages have explicit references to resources, so the program "knows" when a resource is no longer referenced by the rest of the program, and can clean it up.  Therefore, the programmer must explicity free any resources referenced this way.  This explanation is simplifying things somewhat, but it is something I view as a technical problem with Tcl.

If you've been following along, you've noticed a lot of "these days" and "recently".  That's because Tcl is still very actively developed, with a lot of new ideas going into it.  However, if you look at the release dates, it seems that after Ajuba was sold off, and Ousterhout pretty much abandoned an active role in the language for good, placing it in the hands of the Tcl Core Team, there was a lull in the momentum, with Tcl 8.5 taking 5 years to be released.

This is actually, in my opinion, an interesting phenomenon in languages: you risk hitting some kind of local maximum when your language is popular enough to have a lot of users who will be angry if things are changed or accidentally broken in the course of big upheavals.  So you have to slow down, go carefully, and not rock the boat too much.  On the other hand, there is an opportunity cost in that newer languages with less to lose can race ahead of you, adding all kinds of cool and handy new things, or simply fix and remove "broken" features.  Erlang is another system that has, in my opinion, suffered from this problem to some degree, but this article is long enough already!  Once again, though, not really a technical issue, but a problem with how the code was managed (and not an easy one to solve, at that).

A Tcl failure that I was personally involved with was the web infrastructure.  What went on to become Apache Rivet was one of the first open source Apache/Tcl projects, and was actually quite a nice system: it was significantly faster than PHP, and of course made use of Tcl, which at the time had a large library of existing code, and could be easily repurposed for projects outside the web (or the other way around: non-web code could be brought in to a web-based project).  One thing I ought to have done differently with the Apache Rivet project was listen to the wise advice of Damon Courtney, my "partner in crime" on the project, who wanted to see Apache Rivet have a fairly "fat" distribution with lots of useful goodies.  Rails and Django, these days, have shown that that's a sensible approach, rather than relying on lots of little extensions that the user has to go around and collect.  The code was out there, I should have helped make Rivet do more "out of the box".

A problem that is and it isn't: the syntax.  Tcl's syntax is amazingly flexible.  Since everything is a command, you can write new commands in Tcl itself - and that goes for control structures, too!  For instance, Tcl has a "while" command, but no "do ... while".  It's very easy to implement that in Tcl itself.  You simply can't do that in most "everyday" languages.  However, this comes at something of a "cost".  The syntax, for your average programmer who doesn't want to go too far out of their comfort zone, is perhaps a little bit further afield from the C family of languages than they would prefer.  Still though, a "human" problem, rather than a technical one.  Perhaps, sadly, the message is that you'd better not "scare" people when introducing a new language, by showing people something that doesn't look at least a little bit familiar.

Conclusions?  First and foremost, that, gradually, Tcl and Tk continue to be improved.  However, if one visits developer forums, there are also a lot of negatives associated with the Tcl and Tk "brands", and I am not sure if it will be possible to rectify that.   So what can we learn from the rise and subsequent "stagnation" of Tcl?

• Success in the first place came from doing one particular thing very well, and making it a lot easier than other existing systems at the time.  That's a good strategy.
• Not staying up to date can be deadly.  Of course it can be tricky to know what's a genuine trend, and what's just a fad, in this industry, and picking the right things to stay up to date with can be tricky.  That said, there are a number of areas where Tcl and Tk failed to follow what became very clear directions until way too late.
• Do your best not to get trapped between existing users who don't want to rock the boat, and don't lose your agility and ability to iterate with the system you're developing.  Long delays between releases can be deadly.
• Don't lose touch with your "roots".  In this case, the open source community that is a "breeding ground" for new developers and projects.  Tcl and Tk became passe` in that environment, which has led to its lack of adoption for new projects not only in the world of open source, but in businesses as well.
• Don't isolate yourself: Tcl and Tk stopped appearing at a lot of the open source conferences and events and in magazines/books/articles online, either because with no figurehead/leader to invite there was less interest in speakers and authors, or because the rest of the Tcl Core Team wasn't particularly engaged, or for whatever other reason.  This created something of a negative feedback loop where Tcl and Tk were things associated with the past, rather than something currently talked about and discussed.