Erlang and unreliable resources

Here’s something you should be aware of if you’re using Erlang in a large system.  It’s a pity the core team hasn’t included something like it in the Erlang distribution itself, or event documented; because it’s something I think you’ll eventually need if you build a big enough system.

In a large system, you’ll have components that should be up and running – but at some point might not be.  For instance, a web site with a database.  If you write your Erlang code to just “let it crash” because the web server can’t connect to the database – just like you read in the books – then your whole site will fall over, web server and all.  At this point, you can’t even show your visitors a message explaining that “we’re experiencing problems, please try again later”, because the problem has propagated throughout the system, bringing it all down.  Another scenario might be a machine with a user interface and some fragile hardware.  The machine can’t do its job without the hardware, but if the hardware is broken, the software should not crash completely!  The user interface needs to stay active, letting users know that the machine is broken, and perhaps offering some diagnostics or offering some steps to try and correct the problem.  What these have in common is that there are some components of the system that should always try to be available even if they are not 100% functional, and there are other bits and pieces that may be necessary for the correct functionality of the system, but should not cause it to become entirely unavailable when something goes wrong.

The term for this is “circuit breaker”, because it breaks the chain that is part of a normal OTP system, where enough failures of a worker lead to a supervisor crashing, and its supervisors crashing in turn, after enough failures, and so on up the chain.  One high quality implementation is located here: https://github.com/jlouis/fuse – and politely points to some other, similar implementations.  I’m a bit frustrated that I found this only recently, because it’s a nice bit of code that strikes me as quite likely to be used somewhere withing any sufficiently large Erlang system.

As a footnote, I also created something similar, although it’s not battle tested (I just released it, actually!) and operates at a different level: https://github.com/davidw/hardcore – but it might be useful for some people.

A Minor Erlang Rant

In an earlier post, I compared node.js to Erlang: https://journal.dedasys.com/2010/04/29/erlang-vs-node-js – which admittedly has a whiff of apples and oranges about it.  Still, though, I think there's something to it.  Node.js is creating lots of buzz for itself these days.   Some of that will turn into actual production use, and at that point actual companies will have a vested interest in seeing the system improved.  Currently, it is not as 'good' as Erlang.  Erlang's VM has a built in scheduler, so you simply don't have to worry about manually creating bite-sized chunks of processing that feed into one another, it all "just works".  For instance, my current Erlang project involves reading from twitter's stream, and distributing that to IRC and the web.  It's pretty simple, works well, and is quite robust.

I haven't had the best of days though, and one little annoyance of the many I dealt with today really caught my eye.  I need to do an HTTP POST in Erlang, and:

  1. The documentation does not have examples.
  2. Here's an example of how to do a POST:
    http:request( post, { "http://scream-it.com/win/tests/posttest.cfm", [], "application/x-www-form-urlencoded", "x=2&y=3&msg=Hello%20World" }, [], [] ).

  3. Aside from being ugly and not very leggible, you'll note that he's passing the parameters as a string, and also has to manually include the "x-www-form-urlencoded" header.

  4. To create that string, you'd want to url encode it.  Actually, ideally, you'd just pass in a list of key/value pairs and let the library module handle it.

  5. However, there's nothing in the http module that does that.

  6. If you look through various bits of Erlang code on the web, you'll note many, many versions of url encoding and decoding, because the http module makes no mention of how one might go about doing so.

  7. It turns out, that the edoc_lib module does have a uri encode function!

  8. That isn't documented in its own man page.

  9. And certainly isn't linked to in the http page.

So, in 2010, doing an HTTP POST in Erlang is still a pain in the neck.  I'd be happy to put my money where my mouth is and submit a patch (at least one for the documentation), but you have to wonder why no one has fixed the problem so far – maybe they're not very accepting of patches?

Erlang: Log to Files

Since I had a heck of a time finding out how to do something that ought to be really easy, I thought I'd post what I found.  If you want Erlang to write errors to a file instead of stdout, add this to the command line:

-kernel error_logger '{file, "/tmp/erl.log"}'

Maybe there's a better solution, in which case feel free to correct me.

Mochiweb proto-deb and hacking

I had some time over the weekend to play a bit with Mochiweb.  It's a very nice little bit of work, but kind of strange for an open source project.  I think both things come from the fact that it was built to do some very specific things in a business, and once those goals were accomplished, that suited the owner just fine.  There is no nice/fancy project home page, no links to documentation, and in general, all of the stuff you're supposed to do for a successful open source project.   The code, however, is good, fast, and pretty easy to deal with.

Anyway, I wanted to use it for a thing or two I'm working on, so I created some .deb packaging for it.  It's been ages since I've done that (indeed, I officially retired from Debian), so it's probably missing something, but I thought I'd throw it out there for people to have a look.

The git repository is here: http://github.com/davidw/mochiweb – my changes live in the 'deb' branch.

The big thing I've tried to do is make it so that if you have several mochiweb projects on the same machine, they'll all be started/stopped by the init scripts, if you so choose.  What you do is add a file with the path to the application to the /etc/mochiweb/apps directory, and when you do /etc/init.d/mochiweb start, it will start all of those.  At stop time, it will run the stop.sh script, which shuts things down.  It runs things as the user that owns the start.sh script, unless it's root, in which case it simply doesn't run.

The thing that's missing right now is a way to get the thing to write logs somewhere, and I'm not quite sure how to do that, so here's hoping someone will write in with a suggestion.

The .deb file I generated ought to be attached to this post, although there are no guarantees that I'll keep it up to date.

Erlang vs node.js

I've written about Erlang in the past, and my suspicion that, soon or later, other languages/systems will come along and "eat its lunch".  Scala is one such potential contender.  Another that has been gaining some visiblity lately is node.js, a simple framework for creating networked applications on top of Google's V8 Javascript engine.

I should define what Erlang's lunch is a little bit better before we go on.  Erlang does several things really, really well; better than many existing, mainstream systems:

  1. Concurrency – the Actor model that it uses is much easier and more pleasant than dealing with threads.
  2. Distributed systems – Erlang makes this fairly easy and pleasant as well.
  3. Fault tolerant systems – using its distributed powers, Erlang is good for writing things like telephone switches that can't spend any significant time down.

Of these, I think the big one is concurrency.  Distributed systems are nice, but not a critical point for most people.  Same goes with fault-tollerant: those who need it really need it, but the rest of us are willing to make some compromises.  Our web applications are important, but generally not something where people's lives hang in the balance.

How does Erlang do "concurrency"?  The developer creates Erlang "processes" that interact with one another by passing messages.  These are not real, OS level processes, though, and this is critical to how Erlang operates.  Since these processes must all coexist within one real, system level process, it is absolutely essential that no operation that they perform hangs the entire system!  The Erlang runtime system is built around this concept.  Any Erlang process can do things like read and write to the disk or network, or have a "while (true) { …. }" loop (it doesn't actually look quite like that in Erlang, but that's the idea), and it won't wedge the system.  This knowledge is also critical when you want to interface Erlang to the outside world: if your C library contains a routine that might block for a long time, you can't just call it from Erlang as it won't be a well-behaved part of Erlang's world (there are ways around this of course, but make life a bit more complicated for the developer).  All this is done with Erlang's scheduler: each Erlang process gets a number of operations it can run before some other process gets to run, so even our while loop will only run for a bit before the system moves on to something else.   IO is rigorously done with non-blocking calls internally in order to keep that from becoming an issue.

No other system that I know of has such a focus on being non-blocking, and node.js is no exception: a while(true) loop is perfectly capable of wedging the system.  Node.js works by passing functions (closures, in many cases) around so that work can be performed as needs be.  However, the actual functions that run actually do block the system, and thus must be written in order to not run for too long.  Also, and this is important, Node.js also does its best to make IO non-blocking by default, so that you don't have to worry about IO calls.

Node.js isn't up to the level Erlang is at, because it requires more manual intervention and thinking about how to do things, but it's probably "good enough" for many tasks.  How often do you really write code with so many calculations that it slows things down?  Not often in my case – the real world problem I most often encounter is IO, and node.js does its best to make that non-blocking, so that it can be handled in the "background" or a bit at a time, without wedging the system.  And if you really needed to write a long-running calculation (say you want to stream the digits of PI or something), you can break up your calculation manually, which may not be quite as elegant as Erlang, but is "good enough" for many people.

"Good enough" concurrency, combined with a language that is at least an order of magnitude more popular than Erlang, and a fast runtime, combined with ease of use in general (it's way easier to get started with node.js than with most Erlang web stuff) make for a system that's likely to do fairly well in terms of diffusion and popularity, and is going to "eat some of Erlang's lunch".  Or perhaps, rather than actually taking users away from Erlang, it's likely to attract people that might have otherwise gone to Erlang.

The Erlang Paradox

There has been a nice series of interviews with the creators of various programming languages. In this one, it’s Joe Armstrong‘s turn. The creator of Erlang, he’s a very bright guy and well worth reading. One thing that has always struck me as interesting about Erlang, as compared to most of the “open source” style languages I’ve tended to use, is how it was incubated for a long time within Ericsson. It spent years there, and millions of lines of code were put into production use, prior to being open sourced, and certainly before the world at large took much notice of it.

This has been both good and bad for the language. The obvious upside is that a lot of smart people were paid to work on and improve it. Many new languages don’t really get that opportunity, unless they come out of an academic environment. Rather than being a spare time project, or something that slowly builds up speed over the years, Erlang, as far as the rest of the world was concerned, burst forth from Joe’s forehead, fully formed. This is not something like Ruby, where the early adapters were small companies like 37 signals, willing to live on the edge and not follow the pack. Quite the contrary, Erlang was backed by a huge, established player in the telecom market, focused particularly on applications where reliability is paramount. Certainly not a language you can call a ‘toy’!

The downside to all this, though, is that, esentially, they can’t really change the language much without being very, very careful. Here’s what Joe says about what he might have done differently:

Looking back, is there anything you would change in the language’s development?

Removing stuff turns out to be painfully difficult. It’s really easy to add features to a language, but almost impossibly difficult to remove things. In the early days we would happily add things to the language and remove them if they were a bad idea. Now removing things is almost impossible.

The main problem here is testing, we have systems with literally millions of lines of code and testing them takes a long time, so we can only make backwards compatible changes.

Some things we added to the language were with hindsight not so brilliant. I’d happily remove macros, include files, and the way we handle records. I’d also add mechanism to allow the language itself to evolve.

Which is an interesting contrast to more “traditional” open source languages like Ruby and Python, which, over the years, have from time to time bitten the bullet and unloaded stuff they don’t like. On the other hand, Erlang is solid code that you can count on if you do need to write systems with little to no downtime.

What do you think of the two approaches?

Language Geekery: Reia Programming Language

This looks kind of interesting:

http://wiki.reia-lang.org/wiki/Reia_Programming_Language

In short: it’s a Ruby/Python type of language on top of Erlang. It has a mishmash of Ruby and Python syntax, and also does objects, implemented as Erlang processes.

Funny quote:

Reia uses an indentation-sensitive syntax. This allows Reia to look similar to Erlang without relying on “ant turd” tokens (i.e. , ; and .) to structure relationships between forms.

That made me laugh, and is in all seriousness, something that I do find to be a bit annoying in Erlang, especially when refactoring code. You can’t just move a line of code from one place to another; you move it, then change the line endings in both places.

My preference would be to do something more Ruby style rather than Python style for the following reason: the indentation thing makes it harder to use straight-up Python for templating purposes, whereas Ruby can be used as-is. Templates are something that I wouldn’t particularly want to do in Erlang, either, so doing them with this language might be a very nice alternative.

Anyway, it looks to be in its infancy. Unfortunately, it seems to require a more recent version of Erlang than what ships with Ubuntu Hardy. Maybe Intrepid will have something recent enough to run it.

It’s a little known fact that I originally wrote Hecl in Erlang, although, sadly, I can’t find the code any more.

Potential sweet spot for Erlang web apps

I bumped into this while reading various things:

http://beebole.com/blog/2008/09/02/web-application-on-erlang/

And was interested, because it fits nicely with a half-formed idea of my own: Erlang is never going to be a nice language for doing HTML templates, but getting JSON into and out of it is pretty easy, so it could potentially work very well where your application is divided between Javascript for all the template/rendering/frontend duties, and Erlang on the server, with something like Mnesia, Couchdb or Postgres as the data store. That sort of application wasn’t really possible several years ago, but with the right target audience, you could probably get away with doing an app that requires Javascript to function these days.

Languages: worst case vs average

This is a pretty good take on Tim Bray vs Erlang. However, I’d like to focus on something said at the end:

The fact is, all languages optimize for particular problem domains and I don’t think that a universal programming language exists that would perform well on all tasks. Bray rapidly found out that Erlang isn’t optimized for doing line-oriented I/O and it’s regex library sucks. So what? While those problems could be eliminated from Erlang, the fact that Ericsson has deployed large telecom gear without having to fix those issues means that Erlang is ideally suited to its original programming domain.

The problem is that learning new languages is difficult for many people. They’re not going to bother starting from 0 with a new language if what they’ve got is only 10/20% worse than the other guys. For companies, developing expertise (and trying to maintain it) in lots of languages is probably less productive than picking a few (Google only has four: Java, Python, Javascript and C++, and they’re not exactly lacking in brainpower or cash). This is why “the right tool for the right job” works for things like saws and hammers, but not for complicated, idiomatic things like programming languages.

Therefore, it’s important that languages that aim to be widely used do lots of things pretty well, and try to avoid doing anything really badly. This is something I’ve always admired about Python, even if I don’t use it anymore. It’s a language that you can do a very wide variety of tasks with, and you can be sure there’s probably some code out there already, and that Python will be ok for your task, even if it’s not the best thing. Performance is a weak point compared to compiled languages, but even there, get the libraries right and things aren’t so bad.

In any case, my point isn’t to bash Erlang (which I like), or put Python on a pillar (I prefer Ruby and Tcl), but to say that a widely used programming language doesn’t need to be the best at everything, but it should aim to make people comfortable using it, safe in the knowledge that they can probably do most things ok if needs be. That’s a far better situation than being able to do one thing realy well, and perform poorly at many other tasks, forcing its users to look elsewhere for solutions to those tasks.

Personally, I think that for concurrency, Erlang is in a bit of a race right now – they’re pretty far ahead in terms of concurrency (and also fault tolerance, but fewer people care about that), but behind in other areas (less library code). Other languages don’t have as good a tools for concurrency, which isn’t an easy problem. So, what will happen first – Erlang catches up elsewhere, other languages get “good enough” concurrency solutions, or a new language (like Scala) comes along and takes the lead?

Erlang

As someone who tries to read a lot about my field, I feel I have my “ear to the ground”, so to speak. And right now, I’m picking up a lot of hype-rumblings about the Erlang programming language. I have a little bit of experience with it, having used it briefly about 3 years, and having made a few minor contributions.

Why are people interested in it? Picking up new programming languages is fun for some people, but in general, unless it’s something backed by millions of dollars of marketing and development (Java, C#/.Net) there needs to be a ‘killer feature’ to drive adoption. Tk and Tcl, ease of developing simple dynamic web content with PHP, Rails and Ruby, and so on. Erlang, in many people’s eyes:

  1. gets concurrency right, or at least is a big improvement over existing models (threads).

  2. Makes distributed computing easier and does it naturally.

  3. Is a solid base on which to construct reliable applications.

That appears to be enough to have nudged Erlang into a position where it has a chance at the big time.

While I would highly encourage people to learn a bit of Erlang, and to seriously consider its use for the type of systems for which it was created, I have some doubts about it becoming that widespread. Please read the following as the thoughts of a critical mind, and keep in mind that it’s far easier to zero in on defects than it is to create something from scratch, so do not misconstrue my writing as being “against” Erlang (I do use it, enjoy it, and recommend learning it!), but as an honest appraisal colored by my own view of the world, which might not be like yours.

To begin with the superficial, its syntax is a bit odd. For a professional programmer, that shouldn’t present much of an obstacle, and indeed isn’t after a few days of using it, but there are a few things that I do find a bit annoying despite being used to the syntax, first and foremost being the use of . and , to separate lines of code, rather than a newline, or ; (which is used elsewhere). They’re like little mouse droppings – hard to spot, and they get in your way when you swap out bits of code here and there. The last line of a function must end with a ., but other lines are separated with commas, so when you’re cutting and pasting lines from here to there, you end up having to recheck the line endings each time.

Erlang is a functional programming language. Once again, someone with a wide exposure to many programming styles will not find this difficult after a week or so of practice, to wrap your mind around a different way of thinking. However, I have some doubts as to whether a functional language will be able to bring on board the numbers necessary to be widely adopted (see Scaling Down).

One of the benefits of wider adoption is, indeed, having a wider set of libraries – it’s a “positive network externality” in terms of economics. Other people have written about the relative lack of Erlang libraries to do various things, but it’s actually not one of the things I’m worried about – it’s something that will take care of itself if the language grows in popularity. After being spoiled by Ruby’s gems, what I really miss, though, is a nice, simple, easy distribution mechanism.

Related to making it easier for J Q Programmer to adopt, the error messages are at times not the most helpful I’ve seen in the computing industry:

Error in process <0.29.0> with exit value:
{undef,[{shell_default,sfdsf,[]},{erl_eval,do_apply,5},{shell,exprs,6},{shell,eval_loop,3}]}

In English, that means that I tried to execute the function sfdsf(), which doesn’t exist. It seems that there ought to be a way to keep the nice, computer parseable error messages, and add in something that humans would find just a bit easier.

Other people have already covered the problems with Erlang’s strings, so I won’t belabor the point. Suffice it to say that, in my opinion, is_list([foo, bar, baz]) and is_list("hello") should probably not both be true.

Which brings us to another issue. Erlang is a language used in big, reliable systems like telephony switches. That’s not the sort of environment where you want to break things, so I doubt that Erlang will ever see big changes that introduce incompatibilities, like Python has recently done. This one is a tradeoff – stable, reliable and backwards compatible is simply not compatible with rapid development of the language – it’s a zero-sum game, you want more of one, you get less of the other.

Related, is the “open sourceness” of the language. It is open source, which for me is very nearly a prerequisite for dedicating my time to a language, but Ericsson appears to maintain firm control over it. This is another thing that’s both good and bad. Having the backing of a large corporation means that developers are paid to work on and enhance Erlang, which is beneficial to everyone. On the other hand, as some people say, open source is just the start, open development is important too. Random thought: I bet Erlang would make a nice Apache project, but I’m sure the bureaucracy involved would be a nightmare.

In terms of the language itself, beyond the syntax, it feels a little bit too verbose. Perhaps not as much as Java (that would be difficult, wouldn’t it)… maybe I’m just used to Ruby, which is about as concise as I’m comfortable with while still being readable. Sometimes, though, there is a little bit more ‘repeat yourself’ than I would like to see. For instance, in this implementation of string_join (something else that ought to be in the standard library, but still isn’t), it’s necessary to create the same function, with different arity, simply to work around the fact that it’s not possible to say the default separator should be ” “:

string_join(Items) ->
    string_join(Items, " ").
string_join(Items, Sep) ->
    lists:flatten(lists:reverse(string_join1(Items, Sep, []))).

Which is a lot more typing than string_join(Items, Sep=" "). This is pure opinion, and very subjective, but Erlang to me feels like it falls more in the “systems programming language” category than “scripting language”. It’s a bit more verbose, it needs to be compiled to run it, modules need to have an entry point… it’s not as suitable to quick throwaways as Ruby, Tcl or Python. Of course, it is certainly higher level than C, and leaner than Java, but still falls an the other side of an admittedly fuzzy line in my head. Its command line options reinforce that. Whereas “scripting” languages do the obvious thing first:

ruby foo/bar.rb

Erlang is a little bit more complex:

erl -noshell -pa foo/ -run bar

A minor detail, once you get used to it, but still, I prefer the “do the obvious thing first, and let the user have some say in the matter if they want to do something else” approach to systems design.

One of the wonderful things about Ruby on Rails, is that it uses Ruby throughout, templates included. Without getting into the merits of using templates vs other systems, my own opinion is that Rails gets them right, and demonstrates the flexibility of Ruby by using it as the language for the templates, rather than inventing some other solution. I don’t see the same flexibility in Erlang, which is unfortunate, as it blows Ruby out of the water in terms of being a very solid solution to deploy on a server (no fiddling with proxies and load balancers and single threaded mini-servers + hot code replacement = nice).

Of course, there are lots of good things about Erlang, as mentioned at the beginning of this article – concurrency and distribution, but also language features like pattern matching that are a lot of fun to use, a pleasant syntax for dealing with binary data, list comprehensions, first class functions, tail recursion, and lots of other goodies. However, if I had to try and predict the future, I could easily imagine another language grabbing some of Erlang’s good ideas and running with them, perhaps being more “conventional” in other respects – this isn’t uncommon in the languages field, as we’ve seen with things like Lisp and Smalltalk. On the other hand, one of the reasons I am quite happy with Erlang is that it is most certainly not a proof of concept – it is a very practical language that excels at solving certain classes of problem, and does so in many commercial settings. If you’re the kind of person who enjoys new things, check it out!