David's Computer Stuff Journal

Since hosting is the major expense for DedaSys LLC, and obviously a critical part of much of what I do, getting the right one is very important.

Naturally, "the right one" for me may not be the right one for everyone. I am a fair sysadmin for small numbers of machines, so I don't mind doing that myself - I don't need, nor want to pay for hand holding.

I recently moved everything over to Slicehost on the recommendations of friends, and am so far fairly happy with the experience. It's cheaper, simpler, and more flexible than Layered Tech was. To boot, I also like the fact that Slicehost is run out of St. Louis, Missouri, rather than some really expensive tech center like San Francisco or Boston. Hosting is basically a commodity, and much like you wouldn't want to put a factory in San Francisco, you don't want your hosting their either.

However, I discovered something annoying about Slicehost: they use x86_64 servers, which, per se, doesn't really matter to me - I use open source code that can run on any number of architectures. The problem is that this particular architecture uses more memory than plain old x86. Significantly more. My Layered Tech server with 1 gig of memory was hitting the swap space a bit, but the same code on a slice was swapping quite heavily, despite the fact that I'd moved PostgreSQL off to its own slice. Since I pretty much exclusively run Rails, I decided to look into Phusion's "Ruby Enterprise Edition", which is basically just some nice hacking on Ruby's garbage collection mechanism. What they've done is nice, and I may end up using it, but ultimately it's buying me space that I would also gain from simply moving back to x86. With that in mind, I decided to take another look at what appears to be emerging as one of Slicehost's main competitors, Linode, who *do* use x86 servers. Here are my results, which are admittedly not all that scientific, but what the heck - you're getting them for free, and they were pretty quick to whip up.

Service, Support, Setup

But before we begin with the numbers, I want to add a few words of caution. One of the big things about hosting, to me at least, is how they deal with unexpected random negative events. Connections going down, disk breakage, DoS attacks, and so on. It's really hard to get an idea of just what sort of people you're dealing with until you've "ridden the river together". I don't really know what sort of response times, uptime, and all that either Slicehost or Linode have, so there are potentially some big intangibles there that are not quite as easy to draw pretty charts for.

In terms of the console/setup/management tools, I liked Slicehost's simplicity more - Linode gives you more options, but they're a bit fiddly and cluttered feeling at times. For instance, Linode lets you pick how much swap space to give your disk image, but doesn't include a bit of javascript to balance out the swap and regular partitions, so that if you type in a larger swap number, you have to do the math and subtract that from the other number. Annoying, but not a big deal. Linode lets you pick various data centers in the US (no Europe, yet). Slicehost gives you the option to do backups of your disk images, which is nice, and something that Linode lacks.

Comparing plans

Since I'm interested in comparing hardware and machines I actually have access to, here are the plans I am currently signed up with, and the salient numbers:

Memory / Dollar:

Bandwidth / Dollar:

Disk / Dollar:

Linode mostly comes out ahead, but not by that much, except for bandwidth. However, let's take a crude look at x86_64 vs x86 memory usage.

First, ./script/console (a Rails console) for the same code base on two different machines, showing the virtual memory size (VSZ) and resident set size (RSS):

And just to look at something else that's a bit smaller, I ran the following script:

puts File.open("/proc/self/status", "r").read

And got the following data:

Being charitable with those numbers, we get x86 taking up 77% the memory that x86_64 does. So let's add that to our comparison of how much memory you're getting per dollar on Slicehost and Linode:

Wow! That's a fairly significant difference.

Performance

That's most of what we can glean from published numbers, and running a few simple experiments. However, there's another factor that's important: performance. "Virtual Private Server" systems, or simply VPS's, got a bad name in the past because they were "overbooked" - too many virtual servers competing for the same CPU resources on one machine. Slicehost and Linode both look like they want to avoid that kind of bad reputation, and so far all the systems I have used have felt snappy and responsive. Now here's where we get really unscientific... I decided to try and do something, even if it wasn't much, to get some kind of objective measurement of what kind of CPU I was getting on the machines. This is really unscientific, because the machines do have other duties (I don't have the cash, time or inclination to set up random servers just for testing), and of course, who knows what else is sharing the computer with my systems: if you get a relatively unused computer, you can apparently pick up extra cycles for yourself, over and above your guaranteed minimum. But c'est la vie, and so I decided to do some numbers anyway. I picked a C implementation of mandelbrot from the language shootout more or less "for the hell of it" (I'm going to keep repeating "unscientific" and invite someone to do better tests than I have). I ran this code every 20 minutes for a couple days, and then averaged out the run times:

As expected, the higher power Slicehost 1024 machine wins out in terms of raw speed (less time is better), but not by that much over the Linode system. Indeed, when we factor in the relative prices, Linode comes out ahead, again:

So it looks like we're not paying a CPU penalty for having more memory, bandwidth, disk, etc...

Notes

• Nothing in these statistics is indicative of future directions that both Linode and Slicehost might take. Slicehost was recently acquired by Rackspace, and that could have effects on the service they provide. Maybe one or both will come under financial pressure and start "overbooking" their servers.
• The spreadsheet I used is available via Google docs, for the curious. I did most of the work in OpenOffice, and then uploaded it though, because working with the web spreadsheet was kind of painful. Unfortunately, you can't select non-contiguous data ranges in the Google spreadsheet, so the labels are interspersed with the data. Yuck. Also, Google docs doesn't handle 'time' values nicely, which is what the performance data was originally. The spreadsheet is here: http://spreadsheets.google.com/pub?key=pVDeZF6ire2YwATBYb4mBKQ
• The standard deviation on the processing times is higher for both slices than the linode system. Not quite sure what that means in terms of what's going on under the hood, but I thought it was interesting.

I'm doing a bit of research for a microemulator feature request I filed. link

In short, the question is: on your cell phone's Java ME implementation, does the ticker scroll across the top get reset and start again from the side, or does the text get updated without the ticker restarting from the side, as in this example:

http://www.heclbuilder.com/scripts/show/155

On my Nokia 6121, it starts the text over each time it gets updated, which isn't how I wanted it to work. The debate regarding microemulator is how it ought to behave - update the text, or restart from the right, like the Sun emulator and my Nokia phone. To that end, we're looking for you to let us know what your phone does. Thanks!

Besides the nasty Ubuntu bug, which I was unable to do anything meaningful with, yesterday I also found a small bug in Ruby's YAML package:

irb(main):004:0> YAML::load('1900-01-01T00:00:00+00:00')
ArgumentError: time out of range

The problem is that the YAML implementation that Ruby is using, called "syck" interprets that kind of date as something that it should make a Time for:

        else if ( strcmp( type_id, "timestamp#iso8601" ) == 0 )
        {
            obj = rb_syck_mktime( n->data.str->ptr, n->data.str->len );

So I did a little bit of hacking on the C code, and made it create a DateTime object instead. My fix works, although I'm not sure it's the best possible way of tackling the problem. The patch is attached to the bug report here:

http://redmine.ruby-lang.org/issues/show/752

This matters because my Rails text fixtures had some dates in them that were quite old, and this was causing problems.

A lot of people, myself included as of the upgrade I did last night, seem to be having trouble using certain wireless chipsets with WPA:

https://bugs.edge.launchpad.net/ubuntu/intrepid/+source/wpasupplicant/+bug/272185

There are a lot of people following this one, and you have to think that for every person who looked it up, googled it, and dug around to find that page, registered with the site, and so on, there are several more who simply gave up and went back to Windows or something.

I thought a couple of weeks was probably a good amount of time for the bugs to be shaken out of Intrepid, but I thought wrong.

Here is a set of search terms that can be used to look through "regression-potential" bugs, which if I'm not mistaken, ought to mean those things that worked in previous versions of Ubuntu but now no longer work. In my case, there's an extra little bit of annoyance because I am running on a Dell laptop that I bought with Ubuntu preinstalled, and you'd think the Ubuntu guys would have a few sitting around to do some testing with prior to release.

I spent some time digging around in the code for the network manager, wpasupplicant, and a brief look at the kernel code, but there was nothing that made much sense, so I guess I'll just have to wait for the fix like everyone else... wish I could help, somehow, though.

http://adam.blog.heroku.com/past/2008/9/3/ddl_transactions/

I think the real reason there was resistance to this change is that MySQL doesn't support DDL transactions; and wrapping migrations in a transaction is mostly pointless without this feature. Since MySQL doesn't support it, most Rubyists don't know what DDL transactions are - so here's a quick primer on this incredibly useful and certainly underrated feature.

DDL = data definition language, also known as the schema. DDL commands include CREATE TABLE, DROP TABLE, ALTER TABLE, and CREATE INDEX. DDL transaction support means that the following sequence of SQL commands will not modify your database

I never even thought of that. I was fairly confident in my vision of Postgres getting faster and faster, and Mysql become a "real" database... that the two systems were converging on something resembling a common speed/feature set. But things like this still occasionally rear their ugly heads and I'm happy to be a Postgres user.

I have started to look at Passenger (mod_rails) + "Ruby Enterprise Edition" (which is just a corny name for a nice GC hack that significantly reduces memory consumption) because of the aforementioned problem with Slicehost's x86_64 amplifying Rails' already voracious memory gobbling. So I did a few crappy benchmarks, comparing Mongrel to mod_fcgid and mod_rails. I didn't look at concurrency, as that's a can of worms I wasn't prepared to open. I just wanted to see what raw speed was like. I used two methods from a project of mine, one, showall is quite DB intensive, and index does hit the DB, but not so much. 

Looks like mod_fcgid and mod_rails (both running in devel mode) are faster, at least for this very unscientific benchmark. One other advantage those two Apache modules have is that they make life simpler - mod_rails even more than mod_fcgid, and don't require an alphabet soup of mongrels and proxies and nginx, lighttpd or whatever the event-based server du jour is. Like I said though, the benchmarks are quick and dirty, and probably have issues. I'd be curious to see better ones *as long as they include mod_fcgid*.

Incidentally, it looks like linode.com uses regular x86 servers, so if you're doing Rails, this is definitely a point in their favor, or at least something to consider when you compare the memory that they and slicehost give you.

http://groups.google.com/group/emm-ruby/browse_thread/thread/f34bc994d7a83522

That said, I still don't have stable access to a Mac. Does anybody have a 64-bit OS X server and can provide me with SSH access? Preferably something that I can have access to for a long time, so that I don't have to keep asking for OS X access. :)

He's in .nl if it makes any difference. I'm not that wild about "passenger" by itself (mod_fcgid does ok for what I need), but the "enterprise ruby" project, which is basically a reworking of how Ruby's garbage collection functions, is a useful and timely idea, given how much of a memory hog rails is.

I've been working with Tomaso Minelli doing some Rails work, and one thing we needed to do was set up some tests. The situation we are in is that we already have a large set of data to work with in our development environment, but getting that data into .yml files is no easy task. Since we have a fairly large number of interdependent tables, dumping one of them alone is pretty useless, unless you manage to get the other tables it links to, and so on and so forth. Dumping all the data would make testing quite slow, and really is too much to be useful. Here's the rough solution we arrived at:

@fixtures = {}
@seen = {}

# Do something with a fixture.  In this case, we store it for later.
def handle_fixture(record)
  @fixtures[record.class.table_name] ||= []
  @fixtures[record.class.table_name] << record
end

# Get some sort of unique way of identifiying a record - class name and id.
def unique_record(record)
  return record.class.to_s + " " + record.id.to_s
end

# Grab all the relations of a class and add them to the queue.
def add_fixture(record)
  ur = unique_record(record)
  if @seen[ur]
    puts "LOOP!"
    return
  end
  @seen[ur] = true

  puts "Fixture for #{record.class.name} - ID #{record.id}"
  handle_fixture(record)
  record.class.reflections.each do |k, r|
    begin
      if r.macro == :has_many ||
          r.macro == :has_and_belongs_to_many
        related = record.send(r.name).send('find', :all, :limit => STARTING_LIMIT)
      else
        related = record.send(r.name)
      end
      related = [related] unless related.is_a?(Array)
      related.each do |related_record|
        if related_record
          puts "    New Record #{related_record.class.name} #{related_record.id}"
          @todo << related_record
        end
      end
    rescue => e
      puts "    ERROR: #{e}:\n#{e.backtrace.inspect}"
    end
  end
end

@todo = []

namespace :db do
  namespace :fixtures do
    desc 'Create YAML test fixtures starting from a single model using reflections'
    task :modelbase => :environment do
      STARTING_CLASS = Foobar
      STARTING_FIND  = :all
      STARTING_LIMIT = 1

      tmp = 0
      totale = STARTING_CLASS.find(STARTING_FIND, :limit => STARTING_LIMIT).size
      STARTING_CLASS.find(STARTING_FIND, :limit => STARTING_LIMIT).each do |m|
        tmp += 1
        puts "#{tmp} su #{totale}: #{m.id}"
        @todo << m
      end

      while true
        if @todo.length == 0
          break
        end
        add_fixture(@todo.pop)
      end

      puts "\n\n#{'='*80}\nScan Finished\n#{'='*80}\n\n"

      @fixtures.each do |name, fixture|
        if fixture.size > 0
          i = "000"
          filename = "#{RAILS_ROOT}/test/fixtures/#{name}.yml"
          puts "Create #{filename} for class #{fixture.class.name}"
          FileUtils.mkdir(File.dirname(filename)) unless Dir[File.dirname(filename)].size > 0
          File.open("#{RAILS_ROOT}/test/fixtures/#{name}.yml", 'w') do |file|
            file.write fixture.inject({}) { |hash, record|
              hash["#{record.class.table_name}_#{i.succ!}"] = record.attributes
              hash
            }.to_yaml
          end
        else
          puts "Fixture for #{name} is EMPTY"
        end
      end
    end
  end
end

It's written in a somewhat quick and dirty way, but it gets the job done. What it does:

It takes a single class - hopefully a well connected one - fetches a few of them, and goes from there, exploring all the other objects that the initial ones are connected to, and then dumping them in .yml files in the fixtures/ directory. It could certainly use more work:

• Additional starter classes, for ones that aren't connected to others.
• Intelligent examination of data and data sets...
• For example you could check to see what classes aren't connected to the others and use them as starter classes.
• If you wanted to waste a lot of cycles, you could also try and aim for a representative set of examples. For an object that has_many something elses, you could try and find one with zero, one with one, and one with a lot, to cover various cases. That could quickly get out of hand for largish data sets though.
• I'm sure there are various other tweaks, knobs, and levers that could be added to work more efficiently. I'd love to see them!

This is a worrying story:

http://blog.mibbit.com/?p=8

He did get his account reinstated, but being similarly quite dependent on Google for various services, this story was something of a wake up call for me. It's tough - they do offer a lot of attractive things for free, but single points of failure are bad, and even worse when they have a less than stellar support infrastructure to help you through the rough spots.