Ruby’s Mysql interface

I just like open source too much to not be constantly bumping into things that I would love to improve or fix… I have way too much fun getting my hands dirty with something new.

Today’s adventure regards my need to stream large quantities of data from a ruby on rails installation, covered previously here:

https://journal.dedasys.com/articles/2006/06/08/streaming-programmatically-generated-content-from-rails

The current block is that ActiveRecord doesn’t deal with a “stream” of results, but rather, fetches everything into one big, fat array and then deals with that. Luckily, I’m not the only person to have found this suboptimal:

http://dev.rubyonrails.org/ticket/6921

However, as I comment there, to truly make things work, some low level coding is required. For mysql, since it doesn’t seem to support cursors to the degree that something more standards-compliant like Postgresql does, the answer lies in the C library and language interfaces built on top of it:

http://dev.mysql.com/doc/refman/5.0/en/mysql-use-result.html

Unfortunately, Ruby’s C API, or at least the versions that I downloaded, don’t properly support fetching things one record at a time. It didn’t take more than a few minutes to whip this up:

http://dedasys.com/freesoftware/patches/mysql-ruby.patch

(although there’s a good possibility it could be improved…)

However, according to this posting, the upstream maintainer may not be terribly responsive, and the project isn’t really run as much of an open source project, with a public mailing list, bug tracker, version control, and so forth.

Perhaps it’s time to create that infrastructure?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s