I just like open source too much to not be constantly bumping into things that I would love to improve or fix… I have way too much fun getting my hands dirty with something new.
Today’s adventure regards my need to stream large quantities of data from a ruby on rails installation, covered previously here:
The current block is that ActiveRecord doesn’t deal with a “stream” of results, but rather, fetches everything into one big, fat array and then deals with that. Luckily, I’m not the only person to have found this suboptimal:
http://dev.rubyonrails.org/ticket/6921
However, as I comment there, to truly make things work, some low level coding is required. For mysql, since it doesn’t seem to support cursors to the degree that something more standards-compliant like Postgresql does, the answer lies in the C library and language interfaces built on top of it:
http://dev.mysql.com/doc/refman/5.0/en/mysql-use-result.html
Unfortunately, Ruby’s C API, or at least the versions that I downloaded, don’t properly support fetching things one record at a time. It didn’t take more than a few minutes to whip this up:
http://dedasys.com/freesoftware/patches/mysql-ruby.patch
(although there’s a good possibility it could be improved…)
However, according to this posting, the upstream maintainer may not be terribly responsive, and the project isn’t really run as much of an open source project, with a public mailing list, bug tracker, version control, and so forth.
Perhaps it’s time to create that infrastructure?
DedaSys Journal 