I am just working on a brand new release of scRUBYt!, with the intent of bringing AJAX/javascript scraping to the masses (and other great stuff – will announce the release soon).

Scraping js-heavy pages is not that trivial, among other things because of the asynchronous nature of Javascript. Quite frequently you click on a link (or do any other action triggering an AJAX call) which inserts/fills/pops up a div on the page and you want to navigate/get some data from the new content. However, it’s hard to impossible to determine when did the browser finish displaying the new data – the easiest solution is to wait a few seconds after an AJAX update, until the data is properly loaded.

In practice this means that all scRUBYt! navigation methods (click_link, fill_textfield, check_checkbox, select_item, …) need a decorated version, which waits a given amount of time after executing the navigation step. So for example, given the original method:

def click_link(xpath) ... end

we want a decorated version:

def click_link_and_wait(xpath, seconds)
  click_link xpath #the original method
  sleep seconds if seconds > 0
end

For each and every method of the NavigationAction class.

Fortunately Ruby makes this really easy! Decorating the existing methods explicitly upon class creation:

  (instance_methods - Object.instance_methods).each do |old_method|
    define_method "#{old_method}_and_wait" do |seconds|
      send old_method ; sleep seconds
    end
  end

or implicitly, runtime:

alias_method :throw_method_missing, :method_missing

def method_missing(method_name, *args, &block)
  original_method_name = method_name.to_s[/(.+)_and_wait/,1]
  if (method_name.to_s =~ /_and_wait/) && (respond_to? original_method_name)
    self.send original_method_name ; sleep args[0]
  else
    throw_method_missing(method_name, *args, &block)
  end
end

As you can see, we are executing the decorated method only if it is defined on our class and it ends in _and_wait. In all other cases we simulate the normal method_missing_behavior.

Since we know in advance that we want to decorate all the methods of the class, the second way doesn’t make much sense in this case – it’s slower because it has to go through method_missing every time, while there are no advantages – however the technique is interesting and applicable in other scenarios (e.g. when you don’t know in advance which methods are you going to decorate – for example adding a constraint to a filter in scRUBYt! (filters are also created dynamically runtime))

I Don’t want to reiterate what has already been said on this topic (I ended up in the same boat – tried more tools, didn’t really like any of them and settled for giternal).

Giternal seems to work nicely – after you get the initial roadblock out of the way. According to the README, you should install giternal with

gem install giternal

then use the _giternal_ executable to do various things.

The problem is that after installation, there is no _giternal_ executable – the problem is that gem installs version 0.0.1, which doesn’t have it. You have to get the sources from git and set it up yourself:

git clone git://github.com/pat-maddox/giternal.git
cd giternal
sudo ruby setup.rb

and you are good to go.

Hope this saves someone a few minutes.

I decided to join Ruby Quiz for the first time ever – I am realizing that I am extremely late to jump on the RQ bandwagon – but hey, is still better later than never. I have chosen a funny moment though: it’s a quiz everyone was excited about, but almost nobody solved (2 solutions so far except mine, which is maybe the lowest number of solvers ever). So here’s the quiz:

Your task is to write a units converter script. The input to the
script must be three arguments: the quantity, the source units, and
the destination units. The first example above would be run like this:

    $ ruby convert.rb 50 miles kilometers

Or, using abbreviations:

    $ ruby convert.rb 50 mi km

Support as many units and categories of units (i.e. volume, length,
weight, etc.) as you can, along with appropriate abbreviations for
each unit.

and my solution:

require 'rubygems'
require 'cgi'
require 'scrubyt'

begin
google_converter = Scrubyt::Extractor.define do
 fetch "http://www.google.com/search?q=#{CGI::escape(ARGV[0])}+#{CGI::escape(ARGV[1])}+to+#{CGI::escape(ARGV[2])}"

 google_result "//td[@dir='ltr']" do
   final_result(/= (.+) /)
 end
end
 puts google_converter.to_hash[0][:final_result]
rescue
 puts "Sorry, even *google* can't translate that!"
end

Examples:

ex:
ruby converter.rb 10 "meter per second" "mile per hour"
22.3693629

ruby converter.rb 10 USD EUR
7.91201836

ruby converter.rb 7 "ruby gems" "python eggs"
Sorry, even *google* can't translate that!

The biggest downside of the proposed solution is that you need to be on-line. However, nowadays, when this is almost the norm, the advantages outweigh this in my opinion:

• Support for a lot of conversions (I doubt anyone can offer a much richer database of units than google) including their abbreviations
• Up-to-date conversions: for example currency conversion
• Very robust error handling – outsourced to google! Hard to beat that…

“It’s better to be a pirate than to join the Navy” (Steve Jobs)

This post was inspired by the last part of Jim Neath’s article on speeding up Rails development, titled “Seriously, Just Buy a Fucking Mac”. Some commenters insisted that you can do just as well on other systems (Ubuntu, in particular). In three short words, I don’t agree. In my opinion Ubuntu is a valid alternative but it comes in at second place, and the gap between the 1st and 2nd position is significant. Why?

Software

Some people, when it comes to a good PC vs. Mac fight, start to brag about the hardware – it’s overpriced, and not that much better, and my high end PC notebook is better because … etc. Because they most probably never ever used a Mac for a longer period, they don’t know that the real speed gain/ease || joy of use comes from the software (though it doesn’t hurt that there are no hardware conflicts, evar, no matter how much do you upgrade, no messing around with wifi/mic/video cards/you name it and the quality is top-notch etc.) No. The real secret sauce is the software:

The essential stuff

Other niceties

Once you get productive with these extremely powerful tools (and I didn’t even mention the standard built in stuff like Spotlight, iCal, iChat, Time Machine, …) your development time will be greatly reduced compared to that on an Ubuntu/Windows/… machine. For realz!

Everyone and Their Dog is on a Mac

So… why should I keep up with the Joneses – did the me-too club become popular recently? Isn’t Apple’s mantra ‘think different’? Why join the ‘cool kids’?

Valid questions – however, everyone else being on macs also means that

• you’ll have a much bigger chance of getting quick support on the Ruby/Rails forums/IRC – which is not always easy with Ubuntu (I am on Ubuntu Rusty Robot build R2-D2. Everything is fine, but MySQL driver doesn’t compile because I ran dist-upgrade yesterday, and now everything is jammed. All the tutorials are up to Humpty-dumpty Drake only…).
• Easier collaboration – sharing Dropbox folders, skitch images, iCal events etc. is very-very common – of course there are other ways of sharing data/notes/collaboratively editing documents, but these Mac tools take it to a whole new level.
• Growing Mac usage means that more goodness is on the way!

A bit of Mythbusting

• Macs are crazy expensive

You can buy a mac mini starting from $599, or a pre-owned one for $350. Doesn’t sound like bank-breaking. Good PCs are not cheap either – my last DELL was about the same price as my current (almost high-end) MBP. Of course you can buy an Acer for third of the price, but well, the drop in the quality will be proportional.

• Mac software is expensive, on linux everything is free

From the above list, everything essential is free, except TextMate, which is a must, no matter how much it costs. Usually free alternatives exist for almost everything, and the commercial ones are just unbelievably good.

• You are going to deploy to a linux server, so create your app on linux too

While OS X is not linux based, the difference is minimal – Darwin is a POSIX compliant OS so from the Rails development perspective it doesn’t really matter. I do not have accurate data but I don’t think so it’s easier to migrate an app developed on gentoo/fedora/suse/… to an ubuntu box than to do the same for an OS X Rails app. And even deploying from Ubuntu to Ubuntu can be a pain unless the boxes contain the same version of everything etc.

• I think you just bought a Mac because the “cool kids” did

So what? Does this alter the fact that working on a Mac makes you more productive/effective as described above? Not at all.

Use the Best Tool for the Job

Yeah… in other words, seriously, just buy a fucking mac. It’s the best tool when it comes to (not only) Ruby/Rails development.