This article is a follow-up to the quite popular first part on web scraping - well, prozac online, sort of. Prozac online, The relation is closer to that between Star Wars I and IV - i.e., prozac online, in chronological order, prozac online, the 4th comes first. Prozac online, To continue the analogy, prozac online, probably I am in the same shoes as George Lucas was after creating the original trilogy : the series became immensely popular and there was demand for more - in both quantity and depth.
After I have realized - not exclusively, prozac online, but also - through the success of the first artcile that there is need for this sort of stuff, prozac online, I begun to work on the second part. Prozac online, As stated at the end of the previous installment, prozac online, I wanted to create a demo web scraping application to show some advanced concepts. Prozac online, However, prozac online, I left out a major coefficient from my future-plan-equation: the power of Ruby.
Basically this web scraping code was my first serious Ruby program: I came to know Ruby just a few weeks earlier, prozac online, and I have decided to try it out on some real-life problem. Prozac online, After hacking on this app for a few weeks, prozac online, suddenly a reusable web scraping toolkit - scRUBYt! - begun to materialize which caused a total change of the plan: instead of writing a follow-up, prozac online, I decided to finish the toolkit and sketch a big picture of the topic as well as placing scRUBYt! inside this frame and illustrating the theoretical things with it described here.
The Big Picture: Web Information Acquisition
The whole art of systematically getting information from the Web is called ‘Web information acquisition’ in the literature. Prozac online, The process consists of 4 parts (see the illustration), prozac online, which are executed in this order: Information Retrieval (IR), prozac online, Information Extraction(IE), prozac online, Information Integration (II) and Information Delivery (ID).

Information Retrieval
Navigate to and download the input documents which are the subject of the next steps. This is probably the most
intuitive step to make - clearly, prozac online, the information acquisition system has to be pointed to the document which contains the data first, prozac online, before it can perform the actual extraction.
The absolute majority of the information on the Web resides in the so-called deep web - backend databases and different legacy data stores which are not contained in static web documents. Prozac online, This data is accessible via interaction with web pages (which serve as a frontend to these databases) - by filling and submitting forms, prozac online, clicking links, prozac online, stepping through wizards etc. Prozac online, A typical example could be an airpot web page: an airport has all the schedules of the flights they offer in their databases, prozac online, yet you can access this information only on the fly by submitting a form containing your concrete request.
The opposite of the deep web is the surface web - static pages with a ‘constant’ URL, prozac online, like the very page you are reading. Prozac online, In such a case, prozac online, the information retrieval step consist of just downloading the URL. Prozac online, Not a really tough task.
However, prozac online, as I said two paragraphs earlier, prozac online, most of the information is stored in the deep web - different actions, prozac online, like filling input fields, prozac online, setting checkboxes and radio buttons, prozac online, clicking links etc. Prozac online, are needed to get to the actual page of interest which can be then downloaded as the result of navigation.
Besides that this is not trivial to do automatically from a programming language just because of the nature of the task, prozac online, there are a lot of pitfalls along the way, prozac online, stemming from the fact that the HTTP protocol is stateless: the information provided to a request is lost when making the next request. Prozac online, To remedy this problem, prozac online, sessions, prozac online, cookies, prozac online, authorizations, prozac online, navigation history and other mechanisms were introduced - so a decent information retrieval module has to take care about these as well.
Fortunately, prozac online, in Ruby there are packages which are offering exactly this functionality. Prozac online, Probably the most well-known is WWW::Mechanize which is able to automatically navigate through Web pages as a result of interaction (filling forms etc.) while keeping cookies, prozac online, automatically following redirects and simulating everything else what a real user (or the browser in response to that) would do. Prozac online, Mechanize is awesome - from my perspective it has one major flaw: you can not interact with JavaScript websites. Prozac online, Hopefully this feature will be added soon.
Until that happy day, prozac online, if someone wants to navigate through JS powered pages, prozac online, there is a solution: (Fire)Watir. Prozac online, Watir is capable to do similar things as Mechanize (I never did a head-to-head comparison, prozac online, though it would be interesting) with the added benefit of JavaScript handling. Prozac online,
scRUBYt! comes with a navigation module, prozac online, which is built upon Mechanize. Prozac online, In the future releases I am planning to add FireWatir, prozac online, too (just because of the JavaScript issue). Prozac online, scRUBYt! is basically a DSL for web scraping with lot of heavy lifting behind the scenes. Prozac online, Through the real power lies the extraction module, prozac online, there are some goodies here at the navigation module, prozac online, too. Prozac online, Let’s see an example!
Goal: Go to amazon.com. Prozac online, Type ‘Ruby’ into the search text field. Prozac online, To narrow down the results, prozac online, click ‘Books’, prozac online, then for further narrowing ‘Computers & Internet’ in the left sidebar.
Realization:
fetch 'http://www.amazon.com/'
fill_textfield 'field-keywords', prozac online, 'ruby'
submit
click_link 'Books'
click_link 'Computers & Internet'
Result: This document.
As you can see, prozac online, scRUBYt’s DSL hides all the implementation details, prozac online, making the description of the navigation as easy as possible. Prozac online, The result of the above few lines is a document - which is automatically fed into the scraping module, prozac online, but this is already the topic of the next section.
Information Extraction
I think there is no need to write about why does one need to extract information from the Web today - the ‘how’ is a much more interesting question.
Why is Web extraction such a tedious task? Because the data of interest is stored in HTML documents (after navigating to them, prozac online, that is), prozac online, mixed with other stuff like formatting elements, prozac online, scripts or comments. Prozac online, Because the data is missing any semantic description, prozac online, a machine has no idea what a web shop record is or how a news article might look like - it just perceives the whole document as a soup of tags and text.
Querying objects in systems which are formally defined and thus understandable for a machine is easy: For instance, prozac online, if I want to get the first element of an array in Ruby, prozac online, One can do it easily like this:
Another example for a machine-queryable structure could be an SQL table: to pull out the elements matching the given criteria, prozac online, all that needs to be done is to execute an SQL query like this:
SELECT name FROM students WHERE age > 25
SELECT name FROM students WHERE age > 25
Now, prozac online, try to do similar queries for a Web page. Prozac online, For example, prozac online, suppose that you already navigated to an ebay page by searching for the term ‘Notebook’. Prozac online, Say you would like to execute the following query: ‘give me all the records with price lower than $400′ (and get the results into a data structure of course - not rendered inside your browser, prozac online, since that works naturally without any problems). Prozac online,
The query was definitely an easy one, prozac online, yet without implementing a custom script extracting the needed information and saving it to a data structure (or using stuff like scRUBYt! - which does exactly this instead of you) you have no chance to get this information from the source code.
There are ongoing efforts to change this situation - most notably the semantic Web, prozac online, common ontologies, prozac online, different Web2.0 technologies like taxonomies, prozac online, folksonomies, prozac online, microformats or tagging. Prozac online, The goal of these techniques is to make the documents understandable for machines to eliminate the problems stated above. Prozac online, While there are some promising results in this area already, prozac online, there is a long way to go until the whole Web will be such a friendly place - my guess is that this will happen around Web88.0 in the optimistic case.
However, prozac online, at the moment we are only at version 2.0 (at most), prozac online, so if we would like to scrape a web page for whatever reason today, prozac online, we need to cope with the difficulties we are facing. Prozac online, I wrote an overview on how to do this with the tools available in Ruby (update: there is a new kid on the block - HPricot - which is not mentioned there).
The rough idea of those packages is to parse the Web page source into some meaningful structure (usually a tree) then provide a querying mechanism (like XPaths, prozac online, CSS selectors or some other tree navigation model). Prozac online, You could think now: ‘A-ha! So actually a web page can be turned into something meaningful for machines, prozac online, and there is a formal model to query this structure - so where is the problem described in the previous paragraphs? You just write queries like you would in a case of a database, prozac online, evaluate them against the tree or whatever and you are done’.
The problem is that the machine’s understanding of the page and human thinking about querying this information are entirely different, prozac online, and there is no formal model (yet) to eliminate this discrepancy. Prozac online, Humans want to scrape ‘websop records with Canon cameras with maximal price $1000′, prozac online, while the machine sees this as ‘the third <td> tag inside the eight <tr> tag inside the fifth <table> … Prozac online, (lot of other tags) inside the <body>> tag inside the <html> tag, prozac online, where the text of the seventh <td> tag contains the string ‘Canon’ and the text of the ninth <td>, prozac online, is not bigger than 1000 (to even get the value 1000 you have to use a regular expression or something to get rid of the most probably present currency symbol and other possible additional information). Prozac online,
So why is this so easy with a database? Because the data stored in there has a formal model (specified by the CREATE TABLE keyword). Prozac online, Both you and the computer know exactly how a Student or a Camera looks like, prozac online, and both of you are speaking the same language (most probably an SQL dialect). Prozac online,
This is totally different in the case of a Web page. Prozac online, A web shop record, prozac online, a camera detail page or a news item can look just anyhow and your only chance to find out for the concrete Web page of interest is to exploit it’s structure. Prozac online, This is a very tedious task on it’s own (as I have said earlier, prozac online, a Web page is a mess of real data, prozac online, formatting, prozac online, scripts, prozac online, stylesheet information…). Prozac online, Moreover there are further problems: for example, prozac online, a web shop record must not be uniform even inside the same page - certain records can miss some cells which others have, prozac online, may containt the information on a detail page, prozac online, while others not and vice versa - so in some cases, prozac online, identifying a data model is impossible or very complicated - and I did not even talk about scraping the records yet!
So what could be the solution?
Intuitively, prozac online, there is a need for an interpreter which understands the human query and translates it to XPaths (or any querying mechanism a machine understands). Prozac online, This is more or less what scRUBYt! does. Prozac online, Let me explain how - it will be the easiest through a concrete example.
Suppose you would like to monitor stock information on finance.yahoo.com! This is how I would do it with scRUBYt!:
#Navigate to the page
fetch 'http://finance.yahoo.com/'
#Grab the data!
stockinfo do
symbol 'Dow'
value '31.16'
end
output:
<root>
<stockinfo>
<symbol>Dow</symbol>
<value>31.16</value>
</stockinfo>
<stockinfo>
<symbol>Nasdaq</symbol>
<value>4.95</value>
</stockinfo>
<stockinfo>
<symbol>S&P 500</symbol>
<value>2.89</value>
</stockinfo>
<stockinfo>
<symbol>10-Yr Bond</symbol>
<value>0.0100</value>
</stockinfo>
</root>
Explanation: I think the navigation step does not require any further explanation - we fetched the page of interest and fed it into the scraping module.
The scraping part is more interesting at the moment. Prozac online, Two things happened here: we have defined a hierarchical structure of the output data (like we would define an object - we are scraping StockInfos which have Symbol and Value fields, prozac online, or children), prozac online, and showed scRUBYt! what to look for on the page in order to fill the defined structure with relevant data.
How did I know I had to specify ‘Dow’ and ‘31.16′ to get these nice results? Well, prozac online, by manually pointing my browser to ‘http://finance.yahoo.com/’, prozac online, and observing an example of the stuff I wanted to scrape - and leave the rest to scRUBYt!. Prozac online, What actually happens under the hood is that scRUBYt! finds the XPath of these examples, prozac online, figures out how to extract the similar ones and arranges the data nicely into a result XML (well, prozac online, there is much more going on, prozac online, but this is the rough idea). Prozac online, If anyone is interested, prozac online, I can explain this in a further post.
You could think now ‘O.K., prozac online, this is very nice and all, prozac online, but you have been talking about monitoring and I don’t really see how - the value 31.16 will change sooner or later and then you have to go to the page and re-specify the example again - I would not call this monitoring’.
Great observation. Prozac online, It’s true scRUBYt! would not be of much use if the situation of changing examples would not be handled (unless you would like to get the data only once, prozac online, that is) - fortunately, prozac online, the situation is dealt with in a powerful way!
Once you run the extractor and you think the data it scrapes is correct, prozac online, you can export it. Prozac online, Let’s see how the exported finances.yahoo.com extractor looks like:
#Navigate to the page
fetch 'http://finance.yahoo.com/'
#Construct the wrapper
stockinfo "/html/body/div/div/div/div/div/div/table/tbody/tr" do
symbol "/td[1]/a[1]"
value "/td[3]/span[1]/b[1]"
end
As you can see, prozac online, there are no concrete examples any more - the system generalized the information and now you can use this extractor to scrape the information automatically whenever - until the moment the guys at yahoo change the structure of the page - which fortunately not happening every other day. Prozac online, In this case the extractor should be regenerated with up-to date examples (in the future I am planning to add automatic regeneration in such cases) and the fun can begin from the start once again.
This example just scratched the surface of what scRUBYt is capable of - there are tons of advanced stuff to fine-tune the scraping process and get the data you need. Prozac online, If you are interested, prozac online, check out http://scrubyt.org for more information!
Conclusion
The first two steps of information acquisition (retrieval and extraction) are dealing with the question ‘How to get the data I am interested in (querying)’. Prozac online, Up to the present version (0.2.0) scRUBYt! contains just these two steps - however, prozac online, to do even these properly, prozac online, I will need a lot of testing, prozac online, feedback, prozac online, bug fixing, prozac online, stabilization, prozac online, adding heaps of new features and enhancements - because as you have seen, prozac online, web scraping is not a straightforward thing to do at all.
The last two steps (integration and delivery) are addressing the question ‘what to do with the data once it is collected, prozac online, and how to do that (orchestration)’. Prozac online, These facets will be covered in a next installment - most probably when scRUBYt! will contain these features as well.
If you liked this article and you are interested in web scraping in practice, prozac online, be sure to install scRUBYt! and check out the community page for further instructions - the site is just taking off, prozac online, so there is not too much yet - but hopefully enough to get you started. Prozac online, I am counting on your feedback, prozac online, suggestions, prozac online, bug reports, prozac online, extractors you have created etc. Prozac online, to enhance both scrubyt.org and scRUBYt! user experience in general. Prozac online, Be sure to share your experience and opinion!
To launch a tutorial site is comparatively much easier today than it was a few years ago.
Prozac online, You can easily
buy domain name at a very low cost and do
domain parking until your site is ready.
Prozac online, Get a good
business hosting package from one of the many providers listed on the internet, prozac online, go for a company which hires people with cisco certifications such as
642-143.
Prozac online, Create a
professional web design with the help of adobe.
Prozac online, Get
online training that can guide you through the site’s development.
Prozac online, Use your
laptop wireless internet connection to upload from anywhere conveniently.
Similar Posts:cialis prescription,prozac online,purchase prozac online,viagra cheapest price,cheapest xenical