{"id":7,"date":"2006-11-05T07:58:10","date_gmt":"2006-11-05T15:58:10","guid":{"rendered":"https:\/\/oroup.com\/blog\/2006\/11\/05\/the-joys-of-screenscraping\/"},"modified":"2006-11-05T07:58:10","modified_gmt":"2006-11-05T15:58:10","slug":"the-joys-of-screenscraping","status":"publish","type":"post","link":"https:\/\/oroup.com\/blog\/2006\/11\/the-joys-of-screenscraping\/","title":{"rendered":"The joys of screenscraping"},"content":{"rendered":"

A while back, I ran across a great HackDiary entry<\/a> extolling the virtues of using TagSoup<\/a> and XPATH to do screenscraping from the web. TagSoup is a library that coerces all the ugly nasty HTML you find out in the wild into well-formed (although not necessarily valid) XML. While there’s no guarantee that the results are semantically the same as the input, it lets you use all your nice XML tools like XPATH to extract data. The entry does a great job of showing you how to use TagSoup with Xalan<\/a>. However, the JDK has been updated with it’s own XPATH parser so it’s no longer necessary to import the Xalan library. Below is a code sample for using TagSoup and the default XPATH parser to retrieve the stock price of Google from Google Finance. Note that the whole “MutableNamespaceContext” implementation is just a workaround for a missing JDK method as documented in JDK Bug 5101859<\/a>. If that bug gets fixed, the code could be simplified substantially.<\/p>\n

The usual disclaimers apply about this all being sample quality code. All error handling has been punted to keep the example length short, but you’d never really want to do that. Also, I’m having trouble preserving indenting in this HTML View, I’ll work on that. This code assumes JDK 1.5. Click through for the code itself.<\/p>\n

Does this code support the argument<\/a> that Java is WAY too verbose? Absolutely.
\n <\/p>\n