In my current work around adding tag support to Pebble, I came across the Yahoo! Term Extraction service, via TagCloud. Several people have already tried to use this technology as a way to automate the tagging process and there's been lots of talk about how automated tagging has the potential to pollute the tag clouds being created by sites like Technorati. Rather than re-iterate, here are some references.

• Tags vs. Yahoo Term Extraction
• Yahoo! Term Extraction, Technorati, and Tag Pollution
• Updated: Y! Terms Extraction Plugin

Most of this was written around April, so a few months on and I thought I'd have a go at writing a client in Java, which was just a matter of sending an HTTP request to the service and reading back the XML response. Here's the code, wrapped up in a Pebble BlogEntryListener plugin (javadoc stripped for brevity).

package pebble.event.blogentry;

import java.io.InputStream;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.ErrorHandler;
import org.xml.sax.SAXException;
import org.xml.sax.SAXParseException;
import pebble.blog.Blog;
import pebble.blog.BlogEntry;

public class YahooTermExtractionListener extends BlogEntryListenerSupport {

  private static final String URL =
    "http://api.search.yahoo.com/ContentAnalysisService/V1/termExtraction";

  private static final Log log =
    LogFactory.getLog(YahooTermExtractionListener.class);

  public void blogEntryAdded(BlogEntryEvent event) {
    tag(event.getBlogEntry());
  }

  private void tag(BlogEntry blogEntry) {
    Blog blog = blogEntry.getRootBlog();

    try {
      // post to the service using HttpClient
      HttpClient httpClient = new HttpClient();
      PostMethod postMethod = new PostMethod(URL);
      postMethod.addRequestHeader("Content-Type",
        "application/x-www-form-urlencoded; charset=" +
        blog.getCharacterEncoding());
      NameValuePair[] data = {
        new NameValuePair("appid", "PebbleWeblog"),
        new NameValuePair("context", blogEntry.getBody()),
      };
      postMethod.addParameters(data);
      int responseCode = httpClient.executeMethod(postMethod);
      if (responseCode != 200) {
        return;
      }

      // now parse the response and extract the Result elements
      DocumentBuilderFactory factory =
        DocumentBuilderFactory.newInstance();
      factory.setValidating(false);
      factory.setNamespaceAware(true);
      factory.setIgnoringElementContentWhitespace(true);
      factory.setIgnoringComments(true);
      DocumentBuilder builder = factory.newDocumentBuilder();
      builder.setErrorHandler(new ErrorHandler() {
        public void warning(SAXParseException e) throws SAXException {
          log.warn(e);
          throw e;
        }

        public void error(SAXParseException e) throws SAXException {
          log.error(e);
          throw e;
        }

        public void fatalError(SAXParseException e) throws SAXException {
          log.fatal(e);
          throw e;
        }
      });

      // add the terms to the list of tags for the blog entry
      StringBuffer tags = new StringBuffer(blogEntry.getTags());
      InputStream in = postMethod.getResponseBodyAsStream();
      Document doc = builder.parse(in);
      in.close();
      NodeList results = doc.getElementsByTagName("Result");
      if (results != null) {
        for (int i = 0; i < results.getLength(); i++) {
          Node node = results.item(i);
          String tag = getTextValue(node);
          if (tags.length() > 0) {
            tags.append(", ");
          }
          tags.append(tag);
        }
      }

      blogEntry.setTags(tags.toString());
      blogEntry.store();
    } catch (Exception e) {
      log.error(e.getMessage(), e);
    }
  }

  private String getTextValue(Node node) {
    if (node.hasChildNodes()) {
      return node.getFirstChild().getNodeValue();
    } else {
      return "";
    }
  }

}

To test it, I took this short blog entry and added it to my development blog. Here's the result, with the tags at the bottom of the image.

In summary, it's a mixed bag. Some of the terms are very relevant. Some, however, aren't. Automated tagging is certainly an interesting concept, and perhaps a great way of starting out with a larger set of tags which get chopped down by a human. I don't think you can rely on them alone though. Alternatively, and more interestingly, TagCloud are using the Yahoo! term extraction service as a basis and applying additional logic to rank the tags and ensure that they remain relevant. I think that this is something to keep an eye on.

Responses[5]

Posted by Simon Brown on 22 June 2005 12:30:43 BST #