As anyone who’s developed an add-on for Firefox is probably aware, Mozilla’s AMO pages offer a multitude of localization options for your add-on’s webpage: 33 different languages, including English, to be exact. This is an unparalleled opportunity to reach and communicate with users virtually everywhere around the world. However, actually translating your AMO page, quickly and economically, can be challenging. Outside of paying large sums for professional translation services, or hitting up foreign language-speaking friends, we know of two good options.

BabelZilla: www.babelzilla.org

While the UI might be a little obtuse and the process a bit complicated, many add-on developers have told us that they’ve had great success with BabelZilla. If you create an account and upload your add-on, volunteers will translate the embedded text for you. If you would also like to have the text on your AMO page translated, you could drop the strings in .property files. We were never quite able to get it to responses, and the activity seems somewhat low, but it’s free so you might want to give it a try. Apparently there are ways to kindly solicit translators on the forums.

Amazon Mechanical Turk: www.mturk.com

This is not free, but for a very modest amount of money you can get your AMO page quickly translated by native speakers. Create an account and then follow the instructions to upload the text of your AMO page. We used the “Basic Open-ended Question” template. On the Properties page you can specify the location of the worker. For virtually all of our translations we chose people living in the country where the language is spoken. The Reward is up to you, but we paid $5-10 per translation for our AMO page. Publish and then wait for the replies to roll in.

Some translations took a while to be returned, but the majority came in under 1 day, and some came back in hours. The average time a worker spent on a translation was 20-30 minutes. In the end, we were able to get 30 translations in just a few days:

Arabic (ar), Catalan (ca), Czech (cs), Danish (da), German (de), Greek (el), Basque (eu), Spanish (es-ES), Finnish (fi), French (fr), Gaeilge (ga-IE), Hebrew (he), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Korean (ko), Dutch (nl), Polish (pl), Brazilian Portuguese (pt-BR), European Portuguese (pt-PT), Romanian (ro), Russian (ru), Slovakian (sk), Albanian (sq), Swedish (sv-SE), Ukrainian (uk), Vietnamese (vi), Chinese (zh-CN) and Taiwanese (zh-TW). We’re still working on Farsi (fa) and Mongolian (mn).

There are some things to look out for.  When you get the translations you might want to drop them into http://translate.google.com/ which, while far from perfect, can help to make sure they are not way off base. Also, if you get a response from someone that took less than a few minutes, he or she probably just got a machine translation, which is not particularly desirable and naturally you could do yourself. Most people will not do this, but sometimes it might be worth specifying that you want a human translation. (As the Mozilla Devloper Wiki indicates, “You are kindly advised NOT to use any automatic online translation, which can bring unpleasant low-quality output to users.”) Lastly, if there are translations that look suspect, you can even go back into Mechanical Turk and set up another job to have the translations proofread by a different native speaker.

In the end it took a bit of effort, but making your hard development work understandable by so many people around the world makes it worth the while. Feel free to contact us should you have any questions or need any help.

Add-on-Con ’09 Cross Browser Development from Robert on Vimeo.

One of the most useful techniques for comparing the quality of search engine retrieval functions is the technique of result interleaving, invented by Thorsten Joachims of Cornell University. He first introduced the idea in a 2002 paper and has since recently expanded on the idea.

A search engine retrieval function is an algorithm that produces a ranked list of documents given a document collection and a user query. The retrieval function is the secret sauce behind the search engine. It is reported that Google, for instance, considers over 200 different document features when ranking web pages in response to a user query. These features are fed into the retrieval function which tells the web application which links to present and it what order.  In an open collection, such as the World Wide Web, different retrieval functions can produce both different document orderings as well as entirely different sets of documents.

Joachims came up with a very simple test that answers the following question: Given two retrieval functions, which does a search engine user prefer? His idea was that one can interleave the results of the two retrieval functions in an unbiased fashion and then count the user clicks on the links contributed by each retrieval function. The better retrieval function is the one that gets the most clicks.

For instance, assume that we have four documents (A, B, C, and D) that are relevant to a given query. According to the first retrieval function, r1, they should be ordered C-A-D-B. According to the second retrieval function, r2, they should be ordered D-A-C-B. The interleaved order of presentation would be D-C-A-B half the time and C-D-A-B half the time. We need to assure that each retrieval function gets to determine the document in the top spot half the time in order to have an unbiased test.  We would then show these document lists to many users as they conduct searches for this query. If we found that there were more user clicks on document C compared to document D, we can state that users prefer retrieval function r1. We can repeat this test for more documents and more queries, but by simply counting the clicks on documents contributed by each retrieval function we can determine an absolute user preference.

Surf Canyon implemented this test to compare our retrieval function, which employs real-time personalization based on implicit relevance feedback, with Google’s retrieval function. We always show the user the top 10 Google results, even with our application installed, so our interleaving test was only done when users asked for a second page of search results. In those cases, the results presented to the user would be a mixture of results 11 through 15 from Google and the most highly ranked personalized results from Surf Canyon’s retrieval function.

The figure below shows the ratio of link clicks on Surf Canyon results compared to Google results as a function of the number of search results selected by the searcher before preceding to the second page of results. A ratio less than 1.0 would indicate that they prefer un-personalized Google results, whereas a ratio greater than 1.0 would indicate a preference for Surf Canyon’s retrieval algorithm. A ratio of 1.0 would indicate no user preference. Note that the users do not know if they are selecting personalized or un-personalized links. The result is a very clear preference for Surf Canyon’s retrieval algorithm – users are 30-40% more likely to select Surf Canyon links.

We looked at this quantity versus the number of search results selected because the user’s interactions with the search page are what we use to personalize the results. The more the user selects, the more confident we are about the user’s true intent. Interestingly, users also prefer Surf Canyon results, by a significant margin, even when they skip the top 10 Google links entirely. When a user skips a link, we generally assume that the document is not what the user wanted. If the user skips all 10 links, we assume that the search engine misinterpreted the user intent and we start looking for different content that is not represented in the top 10 links.

Even though only 10% of searchers ever venture beyond page 1, we consider a 30-40% improvement in page 2 click-through rates to be significant. Quantitatively measuring the value delivered by real-time implicit personalization to page 1 results is, unfortunately, considerably more difficult. Nevertheless, be believe that these page 2 results are indicative of the value that real-time implicit personalization can delivery to page 1 results as well.

Update (7/15/09) – Our research paper, “Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback,” was selected for oral presentation at SIGIR ’09.

Update (12/18/09) – Our research paper was published by SIGIR.

Quantifying the “web search experience” is, however, very challenging. Nevertheless, search engines are constantly running small (and large) experiments to test how changes affect the user search experience. These experiments, which often use something called “A/B” or “bucket” testing, entail exposing a small, randomly selected set of users to the new features or changes and then comparing their behavior to the behavior of users on the baseline search site. Depending on the feature being tested, different user behavior signals are used to judge user satisfaction with the changes.

Since our technology radically changes the nature of the search results page, evaluating the application is particularly difficult. Once a user installs the Surf Canyon application, the search engine results page becomes dynamic and personalized to each user. Users who install Surf Canyon expect to get the Surf Canyon technology, so a traditional bucket test is not possible. (If we were to have a control sample of users who installed a special version of Surf Canyon that did not personalize their search results, those users would be perplexed and would probably uninstall the product.)

However, we performed a thorough evaluation of the technology using some traditional evaluation metrics from the Information Retrieval community as well as some new evaluation techniques that we invented ourselves. These evaluations are documented in a research paper which we recently drafted. In a subsequent blog post, we will detail our evaluation methodologies and the conclusions of these studies.

A good presentation should naturally start with the conclusions, so we reveal here in advance the conclusion of our studies so far: real-time personalization works.

Update (12/1/08) – Continued in Part 2.

Update (7/15/09) – Our research paper, “Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback,” was selected for oral presentation at SIGIR ’09.

Update (12/18/09) – Our research paper was published by SIGIR.

If the “Hillary Clinton” search results that we reviewed a month ago could be considered “on message,” with many of her official sites lined up on page 1 and only one mishap on the first two pages (position #16), then John McCain has got an SEO (Search Engine Optimization) problem. It doesn’t help either, we imagine, that some people have deployed “Google bombs” to sabotage his results. The outcome, however, consists of three panics in a row on the very first page of results and two Google bombs on the second. Furthermore, the top 100 results are filled with far fewer “official” sites (24 vs 10).

Below we take a light-hearted look at the top 100 results for “John McCain” on Google. We added the “bomb” icons to the results based strictly on information from a Search Engine Land post, however, it’s entirely possible that, unknown to us, there were people bombing Hillary Clinton’s results as well.

(Note: the “bomb” results are perhaps considered as such not necessarily because of the political stance, but because of the perceived unfavorable slant with which the stance is portrayed.)

Now you’ll be asked to forgive us for demonstrating how our application makes all of this searching stuff better.

With almost 40 million results for John McCain, most of which are probably not relevant to your specific needs, how could you hope to dig through them all? Certainly you could page, page, page, or reformulate your query, but all of this is hard work. Fortunately, Surf Canyon has developed a search application to do the heavy lifting for you.

After you install the software, when you enter a search on one of your favorite search engines (currently Google, Yahoo!, MSN Live and Craigslist) small bull’s eyes will appear next to each result. Click a bull’s eye next to the result you like, or the result itself for that matter, and Surf Canyon will figure out what you’re looking for and then dig deep into the result set (up to page 100!) and find what you need.

Here’s a quick example of what might happen if, using the search above, you click on result #2 (John McCain 2008…) and then the 3rd result recommended by Surf Canyon (McCain Agrees With Bush’s Remarks on Appeasement…):

Once again, with virtually no effort at all, relevant results, based on your actions as you search, are pulled out of the result set. If you would like this to happen for you, download and enjoy our search application.

Please let us know how did! Stay tuned for more…

Next to every search result Surf Canyon has added an “Email This” link (indicated by the red circles below) that, when clicked, will open up your email client and load up the result for easy delivery. Simply enter the email addresses of whoever you like and press “Send.”

Important Note: Surf Canyon does not collect email addresses and will never spam. We loathe spam. As a matter of fact, unless you put Surf Canyon on copy, we can’t even know your email address, the addresses of the recipients or even that you sent the email.

How to Find the Needles in Your Search Results Haystack

Enter a search for “Hillary Clinton” on your favorite search engine and the top results will often be bland, official stuff. So where’s the beef?

Few people will bother to venture past the first results page. It’s simply too much work. Nevertheless, given the enormity of information available on the internet, one has to imagine that there’s a lot more good information out there, sometimes the most juicy stuff. Turns out, it’s true!

Below we do the hard work for you and take a look at the top 100 results returned by Google for “Hillary Clinton.” While one person’s “hidden treasure” is another person’s buried garbage, it’s instructive to see how much potentially relevant information there is beyond the first page.

With millions of search results, what’s the best way to get at what you need?

If you’d like to quickly and easily access the buried treasure in your search results, download Surf Canyon and give it a try! You could manually go page by page, or have Surf Canyon do the work for you. We’ve automated the process of digging into the search results to make searching faster and more enjoyable

Here’s an example of the “Recommendations” Surf Canyon will retrieve, for the same query above, based on the user’s interest in result #2:

With virtually no effort, relevant and interesting information is dug out of the results.

Congratulations on getting this far! Please tell us how we did (below).

Stay tuned for more…

Update (6/27/08): “John McCain” gets a similar treatment.

Step 1: Enter you query at search.SurfCanyon.com or, if you like, download our add-on and then go to your favorite search engine (Google, Yahoo! or Bing), just like you always do, and enter your keywords.

Our example will use a search for the term “dolphins” since it is often used to illustrate one of the classic problems with information retrieval. No matter how much information is gathered about the subject, the individual, or all the individuals who have ever searched for “dolphins”, it is impossible to know a priori whether or not the searcher’s intent is related to the animal or to the football team. While search engines may suggest alternate queries or reformulations through the use of keyword suggestions or clusters, the search results produced will always be a mix of the animal and the football team.

Step 2: Review the results and select the one that corresponds to your interests. If it’s the Miami Dolphins that you like, click #2: MiamiDolphins.com. If your search is satisfied, great! If not, return to the search results and Surf Canyon will provide you with some “Recommendations”. You may, if you like, also get the “instant” Recommendations by simply clicking the bull’s eye next to the particular link of your choice.

Recommendations are dug out of the search results based on the Discovery Engine’s inference of our real-time intent. Put another way, Surf Canyon figures out what you want and then goes past the first page of the results and fetches it for you.

Step 3: The process is cumulative, so continue clicking on links and bull’s eyes that are of interest to you. The longer you work on the result set, the better Surf Canyon understands your intent and the more accurate are the Recommendations. The Recommendations can nest to three levels, enabling you to “drill down” on not just your high-level intent, but your sub-intents as well, such as “Miami Dolphins news.”

Click on “More Results” and you’ll get a second page of results automatically customized to your intent. In this case, they are naturally all related to the Miami Dolphins football team and are geared towards news.

Keep going and Surf Canyon will keep working to help you find what you need.

That’s Discovery!

Lastly, don’t worry about ‘mistakes’: clicking the links that are perhaps not of interest. Surf Canyon is smart enough to handle it.