ChengXiang received his Ph.D. in Computer Science from Nanjing University in 1990 and another Ph.D. in Language and Information Technologies from Carnegie Mellon in 2002. His research interests include information retrieval, text mining, natural language processing, machine learning and bioinformatics, and he has published over 100 papers in major conferences and journals in these areas, including five award papers.

He is an Associate Editor of ACM Transactions on Information Systems and Information Processing and Management, and serves on the editorial board of the Information Retrieval Journal. He is a program co-chair of ACM CIKM 2004 , NAACL HLT 2007, and ACM SIGIR 2009, where Mark Cramer was a presenter. He is an ACM Distinguished Scientist and has received many awards, including the 2004 Presidential Early Career Award for Scientists and Engineers (PECASE), the 2008 Alfred P. Sloan Research Fellowship, the 2007 Microsoft Beyond Search – Semantic Computing and Internet Economics Award, the 2009 IBM Faculty Award, the 2010 UIUC Rose Award for Teaching Excellence and the 2011 HP Innovation Research Award.

We are thrilled to have ChengXiang on the team as we look forward to exploring new avenues to improve the efficiency of information retrieval as well as continuing our collaboration with the UIUC.

For ambiguous queries, conventional retrieval systems are bound by two conflicting goals. On the one hand, they should diversify and strive to present results for as many query intents as possible. On the other hand, they should provide depth for each intent by displaying more than a single result. Since both diversity and depth cannot be achieved simultaneously in the conventional static retrieval model, we propose a new dynamic ranking approach. In particular, our proposed two-level dynamic ranking model allows users to adapt the ranking through interaction, thus overcoming the constraints of presenting a one-size-fits-all static ranking.

Surf Canyon is again referenced, along with our 2009 SIGIR research paper:

We argue that a key to solving the conflict between depth and diversity lies in the move to dynamic retrieval models [2] that can take advantage of user interactions. Instead of presenting a single one-size-fits-all ranking, dynamic retrieval models allow users to adapt the ranking dynamically through interaction, as is done by surfcanyon.com [5].

In February 2008, Surf Canyon launched its Dynamic Ranked Retrieval application to rave reviews. As the body of research relating to Dynamic Ranked Retrieval grows, we continue to be encouraged by the potential of this technology to vastly enhance the quality of information retrieval.

In §4.6.1, entitled “Beyond Predicting Static Ranking,” Yisong writes (emphasis added):

“It is common for information retrieval research to focus either on relevance estimation or user interface design, but rarely both simultaneously. However, for many tasks, it can be useful to model both jointly… One major limitation of result diversification over static rankings is that it sacrifices recall in favor of some minimal amount of utility for all usage intents – such a limitation could be dealt with by moving towards more dynamic interfaces.

Consider the example interface shown in Figure 4.8, which is inspired by and adapted from the SurfCanyon.com search engine [48]… by clicking or mousing over a result that matches the user’s intent, additional indented results are inserted into the original ranking… This interaction is quite natural, since the process resembles navigating a dropdown menu and since users are already familiar with result indentation. And yet even this one additional degree of freedom in content display can offer tremendous benefits…”

After the University College Dublin paper “A Recommender System Approach to Enhance Web Search and Query Formulation” and the Universidade Estadual de Maringá paper “An Approach to the Customization of Web Search Results,” this is the third academic reference to Surf Canyon. We’re delighted by the attention we’re receiving from the academic community.

Lastly, the team at Cornell recently drafted a brilliant paper, entitled “Dynamic Ranked Retrieval,” which dives deep into the study of real-time implicit ranking and offers statistical support for the “tremendous benefits” described above. It has been accepted for publication at WSDM 2011. While not yet public (we’ll post here when it is), we’ve been given permission to offer a sneak preview from the introduction (emphasis added):

“… most queries are ambiguous at some level. For such queries, there is often no single ranking that satisfies all users and query intents. While result diversification aims to provide a “compromise ranking” that provides some utility for all intents, diversification necessarily sacrifices recall…

The key idea is to make the ranking “dynamic” – namely, allowing it to change in response to user interactions after the query was issued.

From the user’s perspective, this may look as illustrated in Figure 1. This interface is inspired by and adapted from the SurfCanyon.com search engine…”

[Update 11/28/2010] The final version of “Dynamic Ranked Retrieval” is now available.

[Update 2/9/2011] The WSDM 2011 conference selected “Dynamic Ranked Retrieval” as one of six Best Paper Candidates.

They begin by observing that “the traditional search interface has remained relatively static” before going on to describe “a recommender system approach to Web search which allows users to dynamically interact with the result-space that is of interest to them,” which they see as “an overlay interface as a complement to an existing search engine.”  This will sound familiar to anyone who has used Surf Canyon, which the researchers acknowledge (emphasis added):

“… Surf Canyon is an excellent example of a next-generation approach, and allows users to interact with results returned by existing engines; in short, users can select a result and receive recommendations drawn  from related results that appear further down a result list. Surf Canyon launched in the latter stages of our own research and presents an interface similar to what is proposed here.”

While there are significant differences in terms or implementation, their results are similar to what Surf Canyon has demonstrated:

“A key question is whether the resulting recommendations were found to be relevant. In fact… at least one of the 3 recommended results was selected approximately 25% of the time, a very significant indicator of relevance…”

“Results offered by search engines mix websites that are truly relevant to the context of the user with websites that have a marginal similarity to the query, or that are totally irrelevant… results offered by search engines, which tend to return many irrelevant websites, require the user to perform a manual filtering of results to obtain those that truly reflect his or her interest. The effort employed in this filtering may be high according to the classification accuracy of the results offered by the search engine (HARDTKE, 2009; BRUSILOVSKY, 2009; [B] MICARELLI, 2007; PANT, 2003).”

“According KHOPKAR (2003), search engines that have features to enable user interaction in order to construct a model of user interests have a higher level of complexity of interaction. This interaction can occur in two ways: implicit feedback, in which user actions are captured in a non-intrusive manner, or explicit feedback, which requires the user’s direct intervention… Software such as Surf Canyon (HARDTKE, 2009) uses both modes of interaction, however, lay users have difficulty interacting with additional features such as icons displayed at the end of each link enabling the approximation of similar websites…”

“Surf Canyon is an interactive IR system that dynamically modifies the query results using Web-based personalization… This software is an extension of the user’s browser and operates on various search engines. The personalization of Web query results offered by this tool comes from the combination of implicit and explicit feedback. The classification is based on a user model that infers the immediate interest of the user. Surf Canyon adds an icon at the end of each result offered by the search engine. When you visit the result or click on this icon, Surf Canyon recommends links similar in context to the one clicked (HARDTKE, 2009).”

“User actions that are considered to identify a positive interest in a website are:

• Length of stay on the document between 2 and 30 minutes;
• Action to Save the document on the computer;
• Action to Add the document in the browser favorites.

These choices are given considering the work of Goecks and Shavlik (1999), LIU (2006), [B] MICARELLI (2007) and MANNING (2008), HARDTKE (2009) and also a survey of users described in Chapter II… HARDTKE (2009) comments in his work that a user click on a hyperlink returned by a query already represents an interest in that website…”

“The development of this work meets the discussions exposed by BRUSILOVSKY (2009), HARDTKE(2009), MANNING (2008), [B] MICARELLI (2007), among others, regarding the need to reduce the effort of locating relevant content on the Web by adapting the results obtained from the search engines according to the interests of user.”

We’re flattered and humbled by the continued recognition of our work.

During the submission process, one reviewer, an Associate Professor of Computer Science, had this to say:

“The paper presents an empirical evaluation of an algorithm for improving search results from implicit feedback in web search. The algorithm is proprietary, but nevertheless the evaluation is extremely interesting. Unlike most prior work, the evaluation is done on an operational system with real users, and it gives a lot of insight into the benefits of short term personalization. This is a great paper and one of the most interesting IR papers I have read in a while! This paper by itself would already make me come to the workshop. Very interesting, original, and relevant.”

Charles Knight, at Alt Search Engines, followed up with a post.

[Update 12/18/09] – The “Understanding the User: Logging and Interpreting User Interactions in Information Search and Retrieval” workshop report was recently posted on the SIGIR forum. The workshop proceedings, with a link to our research paper, have also been published.

Personalization is often been divided into two categories, implicit and explicit, the respective merits of which have been debated. Implicit personalization is based on preferences inferred from behavioral information. While this doesn’t require any effort on the part of the user, making accurate determinations of intent can be challenging. On the other hand, explicit personalization is driven by direct indication from the user. The intent is often clearer, however, the onus is on the user to make the effort to specify preferences.

Since April 2006, Surf Canyon has been looking at personalization from a different perspective: real-time vs. long-term. Real-time personalization alters the user experience instantly as behavioral signals are collected. While determining intent “on the fly” is challenging given the requirement for speed and the paucity of data, the signals are typically very strong. Long-term personalization, by contrast, relies on the accumulation of considerable user data over a significant amount of time. Determining the user’s “at the moment” intent can be difficult given how quickly the signals decay and how often users change context.

Nevertheless, as indicated here, all of these options are currently available to the internet searcher:

Surf Canyon v1 – Our flagship product introduced real-time implicit personalization for search. By observing the actions of the user as the search is taking place, the application helps people find information by re-ranking the results instantly, effectively transforming the search page from a static list of links to a dynamic set of results that “work with” the user.

Google Personalization – By observing the search pattern and click history of the user over an extended period of time, Google builds a profile of the user’s long-term interests which are then used to personalize the results for future searches. The first post on this blog analyzed some of the benefits and shortcomings of this technology.

Surf Canyon v2 – Launched in December ’08, v2 of Surf Canyon introduced my.SurfCanyon.com, enabling users, should they so desire, to explicitly indicate sources of content they prefer as well as those that they dislike. These long-term preferences are then taken into account to, once again, personalize search results to the user’s benefit.

Google SearchWiki – Launched in November ’08, SeachWiki offers controls to enable users to manually manipulate search results. By clicking a button, results may immediately be “promoted” to the top of the search page or deleted altogether. The next time the user runs the same search, the user’s personal modifications will be displayed.

Different users will have varying appreciations for the costs and benefits associated with each of these technologies, however, they are all compatible and, to a large extent, compliment each other. Today’s internet searchers are therefore free to use all of them, some of them or none of them, as they prefer.

Update (1/15/09) – Charles Knight at AltSearchEngines ran this post under their “Guest Authors” series.

One of the most useful techniques for comparing the quality of search engine retrieval functions is the technique of result interleaving, invented by Thorsten Joachims of Cornell University. He first introduced the idea in a 2002 paper and has since recently expanded on the idea.

A search engine retrieval function is an algorithm that produces a ranked list of documents given a document collection and a user query. The retrieval function is the secret sauce behind the search engine. It is reported that Google, for instance, considers over 200 different document features when ranking web pages in response to a user query. These features are fed into the retrieval function which tells the web application which links to present and it what order.  In an open collection, such as the World Wide Web, different retrieval functions can produce both different document orderings as well as entirely different sets of documents.

Joachims came up with a very simple test that answers the following question: Given two retrieval functions, which does a search engine user prefer? His idea was that one can interleave the results of the two retrieval functions in an unbiased fashion and then count the user clicks on the links contributed by each retrieval function. The better retrieval function is the one that gets the most clicks.

For instance, assume that we have four documents (A, B, C, and D) that are relevant to a given query. According to the first retrieval function, r1, they should be ordered C-A-D-B. According to the second retrieval function, r2, they should be ordered D-A-C-B. The interleaved order of presentation would be D-C-A-B half the time and C-D-A-B half the time. We need to assure that each retrieval function gets to determine the document in the top spot half the time in order to have an unbiased test.  We would then show these document lists to many users as they conduct searches for this query. If we found that there were more user clicks on document C compared to document D, we can state that users prefer retrieval function r1. We can repeat this test for more documents and more queries, but by simply counting the clicks on documents contributed by each retrieval function we can determine an absolute user preference.

Surf Canyon implemented this test to compare our retrieval function, which employs real-time personalization based on implicit relevance feedback, with Google’s retrieval function. We always show the user the top 10 Google results, even with our application installed, so our interleaving test was only done when users asked for a second page of search results. In those cases, the results presented to the user would be a mixture of results 11 through 15 from Google and the most highly ranked personalized results from Surf Canyon’s retrieval function.

The figure below shows the ratio of link clicks on Surf Canyon results compared to Google results as a function of the number of search results selected by the searcher before preceding to the second page of results. A ratio less than 1.0 would indicate that they prefer un-personalized Google results, whereas a ratio greater than 1.0 would indicate a preference for Surf Canyon’s retrieval algorithm. A ratio of 1.0 would indicate no user preference. Note that the users do not know if they are selecting personalized or un-personalized links. The result is a very clear preference for Surf Canyon’s retrieval algorithm – users are 30-40% more likely to select Surf Canyon links.

We looked at this quantity versus the number of search results selected because the user’s interactions with the search page are what we use to personalize the results. The more the user selects, the more confident we are about the user’s true intent. Interestingly, users also prefer Surf Canyon results, by a significant margin, even when they skip the top 10 Google links entirely. When a user skips a link, we generally assume that the document is not what the user wanted. If the user skips all 10 links, we assume that the search engine misinterpreted the user intent and we start looking for different content that is not represented in the top 10 links.

Even though only 10% of searchers ever venture beyond page 1, we consider a 30-40% improvement in page 2 click-through rates to be significant. Quantitatively measuring the value delivered by real-time implicit personalization to page 1 results is, unfortunately, considerably more difficult. Nevertheless, be believe that these page 2 results are indicative of the value that real-time implicit personalization can delivery to page 1 results as well.

Update (7/15/09) – Our research paper, “Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback,” was selected for oral presentation at SIGIR ’09.

Update (12/18/09) – Our research paper was published by SIGIR.

Quantifying the “web search experience” is, however, very challenging. Nevertheless, search engines are constantly running small (and large) experiments to test how changes affect the user search experience. These experiments, which often use something called “A/B” or “bucket” testing, entail exposing a small, randomly selected set of users to the new features or changes and then comparing their behavior to the behavior of users on the baseline search site. Depending on the feature being tested, different user behavior signals are used to judge user satisfaction with the changes.

Since our technology radically changes the nature of the search results page, evaluating the application is particularly difficult. Once a user installs the Surf Canyon application, the search engine results page becomes dynamic and personalized to each user. Users who install Surf Canyon expect to get the Surf Canyon technology, so a traditional bucket test is not possible. (If we were to have a control sample of users who installed a special version of Surf Canyon that did not personalize their search results, those users would be perplexed and would probably uninstall the product.)

However, we performed a thorough evaluation of the technology using some traditional evaluation metrics from the Information Retrieval community as well as some new evaluation techniques that we invented ourselves. These evaluations are documented in a research paper which we recently drafted. In a subsequent blog post, we will detail our evaluation methodologies and the conclusions of these studies.

A good presentation should naturally start with the conclusions, so we reveal here in advance the conclusion of our studies so far: real-time personalization works.

Update (12/1/08) – Continued in Part 2.

Update (7/15/09) – Our research paper, “Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback,” was selected for oral presentation at SIGIR ’09.

Update (12/18/09) – Our research paper was published by SIGIR.

In their paper entitled “Beyond the Commons: Investigating the Value of Personalizing Web Search,” Teevan et al. make the observation that:

“Web queries are very short, and it is unlikely that a two- or three-word query can unambiguously describe a user’s informational goal.”

Ambiguous intent combined with an exploding quantity of content on the internet makes it increasingly difficult to put all of the relevant results on page one while simultaneously eliminating those that are not pertinent.

Very few people venture past the first page of search results to find what they want, so returning hundreds of thousands or even millions of results is of little value to the user. (You cannot look past the first 1000 even if you wanted to.) Even if the user is particularly motivated, the process of digging through page after page of results is nothing short of tedious, which is the reason users will either quickly turn to reformulating their query or abandoning the search.

The problem is too many results!

The solution to the conundrum is to have a greater understanding of the user’s intent in order to more precisely focus the results. One way to achieve this is to get the user to explicitly specify intent by entering more keywords, although getting people to change behavior is not easy. Another way to achieve this is to implicitly infer intent through the type of long-term personalization offered by Google, although this too has a number shortcomings.

The most effective way to resolve this issue is to implicitly infer intent from real-time behavior signals and then immediately re-rank the results, through the use of instantaneous relevancy calculations, so that the most pertinent results are moved to the top while the less relevant are suppressed. Surf Canyon‘s Discovery for Search™ is such a solution. Disambiguating intent “on the fly” not only enables users to continue searching with their current behavior, but no search histories or profiles are required. Furthermore, the signals are strong so that the results can be reordered dramatically and the user can actually “see” the process working, creating a more encouraging and perhaps entertaining search experience.