• Over 500 thousand people benefited from our search technology
• Over 80 million queries on Google, Yahoo!, Live Search and Craigslist were enhanced using our real-time re-ranking
• Over 350 million search results were “recommended”
• We shared the Mossberg Solution column in the WSJ with Google and were very well received
• ReadWriteWeb selected us as one of the Top 100 Products and Top 10 Alternative Search Engines of 2008
• Mozilla, the makers of Firefox, added us to their list of “Recommended” add-ons

As such, we are today officially removing the “Beta” label from our website and application.

The path for most every start-up is long and difficult and we continue to have a considerable amount of road before us. While we will inevitably face many new challenges, we’d like to take this opportunity to express our gratitude to some of the people who’ve helped us to get this far:

• Mozilla for facilitating the development of add-ons as well as providing tremendous feedback and invaluable assistance in distributing the application
• Microsoft for prominently including our application in the IE Gallery and for SilkRoad
• Yahoo! for BOSS
• All of the companies who’ve built amazing search engines upon which we build
• Our investors for putting their confidence in our vision
• All of the journalists and bloggers who took the time and made the effort to evaluate our software
• All of the people who’ve downloaded our application and the thousands of people who’ve generously offered their encouragement, feedback, praise, criticisms, comments and advice

While it’s fun to look back at our accomplishments, we’re now focused on the year ahead and are very excited about the future.

Personalization is often been divided into two categories, implicit and explicit, the respective merits of which have been debated. Implicit personalization is based on preferences inferred from behavioral information. While this doesn’t require any effort on the part of the user, making accurate determinations of intent can be challenging. On the other hand, explicit personalization is driven by direct indication from the user. The intent is often clearer, however, the onus is on the user to make the effort to specify preferences.

Since April 2006, Surf Canyon has been looking at personalization from a different perspective: real-time vs. long-term. Real-time personalization alters the user experience instantly as behavioral signals are collected. While determining intent “on the fly” is challenging given the requirement for speed and the paucity of data, the signals are typically very strong. Long-term personalization, by contrast, relies on the accumulation of considerable user data over a significant amount of time. Determining the user’s “at the moment” intent can be difficult given how quickly the signals decay and how often users change context.

Nevertheless, as indicated here, all of these options are currently available to the internet searcher:

Surf Canyon v1 – Our flagship product introduced real-time implicit personalization for search. By observing the actions of the user as the search is taking place, the application helps people find information by re-ranking the results instantly, effectively transforming the search page from a static list of links to a dynamic set of results that “work with” the user.

Google Personalization – By observing the search pattern and click history of the user over an extended period of time, Google builds a profile of the user’s long-term interests which are then used to personalize the results for future searches. The first post on this blog analyzed some of the benefits and shortcomings of this technology.

Surf Canyon v2 – Launched in December ’08, v2 of Surf Canyon introduced my.SurfCanyon.com, enabling users, should they so desire, to explicitly indicate sources of content they prefer as well as those that they dislike. These long-term preferences are then taken into account to, once again, personalize search results to the user’s benefit.

Google SearchWiki – Launched in November ’08, SeachWiki offers controls to enable users to manually manipulate search results. By clicking a button, results may immediately be “promoted” to the top of the search page or deleted altogether. The next time the user runs the same search, the user’s personal modifications will be displayed.

Different users will have varying appreciations for the costs and benefits associated with each of these technologies, however, they are all compatible and, to a large extent, compliment each other. Today’s internet searchers are therefore free to use all of them, some of them or none of them, as they prefer.

Update (1/15/09) – Charles Knight at AltSearchEngines ran this post under their “Guest Authors” series.

We are naturally thrilled to be sharing a such prominent column with a company we esteem so highly. Here are some excerpts:

“[SearchWiki and Surf Canyon] don’t necessarily compete against each other; in fact, they can be used in tandem. But after initially entering a search query, SearchWiki requires additional work on the part of the user that many people may not want to do. Surf Canyon works automatically as you go, sorting results according to real-time user behavior.

If you like the idea of more personalized Web searches but would like to use other search engines or don’t want to do extra work, you might like Surf Canyon.

Google’s SearchWiki is asking users to do extra work, which may not be practical for many users. But if you do use it, this tool’s personalized, saved results could be a real boon. Surf Canyon worked well for me with multiple search engines, retrieving data from result pages I likely wouldn’t have opened. Either way, your days of futile Web searching are numbered.”

The real-time implicit personalization offered by Surf Canyon and the real-time explicit personalization offered by SearchWiki are both compatible and complimentary. You don’t have to pick one or the other when you can have both!

Update (1/14/09) – Katherine Boehret added a video. Here is what she had to say at the 3m20s mark:

“At the end of the day, Google SearchWiki makes you do a lot of work. It might be beneficial for some people who want to save their searches, but Surf Canyon works with you as you go, on the fly. It works automatically and doesn’t make you do anything extra other than what you’re already doing.”

Update (2/18/09) – Steven Blinn, PR consultant, talks with David Ward at PRWeek about getting coverage for Surf Canyon in the Wall Street Journal.

One of the most useful techniques for comparing the quality of search engine retrieval functions is the technique of result interleaving, invented by Thorsten Joachims of Cornell University. He first introduced the idea in a 2002 paper and has since recently expanded on the idea.

A search engine retrieval function is an algorithm that produces a ranked list of documents given a document collection and a user query. The retrieval function is the secret sauce behind the search engine. It is reported that Google, for instance, considers over 200 different document features when ranking web pages in response to a user query. These features are fed into the retrieval function which tells the web application which links to present and it what order.  In an open collection, such as the World Wide Web, different retrieval functions can produce both different document orderings as well as entirely different sets of documents.

Joachims came up with a very simple test that answers the following question: Given two retrieval functions, which does a search engine user prefer? His idea was that one can interleave the results of the two retrieval functions in an unbiased fashion and then count the user clicks on the links contributed by each retrieval function. The better retrieval function is the one that gets the most clicks.

For instance, assume that we have four documents (A, B, C, and D) that are relevant to a given query. According to the first retrieval function, r1, they should be ordered C-A-D-B. According to the second retrieval function, r2, they should be ordered D-A-C-B. The interleaved order of presentation would be D-C-A-B half the time and C-D-A-B half the time. We need to assure that each retrieval function gets to determine the document in the top spot half the time in order to have an unbiased test.  We would then show these document lists to many users as they conduct searches for this query. If we found that there were more user clicks on document C compared to document D, we can state that users prefer retrieval function r1. We can repeat this test for more documents and more queries, but by simply counting the clicks on documents contributed by each retrieval function we can determine an absolute user preference.

Surf Canyon implemented this test to compare our retrieval function, which employs real-time personalization based on implicit relevance feedback, with Google’s retrieval function. We always show the user the top 10 Google results, even with our application installed, so our interleaving test was only done when users asked for a second page of search results. In those cases, the results presented to the user would be a mixture of results 11 through 15 from Google and the most highly ranked personalized results from Surf Canyon’s retrieval function.

The figure below shows the ratio of link clicks on Surf Canyon results compared to Google results as a function of the number of search results selected by the searcher before preceding to the second page of results. A ratio less than 1.0 would indicate that they prefer un-personalized Google results, whereas a ratio greater than 1.0 would indicate a preference for Surf Canyon’s retrieval algorithm. A ratio of 1.0 would indicate no user preference. Note that the users do not know if they are selecting personalized or un-personalized links. The result is a very clear preference for Surf Canyon’s retrieval algorithm – users are 30-40% more likely to select Surf Canyon links.

We looked at this quantity versus the number of search results selected because the user’s interactions with the search page are what we use to personalize the results. The more the user selects, the more confident we are about the user’s true intent. Interestingly, users also prefer Surf Canyon results, by a significant margin, even when they skip the top 10 Google links entirely. When a user skips a link, we generally assume that the document is not what the user wanted. If the user skips all 10 links, we assume that the search engine misinterpreted the user intent and we start looking for different content that is not represented in the top 10 links.

Even though only 10% of searchers ever venture beyond page 1, we consider a 30-40% improvement in page 2 click-through rates to be significant. Quantitatively measuring the value delivered by real-time implicit personalization to page 1 results is, unfortunately, considerably more difficult. Nevertheless, be believe that these page 2 results are indicative of the value that real-time implicit personalization can delivery to page 1 results as well.

Update (7/15/09) – Our research paper, “Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback,” was selected for oral presentation at SIGIR ’09.

Update (12/18/09) – Our research paper was published by SIGIR.

Quantifying the “web search experience” is, however, very challenging. Nevertheless, search engines are constantly running small (and large) experiments to test how changes affect the user search experience. These experiments, which often use something called “A/B” or “bucket” testing, entail exposing a small, randomly selected set of users to the new features or changes and then comparing their behavior to the behavior of users on the baseline search site. Depending on the feature being tested, different user behavior signals are used to judge user satisfaction with the changes.

Since our technology radically changes the nature of the search results page, evaluating the application is particularly difficult. Once a user installs the Surf Canyon application, the search engine results page becomes dynamic and personalized to each user. Users who install Surf Canyon expect to get the Surf Canyon technology, so a traditional bucket test is not possible. (If we were to have a control sample of users who installed a special version of Surf Canyon that did not personalize their search results, those users would be perplexed and would probably uninstall the product.)

However, we performed a thorough evaluation of the technology using some traditional evaluation metrics from the Information Retrieval community as well as some new evaluation techniques that we invented ourselves. These evaluations are documented in a research paper which we recently drafted. In a subsequent blog post, we will detail our evaluation methodologies and the conclusions of these studies.

A good presentation should naturally start with the conclusions, so we reveal here in advance the conclusion of our studies so far: real-time personalization works.

Update (12/1/08) – Continued in Part 2.

Update (7/15/09) – Our research paper, “Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback,” was selected for oral presentation at SIGIR ’09.

Update (12/18/09) – Our research paper was published by SIGIR.

In their paper entitled “Beyond the Commons: Investigating the Value of Personalizing Web Search,” Teevan et al. make the observation that:

“Web queries are very short, and it is unlikely that a two- or three-word query can unambiguously describe a user’s informational goal.”

Ambiguous intent combined with an exploding quantity of content on the internet makes it increasingly difficult to put all of the relevant results on page one while simultaneously eliminating those that are not pertinent.

Very few people venture past the first page of search results to find what they want, so returning hundreds of thousands or even millions of results is of little value to the user. (You cannot look past the first 1000 even if you wanted to.) Even if the user is particularly motivated, the process of digging through page after page of results is nothing short of tedious, which is the reason users will either quickly turn to reformulating their query or abandoning the search.

The problem is too many results!

The solution to the conundrum is to have a greater understanding of the user’s intent in order to more precisely focus the results. One way to achieve this is to get the user to explicitly specify intent by entering more keywords, although getting people to change behavior is not easy. Another way to achieve this is to implicitly infer intent through the type of long-term personalization offered by Google, although this too has a number shortcomings.

The most effective way to resolve this issue is to implicitly infer intent from real-time behavior signals and then immediately re-rank the results, through the use of instantaneous relevancy calculations, so that the most pertinent results are moved to the top while the less relevant are suppressed. Surf Canyon‘s Discovery for Search™ is such a solution. Disambiguating intent “on the fly” not only enables users to continue searching with their current behavior, but no search histories or profiles are required. Furthermore, the signals are strong so that the results can be reordered dramatically and the user can actually “see” the process working, creating a more encouraging and perhaps entertaining search experience.

OAKLAND, CALIF. February 19, 2008 – Surf Canyon today announced the official release of its Discovery Engine for Search™ at www.SurfCanyon.com. The browser extension for Internet Explorer and Firefox, designed to disambiguate user queries in real time, significantly accelerates the search process.

As the volume of content on the internet expands exponentially, so has the challenge of sifting through mountains of search results in order to pinpoint the most relevant. By using Surf Canyon’s semantic real-time implicit personalization technology while searching on Google, Yahoo! or MSN, searchers are able to more quickly and easily find relevant information buried within the usually overwhelming quantity of results. Rather than leaving users to manually filter through static sets of links, this innovative application uses real-time behavior signals to calculate “instantaneous relevancies” within a result set in order to dynamically pull forward the most pertinent results while pushing back those less germane. The technology is similarly suited for targeting sponsored links, benefiting both users and advertisers.

“Like many innovations, the concept was born out of frustration,” said Mark Cramer, Surf Canyon CEO. “While existing search technology is certainly more than adequate for many needs, when looking for something relatively obscure I often felt somewhat alone digging through all the results and, as such, decided that I could benefit from a search technology that would work with me during the process. With our product, search is no longer a static lecture being pushed to the user, but a participatory conversation that yields better and faster results.”

“Surf Canyon significantly improves my search experience and the deeper results it delivers are spot on virtually every time,” said Carla Thompson, senior analyst with The Guidewire Group. “I also really like that it functions within my existing search; once installed, I’m able to enjoy the advantages without having to adjust my habits.”

About Surf Canyon
Surf Canyon develops real-time implicit personalization technology for internet search. Its patents-pending Discovery Engine for Search™ transforms static result pages into dynamic knowledge resources, enabling users to more quickly and easily find pertinent information buried among all the irrelevant results, significantly accelerating the search process.
Surf Canyon has headquarters in Oakland, California and is recognized by DEMO.com as an “Innovator to Watch.”

This is particularly problematic since, as discussed previously, it’s increasingly difficult, if not impossible, to map two- and three-word queries to the exploding content on the internet. It is thus increasing unlikely that the top 10 will contain the information sought without some form of additional input from the user, either explicitly (reformulation suggestions or clusters) or implicitly (personalization).

But what if there was a search technology that was able to dig into the results and fetch the relevant information for you?

Surf Canyon’s Discovery for Search™ technology is designed to find the “hidden treasures” in the subsequent search results, and Beta testing has confirmed this. The graph below compares the initial rank of a result (x-axis) with the frequency with which it is clicked (y-axis). The AOL data is from 2006 and the Surf Canyon data is from the most recent 10,000 queries. The result is that Surf Canyon doubles (17% vs. 9%) the users’ ability to find relevant information buried in the search results.

• Explicit personalization, also referred to as “customization,” is the process through which users shape their experiences by actively selecting preferences or responding to questions. Within search, voting for, rating or deleting particular results, or any adjustment on a ‘Preferences’ or ‘Options’ page, are obvious examples of explicit personalization. Less obvious, however, are things such as presenting suggested search terms or similar queries to the user, such as with Yahoo! Search Assist or Google Suggest. Clustering (the process of suggesting possible query reformulations through the analysis of similarities between documents in the result set) is also a form of explicit personalization. In all of these cases the onus is upon to user to make a determination as to the path forward and then actively select an option.

• Implicit personalization is the process through which the users’ experiences are shaped by inferences of intent generated through the careful observation of behavior. Google’s Personalized Search, which delivers a modified set of results based on past search activity, is a clear example. Leveraging behavioral analysis or social relationships to target anything from advertising to search results to any other sort of content is also implicit personalization.

One is not better than the other. They are simply different and, to a great extent, compatible. Explicit personalization by itself, however, suffers from a number of limitations that can be alleviated through the addition of implicit personalization:

1. Explicit personalization, by definition, requires effort on the part of the user and many people merely will not bother. It seems that users want to enter their queries and then simply go straight to the results, scanning from top to bottom.
2. Even if users are willing to make the effort, they don’t necessarily know with enough precision what it is that they want. Furthermore, in some cases they might not yet possess enough knowledge to comprehend the clusters or suggestions. While clusters may sometimes help with reformulation, since it is often easier to recognize terms rather than recall them, search is frequently a process of discovery, with each result leading to better understanding of the subject matter and a greater ability to express intent. Knowing all of this a priori is not always possible.
3. Explicit personalization through a classification or clustering of search results may also create additional work if the user’s intent spans multiple contexts or sub-contexts. Selecting a cluster, for example, will enable the user to ‘zoom in’ on a particular sub-context. However, if the sub-context is not a precise match, or if the nature of the search requires a body of information that covers multiple sub-contexts, then the user will have to expend additional effort going up and down the classification tree.
4. Even if users think they know what they want, they still might be mistaken. Or worse, either intentionally or not, they are not always honest with themselves. Based on the ability to observe user behavior on all of Yahoo!’s properties, Dr. Usama Fayyad stated, “I know more about your intent than any 1000 keywords you could type,” which of course alludes to the difficulty of both accurately knowing and then expressing one’s own intent.

It is for these reasons that implicit personalization is an important compliment to explicit personalization. One could compare it to the focus adjustments of a telescope, with the course adjustment being explicit personalization and the fine adjustment being implicit personalization; you could certainly make out some celestial objects with only the course adjustment, but you need the fine adjustment to really discern the details.

Step 1: Enter you query at search.SurfCanyon.com or, if you like, download our add-on and then go to your favorite search engine (Google, Yahoo! or Bing), just like you always do, and enter your keywords.

Our example will use a search for the term “dolphins” since it is often used to illustrate one of the classic problems with information retrieval. No matter how much information is gathered about the subject, the individual, or all the individuals who have ever searched for “dolphins”, it is impossible to know a priori whether or not the searcher’s intent is related to the animal or to the football team. While search engines may suggest alternate queries or reformulations through the use of keyword suggestions or clusters, the search results produced will always be a mix of the animal and the football team.

Step 2: Review the results and select the one that corresponds to your interests. If it’s the Miami Dolphins that you like, click #2: MiamiDolphins.com. If your search is satisfied, great! If not, return to the search results and Surf Canyon will provide you with some “Recommendations”. You may, if you like, also get the “instant” Recommendations by simply clicking the bull’s eye next to the particular link of your choice.

Recommendations are dug out of the search results based on the Discovery Engine’s inference of our real-time intent. Put another way, Surf Canyon figures out what you want and then goes past the first page of the results and fetches it for you.

Step 3: The process is cumulative, so continue clicking on links and bull’s eyes that are of interest to you. The longer you work on the result set, the better Surf Canyon understands your intent and the more accurate are the Recommendations. The Recommendations can nest to three levels, enabling you to “drill down” on not just your high-level intent, but your sub-intents as well, such as “Miami Dolphins news.”

Click on “More Results” and you’ll get a second page of results automatically customized to your intent. In this case, they are naturally all related to the Miami Dolphins football team and are geared towards news.

Keep going and Surf Canyon will keep working to help you find what you need.

That’s Discovery!

Lastly, don’t worry about ‘mistakes’: clicking the links that are perhaps not of interest. Surf Canyon is smart enough to handle it.