Posts Tagged ‘Microsoft’

A long time ago, in a galaxy far, far away, I was trained as a research scientist. Statistics, research methodology, hours in the library….all the things chicks dig, right? Although I’ve twice worked in the hallowed halls of academia, I’ve always leaned toward the applied side of science. I can theorize and hypothesize with the best of them, but I like my experiments conducted in the real world and I like to see the results that make a difference.

I also appreciate it when I see clever research. And one of the most clever things I’ve seen recently is the Bing game: Page Hunt. Players are shown a series of web pages and asked to guess a keyword or words that can be used on the Bing search engine to make the site appear in the top 5 search results. Players are given 100 points for guessing a result that is in the #1 result slot, 90 points for #2 and so on. If the keyword(s) do not produce a top 5 result, you are given another chance. The game is timed (an interesting 2 minutes and 58 seconds) and there are a few interesting twists like bonuses for avoiding common terms. It’s an easy game and it’s already produced some interesting results. Why not play a round or two yourself?

According to Ars Technica, Page Hunt was developed by Microsoft Research Interns Chris Quirk and Raman Chandrasekar along with Hao Ma and Abhishek Gupta from The Chinese University of Hong Kong and Georgia Institute of Technology respectively. Before they unleashed the game on the unsuspecting and great unwashed of the web, they piloted the game internally. They found that the length of the page URL was negatively correlated to the ability of the player to correctly achieve a top 5 result. In other words, the shorter the URL, the easier it was to win.

While this is an interesting result, I seriously doubt that this is the real gold mine that is to be found in this research game. One of the things that has stuck with me from my graduate studies was the influential work of Karl Popper. Popper’s theories are a little domplex at times, but in a nutshell, he believed that an experimental result that contradicted a hypothesis was vastly more valuable than a result that confirmed the hypothesis. In other words, a single disconfirmation is worth a thousand confirmations.

What does this have to do with the price of tea in China (or Hong Kong for that matter)? Because I believe that the Page Hunt game’s core value is in using it for the results that players get wrong rather than the things they get right.

What is the value in having players guess a result that already appears in the top 5 results? Are you just confirming what is already working? Sure, it helps show that Bing is aligning well with player expectations. But again, these are all just confimatory results. What will be really interesting to mine are the results where there are clustered patterns of results produced by players that DO NOT appear in the top 5. What is it about a particular page that makes a player suggest a keyword they think best describes the page yet does not produce a top 5 result? That’s the real value here.

If you have ever effectively managed an internal search engine for a web site, this process is probably intuitively obvious to you. One of the best uses of internal search data is to look for keywords that users have entered that produce zero results (also referred to as null sets). Of course, this is typically only interesting if the web site in question actually has content that would be an appropriate “answer” for the null search query. The difference here is that the web site owner can typically “fix” this problem pretty directly. The search engine itself often has tools to allow the manual “promotion” of a page based on a specific query. But a search engine that indexes the entire web doesn’t have this luxury.

The Page Hunt game provides a crowd-sourced solution to this problem. Analyzing the keywords generated by players that do not produce a top 5 result could provide fertile ground for improving the Bing algorithm. It won’t be easy and it would certainly require Page Hunt/Bing to collect more page data than what is shown to players. It struck me that a similar game would be a useful tool for assessing the skills of SEO practitioners. If you allowed the person being tested to evaluate the page source, I believe that any SEO’er worth his or her salt should be able to deduce a top 5 search keyphrase at or above 80% of the time. It could be a more objective way to evaluate skill sets. Of course, the test itself would have to be normed first and it would require a large set of sites to avoid cheating. But it is still an interesting idea.

But I digress….

Using crowd-sourcing does carry some inherent risks. A motivated community could “game the game.” For example, let’s say the 4chan community decided to target the game and collectively distort the results of the pages that they are displayed during the game. They could all input the same irrelevant keyword for the same page. So, they could give the answer “Cleveland Steamer” for the page of a political candidate, for example. If enough of them provided the same keyword for the same page, it could carry some weight.

Of course, Microsoft isn’t likely to just automatically accept the results of the Page Hunt game. It’s just a research tool. It’s not a magic bullet answer to improving Bing’s algorithm. Human review is still a big part of the search engine industry. Remember the “miserable failure” Google bomb from a few years back? Ultimately, Google had to hand edit its results to eliminate the results of this prank. So I doubt there is any chance that we’ll see any “Cleveland Steamering” of Bing any time soon. But it would be amusing.

The only thing I can think of to improve the Page Hunt game is to increase the incentives to play. Right now, you can really only compete against yourself. A simple upgrade would be to have a community high scores page that encouraged players to compete against the community. Never under-estimate the human ego. Of course, they could up the odds with just a tiny prize incentive, like a t-shirt for high scorers. You wouldn’t believe the weird things people will do to win a cruddy little prize. Or maybe you would.

Technorati Tags: Bing, Google Bomb, Microsoft, Page Hunt, search engine optimization, Search Marketing, seo