Behind the scenes with universal search

Posted by David Bailey, Tech Lead, and Johanna Wright, Product Manager, universal search

So when we were asked to make the vision Marissa describes about universal search into a reality, we admit we were a little daunted. Googlers had tried before to do this without success -- several times. Finding the best answer across multiple content types is a well-known hard problem in the search field. Besides that, we wondered if we had become too big a company to pull off a project this complex.

Here's the challenge in a nutshell: Until now, we've only been able to show news, books, local and other such results at the top of the page, like this example for [trends in education]. But it's a tall order to earn placement at the top of our search results, so plenty often we end up not showing these kinds of results even when they might be useful. If only we could smartly place such results elsewhere on the page when they don't quite deserve the top, we could share the benefits of these great Google features with people much more often.

One challenge was being able to regularly search through all of the additional content types to find relevant results. After all, you don't know if there might be a minor news story or an obscure book relevant to your query unless you go and check. But Google's massive compute cluster -- and much effort by our infrastructure experts -- gave us a leg up on that one, and we can now search these disparate types of information about as efficiently as we search our massive index of web pages. We may have melted down a data center or two along the way, but then bugs are part of life in this business!

The next challenge was deciding when and where such results should blend in. Fortunately we have some of the world's experts on ranking, and have been able to apply the lessons learned on web search to ensure that we show news only for newsworthy queries, scanned books only when there aren't better web results, etc. It can be tricky. As we learned the hard way, just because everyone under the sun is writing about Anna Nicole Smith doesn't mean news about her should show up for the search [baby names].

Lastly, we faced the challenge of the user interface you see on the screen -- the UI. The new UI for these results is subtle, but this is one reason why the project is fun for our designers and usability experts: they get to focus on creating a simple experience for you. For example, with news results they designed a compact look for the result that includes helpful items like an image and a date, but is limited to just the most salient information. Or take our book search results, which call out the author and number of pages in the book. (Of course, we learned that sometimes you don't even need to design a user interface. In one early usability study, shortly after Barry Bonds broke Babe Ruth's home run record, we asked people "how many home runs has Barry Bonds hit?" hoping they would type [barry bonds] into the search box. Instead, each and every one simply blurted out "715".)

We also called on experts from each of our feature areas such as News and Local, and were delighted to find our startup mindset is alive and well. Folks from all over found spare time and pitched in to get us to the finish line. There were many nights when we went to bed knowing that plenty of the team's IM status still reported they were online.

And after all this elbow grease, finally we have something that works. What does it mean for you?

Although it's just a beginning, this first pass of universal search focuses on video, news, local and books. Now you'll be able to get more information Google knows about directly from within the search results. You won't have to know about specialized areas of content. If you're looking for the [atkins southwestern pork fajitas] recipe, we can now link you right to that page in the book. Or if, like me, you've been busy these past few days and have not caught up with your Tivo, don't type [sopranos] into Google, because our news result will be a giant spoiler. The search for [rachmaninoff concerto 3] includes a video of Vladimir Horowitz performing this piece (scroll down to see it), and [Animator vs. Animation 2] is pretty cool as well. (And as Johanna notes: I was delighted to see that when querying for my son's name a video showed up too.)

This is just the tip of the iceberg in making Google results more comprehensive and useful. It has involved launching a number of new systems that will make it much easier for us to continue making improvements so you get the most relevant information from our varied content areas. We hope you like it. And finally, we're especially happy to know that Google is still very much a place where we can get big things done!