The battle for search supremacy is hardly over. In fact, search is in its early days and poised at the cusp of several major leaps forward. What impact will they have on libraries? And how can libraries benefit?

By Tom Storey

“As any engineer in the search field loves to tell you, search is at best 5 percent solved—we’re not even into the double digits of its potential.” John Battelle

Ten years ago, a small Mountain View, California company rolled out a plain search box and a new computer algorithm to help people find information. In the month of January 2007, more than 3.9 billion searches were performed on Google, which held a commanding 72 percent of the global search market, according to Nielsen//NetRatings and Enquisite Software.

In addition, more than 255 million people worldwide used a search engine—81 percent of the global Internet population—and the audience for search grew more than 10 percent, outpacing the growth of the Internet itself.

Clearly, it’s fair to say that Google, along with a host of other Internet search engines, have fundamentally changed the relationship between humanity and knowledge, says John Battelle, entrepreneur, journalist, professor and author of The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture. “Search is the new interface to knowledge,” he says. “All of a sudden, the world is quite literally at your feet—or rather your fingertips.”

So powerful is Google’s impact that in 2006, Merriam-Webster’s Collegiate Dictionary and the Oxford English Dictionary added the verb “google” to their hardbound editions. The company also was named one of the top global brands by Business Week and brandchannel.com. To an army of information seekers, Google means search.

But the battle for search supremacy is hardly over. In fact, search is in its early days. And several experts believe that search stands today at the threshold of a new era. Opportunities exist for other companies and organizations, including libraries, to grab a portion of this rapidly growing phenomenon.

“I totally believe that the search landscape can change,” says Roy Tennant, former User Services Architect, California Digital Library, University of California (Tennant recently accepted a position as Senior Programs Officer with the OCLC Programs and Research Division). “In fact, since change is the only constant, the question is how will it change? I think it will change by people waking up to the fact that the lowest common denominator searching is fine for some things but not for others.”

Battelle says that when it comes to search, the most interesting stuff is yet to come. “As any engineer in the search field loves to tell you, search is at best 5 percent solved—we’re not even into the double digits of its potential.”

Don Dodge, Director of Business Development for Microsoft’s Emerging Business Team and author of the blog The Next Big Thing, adds that it is important to remember that when Google was emerging, there were already huge dominant players in search. “It didn’t look promising for a start-up in the search business. Personalized portals were the rage. AOL, Yahoo and MSN actually outsourced their search services. Search was simply one of many services, and not a good one at that because it didn’t keep users on your site.”

Today’s search explosion

Experts believe that search stands today at the threshold of a new era. Opportunities exist for other companies and organizations, including libraries, to grab a portion of this rapidly growing phenomenon.

Starting with Archie and Veronica in the early 1990s, search has been evolving from the domain of academics and technologists to mainstream culture. WWW Wander, WebCrawler, Lycos, Excite, AltaVista and Yahoo, which introduced new and exciting features and functionality, helped search gather momentum and move closer toward critical user mass.

But it wasn’t until 1998 when Google came along and reinvented search that the world beat a path to Internet search engines, particularly Google. Google’s breakthrough technology for page rank was far superior to keyword matching and index hierarchy used by early Internet search services, and it helped establish a new level of relevance to the Web as a viable part of the information landscape. “Google did something that few companies, institutions or people have managed to do—provide a service that dramatically increases millions of people’s ability to connect to useful information,” Battelle says.

Tennant agrees. “They effectively solved a problem that was meaningful to people. They built a robust Web-crawling infrastructure and came up with a useful and unique algorithm for ranking search results that caused the most typically relevant items to sift to the top. Then they made it pay by figuring out the right advertising model for the Internet.”

OCLC research confirms the dominance of search engines as the tool to discover resources. In the OCLC report Perceptions of Libraries and Information Resources, respondents indicated that search engines deliver better quality and quantity of information than library-assisted searching—and at greater speed. Information consumers also trust the information they get from search engines as much as they trust information from libraries.

In addition, over half of respondents indicated that search engines fit their lifestyles compared to only 17 percent who said libraries are a perfect fit. More than 20 percent said libraries do not fit their lifestyles. Of the activities that respondents are doing less since they began using the Internet, watching television was number one and using the library was number two. Reading books was third.

That library resources and librarians add value to information search was not disputed by respondents but the data suggests that the relevancy and lifestyle fit of that value are in question.

Recent independent research verifies OCLC’s findings. In its report The User Revolution, investment firm Piper Jaffray states that in less than 10 years, the Internet has grown from a tool that almost no one used, to one of the most widely used commerce, entertainment and information mediums, rivaling the use of television and perhaps exceeding the use of the library as an information source.

The firm estimates that worldwide search query volume will grow at an annual rate of 23 percent. “Search has changed from a tool for finding Web sites to what we consider the new navigational platform,” the report says. “Search encompasses nearly everything that users want to do online. As such we believe search and search engines have become the new portals.”

How big can search get?

Every day, search continues to evolve and grow in popularity. Today, searchers have access to an estimated 10 billion pages of online text, thanks to Google, Yahoo!, MSN and other search engines. And in the short term, search engines, particularly Google, are forecast to continue to dominate the search market.

“Search engines today tend to ignore metadata. But using metadata will sweeten the indexing and ranking.” Roy Tennant

A new frontier of search is on the horizon, however, ready to disrupt the current environment and present a wide range of new possibilities. The next generation of search will be smarter, more tailored to the individual, embrace video and music and be accessible from any device with a chip. Among the possibilities:

Smart search. Many people refer to this as natural language search or semantic search, where search engines scan indexes and use logic to look for meaning, not just matching keywords. Smart search is based on understanding user intent, and understanding the meaning of information available on various Web documents. In this way search engines could begin to answer questions in everyday language. Although he is skeptical about the practicality of the Semantic Web, Tennant sees the benefit of using rich metadata associated with scholarly works to sharpen search results.

“Search engines today tend to ignore metadata,” says Tennant. “But using metadata will sweeten the indexing and ranking. This is becoming increasingly important as millions of scanned books become available. Without metadata, good luck with providing good search results.”

Personalized search. What are your likes, dislikes? What are your preferences? If a search engine knew, search results could be highly customized and mirror your specific interests. Personalized search would deliver results tailored to who you are, what you are researching and your past search history, Battelle says. “The idea behind personal search is pretty simple: the more an engine knows about you, the more it can weed out irrelevant results,” he says.

Tennant believes there is much experimentation to be done on useful ways of sifting through results.“ If someone can create a search engine that provides more sophisticated ways of narrowing in on what you want from among thousands or millions of items, that will be a tool worth using.”

Domain-specific search. Domain-specific search would focus on one area of knowledge, creating customized search experiences that, because of the domain’s limited corpus and clear relationships between concepts, provide extremely relevant results for searchers. Battelle calls this structured search—the ability to use search as an interface to knowledge in specific subject areas.

Tennant sees possibilities here as well. “Niche search engines will find audiences since they can be tuned to the particular needs of a user community.”

“Mobile devices will become the communication and computing platform of choice within five years. It already is with teenagers.” Don Dodge

Picture, audio and video search. As sound and still and moving images are increasingly digitized and broadband penetration grows, video, audio and image search are on the upswing and quickly gaining prominence with Internet users. Think of the phenomenal growth of iPods, podcasting, Flickr, YouTube, Blinkx, Metacafe.com and searchforvideo.com to name a few. Possible technological breakthroughs that could accelerate the trend are enhanced online playback and improved algorithms in speech recognition and visual analysis.

Mobile search. Mobile search is the convergence of search engines and mobile devices, such as cell phones, PDAs and other pocketsized computing devices. With millions of cell phones and mobile devices worldwide, Dodge believes that mobile search is potentially the largest opportunity in the search landscape. The secret to success, he says, will be user interface design, which means both the ease of entering queries and the clear presentation of search results tailored for mobile devices.

“Mobile devices will become the communication and computing platform of choice within five years,” he says. “It already is with teenagers.”

Dodge also believes that cell phones will lead the way with voice-driven search. “Voice recognition is way better today that it was five years ago, and the phone has a perfect microphone already built in. Why not allow the user to enter commands and search queries via voice rather than text entry? Why not return the results via computer-generated voice rather than text? Why not allow the user to get the results both ways? This will be big!”

Social search. Social search determines the relevance of search results by considering the interactions or contributions of users. One of the fastest-growing social search techniques is tagging, a grassroots phenomenon whereby users label Web sites with descriptive tags, building a network of knowledge dubbed folksonomy—a taxonomy of knowledge organized by ordinary folk. Battelle calls this“finding that which your community finds interesting.”

Search everywhere

Beyond the frontiers of the next generation of search stands yet another world with unimagined network devices and objects. Within two decades, Battelle says, nearly everything of value will be tagged with tiny computing devices and incorporate network-aware search. RFID chips will enable computers to automatically recognize and identify everyday objects and trace, track, monitor and trigger events, and perform actions on those objects.

“Think about that—Google your dog, your kid, your luggage, your purse, your cell phone, your car,” Battelle says. “The list quickly stretches toward the infinite. Anywhere there might be a chip, there can and most likely will be search.”

What does it mean for libraries?

The dominance of search engines for discovering resources is unlikely to diminish substantially in the future, but libraries can increase their participation in the online world and the search engine revolution.

“Search encompasses nearly everything that users want to do online. As such we believe search and search engines have become the new portals.” Piper Jaffray

OCLC is building a platform, WorldCat.org, to make it easy for libraries, collectively and individually, to deliver their services to the network and build a unified, high-value consumer presence on the Web. WorldCat.org integrates library content and services with Web search engines, Internet booksellers, online bibliographies and commercial publishers. It also provides a permanent Web page dedicated solely to searching the world’s libraries and a downloadable search box that anyone can download to a blog or Web site.

With WorldCat.org, Web searchers discover library resources in their results lists and move from the Web to their local libraries. And OCLC members become more visible and their collections and eServices more accessible from sites where many people start their search for information.

Tennant notes that despite the fact people start at search engines, that does not mean they end there. “We find that although many of our students begin with Google, they realize the benefit of commercial databases and they will often end there. What libraries need to do is to create search services tailored to the particular needs of our clientele. If we understand their needs well enough, and do a good enough job in meeting those needs, they will come. All we need to do is effectively solve a problem that people care about.”

Battelle says that, simply put, librarians need to become the experts in using the tools we all use to gather information.“Experts will always be in demand by the public. This means become experts in search.”

The importance of understanding search and its cultural ramifications cannot be understated, he says. “Search is no longer a stand-alone application, a useful but impersonal tool for finding something on a new medium called the World Wide Web. Increasingly search is our mechanism for how we understand ourselves, our world and our place within it. It’s how we navigate the one infinite resource that drives human culture: knowledge. Perfect search—every single possible bit of information at our fingertips, perfectly contextualized, perfectly personalized—may never be realized. But the journey to find out if it just might be is certainly going to be fun.”

Libraries will be a part of the journey.

Updates | The Internet of Things