OCLC Researchers estimate 1-percent decrease in size of public Web

Although the public Web has more than doubled in size since 1998, the speed of its growth steadily declined over that period. The trend culminated in a slight contraction in size over the past year, according to the latest results from OCLC Research's annual survey of the Web.

1-percent decrease in past year

Estimates from the Web Characterization Project (WCP) place the size of the public Web at 3,080,000 sites in June 2002, a 1-percent decrease over 3,119,000 sites in 2001. A public Web site is defined as a distinct location on the Internet, offering content accessible without restrictions such as special authorization or fee payment.

About the public Web

A "public Web site" provides free, unrestricted access to a significant portion of its content. In other words, the content of the site is freely available for access by the general community of Web users. The set of all public Web sites is called "the public Web."

Maturing technology

"The Web is maturing as a technology," said Ed O'Neill, OCLC consulting research scientist and manager of the WCP. "Individuals, organizations, and businesses are no longer engaged in the rush to 'get on the Web' witnessed five to ten years ago. Most people who want to establish a Web presence have already done so. The focus now is on refining and expanding that presence, and figuring out how to utilize Web technologies effectively to fulfill mission-critical objectives."

In the last five years, OCLC researchers calculate that the public Web has more than doubled in size, increasing from 1,457,000 sites in 1998 to just over three million in 2002. Yet the annual growth rate of the public Web has been slowing steadily over that same period. Between 1998 and 1999, the public Web grew by 53 percent, but increased by only 32 percent the following year. Between 2000 and 2001 the growth rate was just 6 percent, while results for 2001-2002 indicate that the public Web shrank by 1 percent.

Public sites may be fewer but bigger

The public Web is a particularly important part of the Web as a whole, since it is within this portion that the average Web user typically navigates.

The apparent shrinking of the public Web over the last year does not necessarily mean that less content is available. Results from the WCP survey suggest that while the number of sites may have plateaued, the size of Web sites is increasing. For example, the average number of Web pages contained within a public site in 2002 was 441, compared to 413 in 2001. The amount of so-called "dark matter" on the Web—information in databases and other formats not accessible by traditional web-crawling techniques—is said to be large and growing. The use of virtual hosting technologies permits the grouping of multiple "virtual sites" at a single Internet location.

Adult sites increase in number and proportion

The WCP's 2002 survey indicates that adult sites—those offering sexually explicit content—constitute approximately 3 percent of the public Web, or a little more than 100,000 sites. This represents a small increase over the previous year. The United States Supreme Court is reviewing legislation that would tie Federal funds supporting Internet access in public libraries to the installation of filtering software that blocks access to sites with content that could be considered objectionable, such as adult sites.

More information about the OCLC Web Characterization Project is available at the project web site http://wcp.oclc.org. A report discussing the results of the project's five surveys to date (1998 - 2002) will be published in the April 2003 issue of D-Lib Magazine.

For more information:

Ed O'Neill
Research Scientist
OCLC Research

Bob Bolander
Communications & Programs Manager
OCLC Research