The Internet beyond Google

A simple Google search combs through the more than 1 trillion web pages the search giant has indexed.  While this is a staggering amount of information, it is only the tip of the iceberg.  Typical search engines search only the “surface web,” missing the much larger Web made up of hidden data, called the “Deep Web.”  The Deep Web is orders of magnitude larger than the surface web that most internet users interact with everyday.  Deep Web is largely made up of information stored in databases, such as shopping catalogues, flight schedules, financial information, and museum archives.  This information is largely inaccessible to search engines.

While most of Deep Web is innocuous, some web designers and believe that the anonymity of Deep Web best reflects the original free speech and libertarian impulses of the internet.  In 2000, Ian Clarke released software called “Freenet,” which allows people to chat online, set up or read a website, and share files with complete anonymity.  Freenet contains sites about terrorism, child pornography, and extreme political views, in addition to more usual blogs and websites.  Andy Beckett, a reporter for The Guardian, summed up Freenet: “There is all the teeming life of the everyday internet, but rendered a little stranger and more intense.”  Clarke admits that these illicit activities take place in Freenet, but points out that they take place elsewhere on the web as well.  He firmly believes that modifying Freenet would be the end of Freenet.

Freenet is just one of many hidden networks that populate Deep Web.  The well-known crime syndicate the Russian Business Network, known as RBN, takes advantage of the unused or discarded web addresses found in the Deep Web.  RBN uses those websites for minutes at a time to send out millions of spam emails.  They also rent old websites to criminals who perpetrate identity theft, child pornography, and release computer viruses.

However, these dark networks might not be able to stay hidden forever.  Search engines are constantly looking for ways to access the information contained in Deep Web.  The commercialization of the internet is making it more transparent.  Anand Rajaraman, of the search engine Kosmix, likens the future internet to a panopticon, where everybody is observed and can observe everybody else.  It is unclear if the dredging of Deep Web and uncovering hidden networks like Freenet is beneficial in the long-term for people who prefer that the internet stay private and anonymous.  However, in order to monitor the illegal activities of the Deep Web, transparency might be the best and only answer.

References
http://www.guardian.co.uk/technology/2009/nov/26/dark-side-internet-freenet
http://www.nytimes.com/2009/02/23/technology/internet/23search.html?_r=1&th&emc=th

Faith Fried

Intern, CSIS Technology and Public Policy