Hidden Web / Invisible Web / Deep Web

The Invisible or Hidden Web consists of
    * 3/4 of the information on the internet  
    * Information not indexed by major search engines
    * Usually more current information
    * Business intranets
    * Academic single source institutions
    * Content-rich DATABASES!

Five types of Sites
    * Visible - crawled and indexed often by search engines
    * Opaque - not linked to other sites (homepage with no links)
    * Private - password protected, secure server (https://)
    * Proprietary - registration required (fee or pay for view)
    * Truly invisible - unsearchable file formats and databases

                                         Bergman, Michael K. "The Deep Web: Surfacing Hidden Value." Journal of Electronic Publishing, August 2001.

Deep Web Facts
    * More in-depth information than surface sites
    * Quality content 1,000 - 2,000 times greater than that of surface sites
    * Academic disciplines highly represented
    * Most content resides in subject-specific databases
    * 95% is publicly accessible information - not subject to fees or subscriptions

How to tell if a Web Site is Invisible - Examine the URL
    * Direct URLs point to specific web pages. Crawlers can follow these pages
    * Indirect URLs
            * Do not point to specific web pages
            * Contain server executed script information
            * Contain symbols (?, &, %. $. +) or words (cgi-bin or javascript)

Is a site indexed?
    1. Pick a URL.
    2. Place in the Google search box with the word info: before the URL with no spaces.
    3. If it displays, it is indexed by Google; if not, then it is not indexed.

To find Hidden Web Databases
    1. Use search engines and add the word "database" to search terms.
    2. Use ADVANCE search features
                AlltheWeb  http://www.alltheweb.com/advanced?advanced=1&&q= 
                Google        http://www.google.com/advanced_search?hl=en
    3. Use Specialty search engines
                 Teoma    http://www.teoma.com/
                 IxQuick   http://ixquick.com/ 
                 Surfwax  http://www.surfwax.com 

For better searches
    * Be specific
    * Put the most important and most rare words first
    * Use 6-8 key-words in query to increase efficiency
    * Use phrases when possible to eliminate irrelevant material
    * Use quotation marks around phrase
    * Use more than one search tool for your research
    * Read the instructions or help screens
    * Teachers - use "grade 4" rather than "4th grade" ;  use educator rather than teacher