# robots.txt file # Version 2.2 - 10/26/06 # for http://www.carolinasmedicalcenter.org/ # BASIC robots.txt file to reduce crawler load. # Must be named "robots.txt" and be in the root folder. # Only Google, MSN, Yahoo, and Google-Image are allowed. # Certain files excluded. # Yahoo-Slurp held back to 1 download every 45 seconds # Throttle Yahoo Slurp User-agent: Slurp Crawl-delay: 45 Disallow: Disallow: /blank Disallow: /BLANK Disallow: /print Disallow: /PRINT Disallow: /secure Disallow: /images Disallow: /admin Disallow: /cfide Disallow: /search Disallow: /scripts/ Disallow: /database/ Disallow: /edit/ Disallow: /ErrorTemplates/ # Googlebot Yahoo MSN User-agent: Googlebot User-agent: msnbot Disallow: Disallow: /blank Disallow: /BLANK Disallow: /print Disallow: /PRINT Disallow: /secure Disallow: /images Disallow: /admin Disallow: /cfide Disallow: /search Disallow: /mapquest/ Disallow: /scripts/ Disallow: /database/ Disallow: /edit/ Disallow: /ErrorTemplates/ # Google Mini User-agent: gsa-crawler Disallow: Disallow: /blank Disallow: /BLANK Disallow: /print Disallow: /PRINT Disallow: /secure Disallow: /images Disallow: /admin Disallow: /cfide Disallow: /search Disallow: /scripts/ Disallow: /mapquest/ Disallow: /database/ Disallow: /edit/ Disallow: /ErrorTemplates/ # Google Images User-agent: Googlebot-Image Disallow: Disallow: /admin # Exclude all other robots User-agent: * Disallow: /