jCay.org

last scripts

popular scripts

top rated

search
 
search:

Advertisment

robots.spider(tm)

DeEnFrSp
Hits: 880

robots.spider(tm)

Rating 0.0 0.0
robots.spider(tm) reads the robots.txt file, then spiders the site (depth is adjustable in the downloaded code), to check for references to entries in the robots.txt file. robots.spider(tm) can help you avoid disclosing the names of directories you don't want indexed, that aren't referenced in your site code. It is an open source (PHP) script, you can run it for free off the demo link, or download the code, adjust the depth, and use it to check your sites from a different server.


Platform(s): Linux
Date: Oct, 09 2006
Author: Betsy A. Gamrat, http://robots-wizard.com

Ratings

Number of Ratings: 100 Votes
5     
15%
4
85%
3
0%
2
0%
1
0%
Visitor Voting Booth: Rate It!