People who viewed this page also visited
LinkChecker
Smart Referrer
Pyracash
Form Mail: Email Form Processor Pro
Form Mail: Email Form Processor Pro
SEO Toolkit
Free Perl Guestbook
Cisco Network Discovery Script
MVW Countdown
Robots.txt parser
Description
Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).
Visits : 118
Google Page Rank :
Requirements
| Date Included | 2006-03-20 13:18:00 |
| Date Modified | 2006-03-20 13:18:00 |
| URL | http://www.hockeytrader.nl/content/view/3/2/ |
| Download URL | |
| Version | 1 |
| Platform | linux, windows, freebsd, |
| Filesize | KB |
| Price | Free |
| Author | Toon Goedhart |
If you have used Robots.txt parser or tried the demos, you can go ahead and rate it.
Comments for Robots.txt parser
Previous visitors and users reviewed Robots.txt parser and their observations and experience has been written below by them.
Commenting and Review Guidelines
Member Reviews are intended to provide valuable information regarding a listing to other people considering using the resource. As such, the following guidelines have been established and are utilized when validating reviews.
No review shall be validated if:
the contents are more than 600 characters;
any field contains any profane or obscene language.
it suggests or contains information about an alternate script (no suggestions are allowed);
it contains advertising of an alternate program or website;
it contains HTML or other programming language code in it;
it is vague or generic in nature (i.e. "This script is awesome!" does not contain valuable information regarding the program and thus will not be validated);
it contains a URL to any website;
is written in any language except for English;
it appears the user has been forced to write a review (i.e. requiring a review to demo a program);
it is written by the author or an employee of the company who submitted the link to websiteingredients.com; or
it appears to be asking for technical support.
It is websiteingredients.com's policy not to intervene in a dispute between a reviewer and a link owner. Once a review is validated, it will not be removed unless it is later determined that it violates any of these Review Guidelines.
As Member Reviews are an important tool for WebsiteIngredients.com visitors and in order to provide Link Owners the ability to respond to criticism, WebsiteIngredients.com allows Link Owners to respond to reviews submitted on their listing. This tool is provided so that WebsiteIngredients.com Link Owners may provide their side to a negative review, without the need for WebsiteIngredients.com to intervene. As with any review, all responses by Link Owners must be validated by WebsiteIngredients.com, and the Review Guidelines apply to these responses as well.
Report Robots.txt parser