Some Security Tips
The "robots.txt" file has its legitimate uses but not everyone who reads it is legitimate. In many cases the file will give nefarious individuals a glimpse into your directory structure and files you didn't intend anyone to look at.
For example, if you list a directory like 'admin' and have not made it unreadable either via a password or by having an 'index.html' file in the directory or use the 'Options -Indexes' server directive, then even if there are no links anywhere to 'admin' the person reading your robots.txt file knows you have that directory and may therefore be able to look at it and all the files therein.
Another thing to keep in mind when writing a web page is that anything you get via the web may be evil. For example, if you have a form with an 'input' box you should make sure that the input data does not contain nefarious markup. You may be asking for someone's name which you intend on displaying on the form action page. What if the person entered the following markup:
If the browser rendering your site has JavaScripts enabled that little snippet would redirect from your site to another site with a big advertisement. The result could be worse depending on where the redirection goes.
This type of devilishness is not restricted to 'input' boxes. Say your site gets the 'HTTP_USER_AGENT' and displays it. Many browsers have html markup in the User Agent Strings. I have seen <script> tags as well as anchor tags (<a).
What to do? Well it is good practice to escape all tag markers, that is less than (<) and greater than (>) symbols, as < and >. Using PHP you could make a function:
function escapeltgt($value) {
$value = preg_replace(
array("/</", "/>/"),
array("<", ">"),
$value);
return $value;
}
Use the function on anything that could be dangerous.
