# robots.txt is read by spiders/robots to add my site # to a directory, and controls some of their behaviour. User-agent: * Disallow /defaults Disallow /dev Disallow /webalizer