More information is now available within the Webmaster site (Google Code section) for controlling how the Googlebot crawls and indexes a website. The section is broken down into the following categories:
This new section can help give web designers information on how to, for example, prevent a PDF file from being indexed. This new section housed on code.google.com provides web designers and webmasters a comprehensive resource for understanding the robots.txt file usage, X-Robots-Tag HTTP header directives and robots meta tags. From basic definitions of technologies involved to instructions on how to control robots for a website, the new area brings us closer to everyday robots.