It may be insignificant in the giant scheme of things but the robots.txt file for the White House website has changed from a huge 2,400 line file yesterday to this today.
User-agent: *
Disallow: /includes/
For the less tech-savvy, the robots.txt file instructs search engines on what to include and not include in their indexes. [via]
Recent Comments