# All robots will spider the domain User-agent: * Disallow: # Disallow directorys User-agent: * Disallow: /test/ Disallow: /test2/ Disallow: /backup Disallow: /images/ # Disallow certain files User-agent: * Disallow: add Disallow: area Disallow: clear Disallow: confirm Disallow: connection Disallow: contactusdone.asp Disallow: data Disallow: display Disallow: report Disallow: update