# spammer or archiver User-agent: LinkWalker User-Agent: sitecheck.internetseer.com User-agent: ia_archiver Disallow: / # company letting crawl the community: nono User-agent: grub-client Disallow: / # allow google ad crawler User-agent: Mediapartners-Google Disallow: # disallow some stuff User-agent: * Disallow: /oz/ Disallow: /lukas/ Disallow: /roadtrip/ Disallow: /forum/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: */feed/ Disallow: */trackback/ Disallow: */feed/ Disallow: /malediven/ Disallow: /piccoz/ Allow: /feed/