User-agent: *
Disallow: *.dat

だったものに、
Last-Modified: Thu, 23 Jul 2015 19:17:39 GMT
で

User-agent: Googlebot
User-agent: Twitterbot
User-agent: bingbot
User-agent: "Yahoo! Slurp"
User-agent: YahooCacheSystem
User-agent: "Y!J-ASR/0.1 crawler"
Disallow: bbs.cgi

が付いた