diff options
| author | Nicolas Vigier <boklm@mageia.org> | 2013-08-14 19:53:49 +0000 |
|---|---|---|
| committer | Nicolas Vigier <boklm@mageia.org> | 2013-08-14 19:53:49 +0000 |
| commit | fa3aecb7e779de3ee9094d4b8ec8ff0db67bad98 (patch) | |
| tree | 63d661ef16cf854c2faf0ffde9b14abd82867e38 /modules/viewvc | |
| parent | 1c2a91f8c529be51cb2d994c461dd5fc0996dfaf (diff) | |
| download | puppet-fa3aecb7e779de3ee9094d4b8ec8ff0db67bad98.tar puppet-fa3aecb7e779de3ee9094d4b8ec8ff0db67bad98.tar.gz puppet-fa3aecb7e779de3ee9094d4b8ec8ff0db67bad98.tar.bz2 puppet-fa3aecb7e779de3ee9094d4b8ec8ff0db67bad98.tar.xz puppet-fa3aecb7e779de3ee9094d4b8ec8ff0db67bad98.zip | |
viewvc/robots.txt: ban AhrefsBot
It doesn't respect Crawl-delay, or disallowed URLs.
So try to disallow everything for them.
Diffstat (limited to 'modules/viewvc')
| -rw-r--r-- | modules/viewvc/files/robots.txt | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/modules/viewvc/files/robots.txt b/modules/viewvc/files/robots.txt index b377dfc8..1c08ba9e 100644 --- a/modules/viewvc/files/robots.txt +++ b/modules/viewvc/files/robots.txt @@ -2,7 +2,6 @@ User-agent: Googlebot User-agent: Baiduspider User-agent: bingbot User-agent: YandexBot -User-agent: AhrefsBot User-agent: Mail.RU_Bot User-agent: MJ12bot Disallow: /*/tags/ @@ -17,5 +16,7 @@ Disallow: *?pathrev=* Disallow: *?*&view=log* Disallow: *view=log&* Disallow: *diff_format=* +User-agent: AhrefsBot +Disallow: / User-agent: * Crawl-delay: 30 |