diff options
author | Dan Fandrich <danf@mageia.org> | 2024-04-25 09:32:35 -0700 |
---|---|---|
committer | Dan Fandrich <danf@mageia.org> | 2024-04-25 09:32:35 -0700 |
commit | 88e58bb092c7832e34513b266bd8dddbcd522649 (patch) | |
tree | 127da11e40eee2da9224c81b5b074161d2280418 /modules | |
parent | d19b592784258522f0693158c949c9c23bdea477 (diff) | |
download | puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar.gz puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar.bz2 puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar.xz puppet-88e58bb092c7832e34513b266bd8dddbcd522649.zip |
Add two more robots to the svnweb robots.txt file
These two are hammering the site right now, causing severe slowdowns.
Restrict the paths they are allowed to access.
Diffstat (limited to 'modules')
-rw-r--r-- | modules/viewvc/files/robots.txt | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/modules/viewvc/files/robots.txt b/modules/viewvc/files/robots.txt index 1c08ba9e..3e0fc5c5 100644 --- a/modules/viewvc/files/robots.txt +++ b/modules/viewvc/files/robots.txt @@ -4,6 +4,8 @@ User-agent: bingbot User-agent: YandexBot User-agent: Mail.RU_Bot User-agent: MJ12bot +User-agent: ClaudeBot +User-agent: Amazonbot Disallow: /*/tags/ Disallow: *?view=annotate* Disallow: *?annotate=* @@ -12,7 +14,7 @@ Disallow: *?r1=* Disallow: *sortby=* Disallow: *sortdir=* Disallow: *?revision=*&view=markup&* -Disallow: *?pathrev=* +Disallow: *pathrev=* Disallow: *?*&view=log* Disallow: *view=log&* Disallow: *diff_format=* |