diff options
| author | Dan Fandrich <danf@mageia.org> | 2024-04-25 09:32:35 -0700 |
|---|---|---|
| committer | Dan Fandrich <danf@mageia.org> | 2024-04-25 09:32:35 -0700 |
| commit | 88e58bb092c7832e34513b266bd8dddbcd522649 (patch) | |
| tree | 127da11e40eee2da9224c81b5b074161d2280418 | |
| parent | d19b592784258522f0693158c949c9c23bdea477 (diff) | |
| download | puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar.gz puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar.bz2 puppet-88e58bb092c7832e34513b266bd8dddbcd522649.tar.xz puppet-88e58bb092c7832e34513b266bd8dddbcd522649.zip | |
Add two more robots to the svnweb robots.txt file
These two are hammering the site right now, causing severe slowdowns.
Restrict the paths they are allowed to access.
| -rw-r--r-- | modules/viewvc/files/robots.txt | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/modules/viewvc/files/robots.txt b/modules/viewvc/files/robots.txt index 1c08ba9e..3e0fc5c5 100644 --- a/modules/viewvc/files/robots.txt +++ b/modules/viewvc/files/robots.txt @@ -4,6 +4,8 @@ User-agent: bingbot User-agent: YandexBot User-agent: Mail.RU_Bot User-agent: MJ12bot +User-agent: ClaudeBot +User-agent: Amazonbot Disallow: /*/tags/ Disallow: *?view=annotate* Disallow: *?annotate=* @@ -12,7 +14,7 @@ Disallow: *?r1=* Disallow: *sortby=* Disallow: *sortdir=* Disallow: *?revision=*&view=markup&* -Disallow: *?pathrev=* +Disallow: *pathrev=* Disallow: *?*&view=log* Disallow: *view=log&* Disallow: *diff_format=* |