From 88e58bb092c7832e34513b266bd8dddbcd522649 Mon Sep 17 00:00:00 2001 From: Dan Fandrich Date: Thu, 25 Apr 2024 09:32:35 -0700 Subject: Add two more robots to the svnweb robots.txt file These two are hammering the site right now, causing severe slowdowns. Restrict the paths they are allowed to access. --- modules/viewvc/files/robots.txt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/viewvc/files/robots.txt b/modules/viewvc/files/robots.txt index 1c08ba9e..3e0fc5c5 100644 --- a/modules/viewvc/files/robots.txt +++ b/modules/viewvc/files/robots.txt @@ -4,6 +4,8 @@ User-agent: bingbot User-agent: YandexBot User-agent: Mail.RU_Bot User-agent: MJ12bot +User-agent: ClaudeBot +User-agent: Amazonbot Disallow: /*/tags/ Disallow: *?view=annotate* Disallow: *?annotate=* @@ -12,7 +14,7 @@ Disallow: *?r1=* Disallow: *sortby=* Disallow: *sortdir=* Disallow: *?revision=*&view=markup&* -Disallow: *?pathrev=* +Disallow: *pathrev=* Disallow: *?*&view=log* Disallow: *view=log&* Disallow: *diff_format=* -- cgit v1.2.1