diff options
| author | Jani Välimaa <wally@mageia.org> | 2024-06-17 18:53:54 +0300 |
|---|---|---|
| committer | Jani Välimaa <wally@mageia.org> | 2024-06-17 18:53:54 +0300 |
| commit | 222574e4822bdd63488c0acb8e51e379170458e8 (patch) | |
| tree | 7a0b1ba1dfcf339007db3a658ae8b41c87e94a90 /modules | |
| parent | 28ae203a9931ccc11243d8d652efc81ac0e401c9 (diff) | |
| download | puppet-222574e4822bdd63488c0acb8e51e379170458e8.tar puppet-222574e4822bdd63488c0acb8e51e379170458e8.tar.gz puppet-222574e4822bdd63488c0acb8e51e379170458e8.tar.bz2 puppet-222574e4822bdd63488c0acb8e51e379170458e8.tar.xz puppet-222574e4822bdd63488c0acb8e51e379170458e8.zip | |
svnweb: add more bots to robots.txt
Diffstat (limited to 'modules')
| -rw-r--r-- | modules/viewvc/files/robots.txt | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/modules/viewvc/files/robots.txt b/modules/viewvc/files/robots.txt index 3e0fc5c5..416168b6 100644 --- a/modules/viewvc/files/robots.txt +++ b/modules/viewvc/files/robots.txt @@ -6,6 +6,8 @@ User-agent: Mail.RU_Bot User-agent: MJ12bot User-agent: ClaudeBot User-agent: Amazonbot +User-agent: PetalBot +User-agent: Bytespider Disallow: /*/tags/ Disallow: *?view=annotate* Disallow: *?annotate=* @@ -20,5 +22,7 @@ Disallow: *view=log&* Disallow: *diff_format=* User-agent: AhrefsBot Disallow: / +User-agent: Sogou web spider +Disallow: / User-agent: * Crawl-delay: 30 |