canonical-ubuntu-qa team mailing list archive

[Tim Andersson <mp+472964@xxxxxxxxxxxxxxxxxx>]

Tim Andersson has proposed merging ~andersson123/autopkgtest-cloud:robots.txt into autopkgtest-cloud:master.

Requested reviews:
  Canonical's Ubuntu QA (canonical-ubuntu-qa)

For more details, see:
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/472964
-- 
Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:robots.txt into autopkgtest-cloud:master.

diff --git a/charms/focal/autopkgtest-web/reactive/autopkgtest_web.py b/charms/focal/autopkgtest-web/reactive/autopkgtest_web.py
index 1e0f84d..fe3cae7 100644
--- a/charms/focal/autopkgtest-web/reactive/autopkgtest_web.py
+++ b/charms/focal/autopkgtest-web/reactive/autopkgtest_web.py
@@ -324,6 +324,9 @@ def set_up_web_config(apache):
                 ScriptAlias /private-results {webcontrol_dir}/private-results.cgi/
                 ScriptAlias / {webcontrol_dir}/browse.cgi/
 
+                # robots.txt for disallowing bots
+                Alias /robots.txt {webcontrol_dir}/static/robots.txt
+
                 {server_name}
         </VirtualHost>
         """.format(
diff --git a/charms/focal/autopkgtest-web/webcontrol/static/robots.txt b/charms/focal/autopkgtest-web/webcontrol/static/robots.txt
new file mode 100644
index 0000000..fd4fb5c
--- /dev/null
+++ b/charms/focal/autopkgtest-web/webcontrol/static/robots.txt
@@ -0,0 +1,119 @@
+User-agent: StractBot
+Disallow: /
+
+User-agent: MJ12bot
+Disallow: /
+
+User-agent: RedekenBot
+Disallow: /
+
+User-agent: SEOkicks
+Disallow: /
+
+User-agent: wpbot
+Disallow: /
+
+User-agent: SurdotlyBot
+Disallow: /
+
+User-agent: SemrushBot
+Disallow: /
+
+User-agent: wpbot
+Disallow: /
+
+User-agent: MojeekBot
+Disallow: /
+
+User-agent: DataForSeoBot
+Disallow: /
+
+User-agent: YandexBot
+Disallow: /
+
+User-agent: VelenPublicWebCrawler
+Disallow: /
+
+User-agent: SeznamBot
+Disallow: /
+
+User-agent: Qwantbot-prod
+Disallow: /
+
+User-agent: YioopBot
+Disallow: /
+
+User-agent: Bravebot
+Disallow: /
+
+User-agent: coccocbot-web
+Disallow: /
+
+User-agent: SeekportBot
+Disallow: /
+
+User-agent: Yeti
+Disallow: /
+
+User-agent: Barkrowler
+Disallow: /
+
+User-agent: GPTBot
+Disallow: /
+
+User-agent: BLEXBot
+Disallow: /
+
+User-agent: ImagesiftBot
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+
+User-agent: OAI-SearchBot
+Disallow: /
+
+User-agent: YaK
+Disallow: /
+
+User-agent: LinerBot
+Disallow: /
+
+User-agent: Qwantify-prod
+Disallow: /
+
+User-agent: IustusBot
+Disallow: /
+
+User-agent: PerplexityBot
+Disallow: /
+
+User-agent: AhrefsBot
+Disallow: /
+
+User-agent: AwarioBot
+Disallow: /
+
+User-agent: ClaudeBot
+Disallow: /
+
+User-agent: ChatGPT-User
+Disallow: /
+
+User-agent: WireReaderBot
+Disallow: /
+
+User-agent: SiteAuditBot
+Disallow: /
+
+User-agent: GPTBot
+Disallow: /
+
+User-agent: Apollo Web Spider
+Disallow: /
+
+User-agent: DotBot
+Disallow: /
+
+User-agent: SemrushBot
+Disallow: /