widelands-dev team mailing list archive

[kaputtnik <somal@xxxxxxxx>]

kaputtnik has proposed merging lp:~widelands-dev/widelands-website/static_robots_txt into lp:widelands-website.

Requested reviews:
  Widelands Developers (widelands-dev)
Related bugs:
  Bug #336020 in Widelands Website: "robots.txt should exist"
  https://bugs.launchpad.net/widelands-website/+bug/336020

For more details, see:
https://code.launchpad.net/~widelands-dev/widelands-website/static_robots_txt/+merge/313398

Add a robots.txt.

The content is discussed with janus and should be ok.
-- 
Your team Widelands Developers is requested to review the proposed merge of lp:~widelands-dev/widelands-website/static_robots_txt into lp:widelands-website.

=== added file 'templates/robots.txt'
--- templates/robots.txt	1970-01-01 00:00:00 +0000
+++ templates/robots.txt	2016-12-15 21:11:46 +0000
@@ -0,0 +1,23 @@
+# robots.txt for wl.widelands.org
+# Disallowed Webcrawlers
+# User-agent: *
+# Disallow: /
+
+
+# Disallow some things and explicit Allow some things
+User-agent: *
+Disallow: /
+Disallow: /profile
+Disallow: /admin
+Disallow: /accounts
+Allow: /news
+Allow: /wiki
+Allow: /encyclopedia
+Allow: /forum
+Allow: /maps
+Allow: /changelog
+Allow: /docs/wl
+Allow: /developers
+
+# url to sitemap
+Sitemap: https://wl.widelands.org/sitemap.xml/

=== modified file 'urls.py'
--- urls.py	2016-12-13 18:28:51 +0000
+++ urls.py	2016-12-15 21:11:46 +0000
@@ -7,6 +7,7 @@
 from mainpage.views import mainpage
 from news.feeds import NewsPostsFeed
 from django.views.generic.base import RedirectView
+from django.views.generic import TemplateView
 from django.contrib.syndication.views import Feed
 from registration.backends.hmac.views import RegistrationView
 from mainpage.forms import RegistrationWithCaptchaForm
@@ -15,6 +16,8 @@
 urlpatterns = [
     # Creating a sitemap.xml
     url(r'^sitemap\.xml/', include('sitemap_urls')),
+    # Static view of robots.txt
+    url(r'^robots\.txt/',TemplateView.as_view(template_name='robots.txt', content_type="text/plain")),
 
     # Uncomment the next line to enable the admin:
     url(r'^admin/', admin.site.urls),