widelands-dev team mailing list archive

[SirVer <SirVer@xxxxxx>]

Review: Needs Information

I feel really uncomfortable about the black listing, I think it should be a whitelist.

Otherwise some nits.

Diff comments:

> === added file 'templates/robots.txt'
> --- templates/robots.txt	1970-01-01 00:00:00 +0000
> +++ templates/robots.txt	2016-12-15 21:35:24 +0000
> @@ -0,0 +1,25 @@
> +# robots.txt for wl.widelands.org
> +# Disallowed Webcrawlers

Remove lines 6-9, i.e. the example?

> +# User-agent: *
> +# Disallow: /
> +
> +
> +# Disallow some things and explicit Allow some things
> +User-agent: *
> +Disallow: /

Turn whitelist into blacklist, i.e. remove all 'Allow' and the 'Disallow: /'? As discussed in IRC yesterday I think this is not appropriate for a page with mostly public content. If we add new content we want to have it found by default, not hidden by default.

> +Disallow: /profile
> +Disallow: /admin
> +Disallow: /accounts
> +Allow: /news
> +Allow: /wiki
> +Allow: /encyclopedia
> +Allow: /forum
> +Allow: /maps
> +Allow: /changelog
> +Allow: /docs/wl

why not all of /docs?

> +Allow: /developers
> +Allow: /screenshots
> +Allow: /polls
> +
> +# url to sitemap
> +Sitemap: https://wl.widelands.org/sitemap.xml/
> 
> === modified file 'urls.py'
> --- urls.py	2016-12-13 18:28:51 +0000
> +++ urls.py	2016-12-15 21:35:24 +0000
> @@ -15,6 +16,8 @@
>  urlpatterns = [
>      # Creating a sitemap.xml
>      url(r'^sitemap\.xml/', include('sitemap_urls')),
> +    # Static view of robots.txt
> +    url(r'^robots\.txt/',TemplateView.as_view(template_name='robots.txt', content_type="text/plain")),

space before TemplateView

>  
>      # Uncomment the next line to enable the admin:
>      url(r'^admin/', admin.site.urls),


-- 
https://code.launchpad.net/~widelands-dev/widelands-website/static_robots_txt/+merge/313398
Your team Widelands Developers is subscribed to branch lp:widelands-website.