openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #24215
Re: [Swift] Intermittent error 403 "Access was denied to this resource"
On Tue, Jun 4, 2013 at 2:41 PM, Andrii Loshkovskyi
<loshkovskyi@xxxxxxxxx> wrote:
> Thank you for answer.
>
> Chmouel, do you mean the auth_token on Keystone or swift proxy server?
>
> from /etc/keystone/keystone.conf
>
> [filter:token_auth]
> paste.filter_factory = keystone.middleware:TokenAuthMiddleware.factory
>
> from /etc/swift/proxy-server.conf
>
> [filter:authtoken]
> paste.filter_factory = keystone.middleware.auth_token:filter_factory
> ...
> memcache_servers = 127.0.0.1:11211
in here, remove that line and use cache=swift.cache make sure you
configure the cache middleware properly.
> Further debugging proved that hosts without memcached don't return the error
> 403. I'm still investigating what service can return such error body
> message.
>
well you probably want to have caching...
>
>
> On Tue, Jun 4, 2013 at 12:55 PM, Chmouel Boudjnah <chmouel@xxxxxxxxxxxx>
> wrote:
>>
>> I have seen this when keystone is too busy for validating tokens.
>> getting keystone behind apache or scaling up keystone make things a
>> better (and make sure you are using swift memcache connection in
>> auth_token).
>>
>>
>> Chmouel.
>>
>> On Mon, Jun 3, 2013 at 8:15 PM, Andrii Loshkovskyi
>> <loshkovskyi@xxxxxxxxx> wrote:
>> > Hello,
>> >
>> > I would appreciate if you help me to troubleshoot the following issue:
>> >
>> > I am having error 403 intermittenly when listing containers in swift.
>> > Sometimes the error appears a few times per hour, sometimes once per
>> > day.
>> > Basically, it's possible to reproduce the error with a simple curl
>> > command:
>> >
>> > curl --get -v -H 'X-Auth-Token: ef644...'
>> > http://swift-proxy.example.com:8080/v1/AUTH_323d0...
>> > <body>
>> > <h1>403 Forbidden</h1>
>> > Access was denied to this resource.<br /><br />
>> > </body>
>> >
>> > The token and swift proxy endpoint are all correct as most of the time
>> > the
>> > command works.
>> >
>> > A few words about infrastructure: I use swift 1.7.4 and several swift
>> > proxies. Users are authenticated via Keystone. Tokens are cached with
>> > memcached on swift proxy servers.
>> >
>> > I did a lot of tests to figure out what service generates such error:
>> >
>> > - same issue happens with each swift proxy server, with or without
>> > memcached
>> > enabled
>> > - it happens with different users and in different tenants
>> > - I downloaded sources of swift and Keystone and grepped on that error.
>> > There are some HTTPForbidden values returned in code but no one with the
>> > body 'Access denied to this resource'
>> > - I tried monitoring traffic with tcpdump to catch the error and
>> > understand
>> > who's sending it but with no success yet
>> > - the issue might be related to swift ACL rules but I haven't set any
>> > read/write permissions for containers
>> > - set debug logs for swift proxy but nothing has been found yet
>> >
>> > Please help me to understand how this error is returned. Thank you for
>> > your
>> > time.
>> >
>> >
>> > --
>> > Kind regards,
>> > Andrii Loshkovskyi
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack@xxxxxxxxxxxxxxxxxxx
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>> >
>
>
>
>
> --
> Kind regards,
> Andrii Loshkovskyi
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
Follow ups
References
