← Back to team overview

fuel-dev team mailing list archive

Re: memcache issue

 

Vova, bug like this was reproduced with horizon about month ago.
roots of this bug -- we have unsyncronize mamcache on controllers.

in Horizon we fix it by django-specific config options.
Nadya present Nova-specific options -- for Nova.

another way -- make replication between memcached on controllers.


On Tue, Dec 3, 2013 at 7:15 PM, Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>wrote:

> Nadya
>
> Is everything required described here:
> https://bugs.launchpad.net/fuel/+bug/1256893 ? Currently bug is closed as
> incomplete. Could you please provide clear reproducing steps and more
> information in format:
>
> expected result:
>
> a)
> b)
> c)
>
> actual result
>
> d)
> e)
> f)
>
>
> On Thu, Nov 28, 2013 at 8:20 PM, Nadya Privalova <nprivalova@xxxxxxxxxxxx>wrote:
>
>> Hi guys,
>>
>> Today we faced with the issue on the lab created using Fuel.
>>
>> Problem:
>> On Horizon there were a lot of pop-ups with "Unauthorized" error. When we
>> tried to send REST-requests only several first attempts were successful.
>> E.g. http://paste.openstack.org/show/54123/ (successful attempt, the
>> second one was "Unauthorized"). The problem appeared only with nova. No
>> issue with cli.
>>
>> Investigations:
>> From the logs:
>> Nov 13 06:58:49 controller-1461 nova keystoneclient.middleware.auth_token
>> DEBUG Token validation failure.
>> Traceback (most recent call last):
>>   File
>> "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py",
>> line 684, in _validate_u
>> ser_token
>>     cached = self._cache_get(token_id)
>>   File
>> "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py",
>> line 898, in _cache_get
>>     raise InvalidUserToken('Token authorization failed')
>> InvalidUserToken: Token authorization failed
>> Nov 13 06:58:49 controller-1461 nova keystoneclient.middleware.auth_token
>> DEBUG Marking token 211b590c4ba94d62a3981fbf91e934dc as unauthorized in
>> memcache
>>
>> Memcache restarting didn't help. Issues've gone when we turned memcache
>> off.
>>
>> Fix:
>> As it turned out a cluster with no memcache works slow. So the fix is to
>> add additional configs to nova.conf:
>> [keystone_authtoken]
>> token_cache_time=300
>> memcache_security_strategy=ENCRYPT
>> memcache_secret_key=hjhs445
>>
>> All secrets keys are equal in all controllers. More configs you may find
>> here
>> https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L273.
>>
>> I decided to let you know about this issue. Maybe it is needed to add
>> additional configuration to Fuel. If this is our mistake during lab setup -
>> I'm sorry for inconvenience :)
>>
>> Thanks for attention,
>> Nadya
>>
>>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Senior Deployment Engineer,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 45bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin@xxxxxxxxxxxx <nveschikova@xxxxxxxxxxxx>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>

Follow ups

References