← Back to team overview

fuel-dev team mailing list archive

Re: memcache issue

 

Sergey, could you please find a bug and mark this bug as its duplicate?


On Tue, Dec 3, 2013 at 8:25 PM, Sergey Vasilenko <svasilenko@xxxxxxxxxxxx>wrote:

> Vova, bug like this was reproduced with horizon about month ago.
> roots of this bug -- we have unsyncronize mamcache on controllers.
>
> in Horizon we fix it by django-specific config options.
> Nadya present Nova-specific options -- for Nova.
>
> another way -- make replication between memcached on controllers.
>
>
> On Tue, Dec 3, 2013 at 7:15 PM, Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>wrote:
>
>> Nadya
>>
>> Is everything required described here:
>> https://bugs.launchpad.net/fuel/+bug/1256893 ? Currently bug is closed
>> as incomplete. Could you please provide clear reproducing steps and more
>> information in format:
>>
>> expected result:
>>
>> a)
>> b)
>> c)
>>
>> actual result
>>
>> d)
>> e)
>> f)
>>
>>
>> On Thu, Nov 28, 2013 at 8:20 PM, Nadya Privalova <nprivalova@xxxxxxxxxxxx
>> > wrote:
>>
>>> Hi guys,
>>>
>>> Today we faced with the issue on the lab created using Fuel.
>>>
>>> Problem:
>>> On Horizon there were a lot of pop-ups with "Unauthorized" error. When
>>> we tried to send REST-requests only several first attempts were successful.
>>> E.g. http://paste.openstack.org/show/54123/ (successful attempt, the
>>> second one was "Unauthorized"). The problem appeared only with nova. No
>>> issue with cli.
>>>
>>> Investigations:
>>> From the logs:
>>> Nov 13 06:58:49 controller-1461 nova
>>> keystoneclient.middleware.auth_token DEBUG Token validation failure.
>>> Traceback (most recent call last):
>>>   File
>>> "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py",
>>> line 684, in _validate_u
>>> ser_token
>>>     cached = self._cache_get(token_id)
>>>   File
>>> "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py",
>>> line 898, in _cache_get
>>>     raise InvalidUserToken('Token authorization failed')
>>> InvalidUserToken: Token authorization failed
>>> Nov 13 06:58:49 controller-1461 nova
>>> keystoneclient.middleware.auth_token DEBUG Marking token
>>> 211b590c4ba94d62a3981fbf91e934dc as unauthorized in memcache
>>>
>>> Memcache restarting didn't help. Issues've gone when we turned memcache
>>> off.
>>>
>>> Fix:
>>> As it turned out a cluster with no memcache works slow. So the fix is to
>>> add additional configs to nova.conf:
>>> [keystone_authtoken]
>>> token_cache_time=300
>>> memcache_security_strategy=ENCRYPT
>>> memcache_secret_key=hjhs445
>>>
>>> All secrets keys are equal in all controllers. More configs you may find
>>> here
>>> https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L273.
>>>
>>> I decided to let you know about this issue. Maybe it is needed to add
>>> additional configuration to Fuel. If this is our mistake during lab setup -
>>> I'm sorry for inconvenience :)
>>>
>>> Thanks for attention,
>>> Nadya
>>>
>>>
>>>
>>> --
>>> Mailing list: https://launchpad.net/~fuel-dev
>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>> Yours Faithfully,
>> Vladimir Kuklin,
>> Senior Deployment Engineer,
>> Mirantis, Inc.
>> +7 (495) 640-49-04
>> +7 (926) 702-39-68
>> Skype kuklinvv
>> 45bk3, Vorontsovskaya Str.
>> Moscow, Russia,
>> www.mirantis.com <http://www.mirantis.ru/>
>> www.mirantis.ru
>> vkuklin@xxxxxxxxxxxx <nveschikova@xxxxxxxxxxxx>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>


-- 
Yours Faithfully,
Vladimir Kuklin,
Senior Deployment Engineer,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
45bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com <http://www.mirantis.ru/>
www.mirantis.ru
vkuklin@xxxxxxxxxxxx <nveschikova@xxxxxxxxxxxx>

References