openstack team mailing list archive

["Day, Phil" <philip.day@xxxxxx>]

Hi Folks,

We've been looking into a problem which looks a lot like:

https://bugs.launchpad.net/nova/+bug/855660



2011-10-21 14:13:31,035 ERROR nova.api [5bd52130-d46f-4702-b06b-9ca5045473d7 smokeuser smokeproject] Unexpected error raised: Parent instance <FixedIp at 0x4e74490> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed
(nova.api): TRACE: Traceback (most recent call last):
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/__init__.py", line 363, in __call__
(nova.api): TRACE: result = api_request.invoke(context)
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py", line 90, in invoke
(nova.api): TRACE: result = method(context, **args)
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1195, in describe_instances
(nova.api): TRACE: instance_id=instance_id)
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1204, in _format_describe_instances
(nova.api): TRACE: return {'reservationSet': self._format_instances(context, **kwargs)}
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1309, in _format_instances
(nova.api): TRACE: if fixed['network'] and use_v6:
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/models.py", line 76, in __getitem__
(nova.api): TRACE: return getattr(self, key)
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 163, in __get__
(nova.api): TRACE: instance_dict(instance))
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 383, in get
(nova.api): TRACE: value = callable_(passive=passive)
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 595, in __call__
(nova.api): TRACE: (mapperutil.state_str(state), self.key)
(nova.api): TRACE: DetachedInstanceError: Parent instance <FixedIp at 0x4e74490> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed
(nova.api): TRACE:


As far as we can see the problem seems to be related to some conflict between multiple threads in the same API server instance and lazy loading of some part of the object.

Looking at the sqlalchemy documentation it seems to strongly suggest that when used from multi-threaded WSGI applications that scoped_sessions should be used (I'm not clear on the details but it seems that this effectively makes lazy load operations thread safe).    However whilst this fixes the problem it has a bad effect on the unit tests - in particular it seems to upset all of the DB migration code used in the unit tests.

So does anyone know if there was an explicit decision / reason not to use scoped_sessions in Nova ?

Thanks,
Phil

PS:  The other possible fix we've found is to change sqlalchemy/models.py so that the associations are explicitly set to use eager load - which also seems to fix the problem but feels like a more clumsy way to go about it.   Any thoughts on that would also be appreciated ?