yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1694666] [NEW] metadata service PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Eduardo Gonzalez <dabarren@xxxxxxxxx>
Date: Wed, 31 May 2017 10:12:41 -0000
Reply-to: Bug 1694666 <1694666@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

Launching an instance causes a lot of error messages in nova-api logs.
The instance is not able to retrieve metadata.

How to reproduce
================

Deploy nova master branch.
Spawn an instance.
Wait the instance to be active.
In nova-api logs will see error messages.

Expected results
================

Instance retrieve metadata information

Actual results
==============

Instance is not able to retrieve metadata

Environment configuration
=========================

OpenStack deployed with kolla
Only source images from master fail, binary(rdo or ubuntu packages) works for now
Affected CentOS, Ubuntu and OracleLinux distributions
Database and memcached works as expected, other services consuming them are not affected.

Logs
====

All logs can be found at kolla gates:

Nova: http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-
dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/nova/

Neutron: http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-
dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/neutron/

Related errors:

Nova API:

2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler [req-3daa8e91-93e5-4676-b77a-048ad3dd53d2 - - - - -] Failed to get metadata for instance id: 8cbd067f-8cd6-4365-b299-3ffc146d0790
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler Traceback (most recent call last):
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 285, in _get_meta_by_instance_id
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     remote_address)
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 87, in get_metadata_by_instance_id
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     self._cache.set(cache_key, data)
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return self.region.set(key, value)
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/region.py", line 973, in set
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     self.backend.set(key, self._value(value))
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     **self.set_arguments
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return getattr(client, __name)(*args, **kwargs)
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 740, in set
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return self._set("set", key, val, time, min_compress_len, noreply)
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1060, in _set
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return _unsafe_set()
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     store_info = self._val_to_store_info(val, min_compress_len)
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     pickler.dump(val)
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler

Neutron metadata:

2017-05-31 09:09:24.985 23 DEBUG neutron.agent.metadata.agent [-] Request: GET /2009-04-04/meta-data/instance-id HTTP/1.0
Accept: */*
Connection: close
Content-Type: text/plain
Host: 169.254.169.254
User-Agent: curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6
X-Forwarded-For: 10.0.0.5
X-Neutron-Network-Id: 50395d7c-fcc6-4aef-afdc-931c4573b0d1 __call__ /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/metadata/agent.py:86
2017-05-31 09:09:25.269 23 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
2017-05-31 09:09:25.269 23 INFO eventlet.wsgi.server [-] 10.0.0.5,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500  len: 361 time: 0.2839720
2017-05-31 09:09:27.460 23 DEBUG eventlet.wsgi.server [-] (23) accepted '' server /var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py:883

Let me know if need more information.

Regards, Eduardo

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Description
  ===========
  
  Launching an instance causes a lot of error messages in nova-api logs.
  The instance is not able to retrieve metadata.
  
  How to reproduce
  ================
  
  Deploy nova master branch.
  Spawn an instance.
  Wait the instance to be active.
  In nova-api logs will see error messages.
  
  Expected results
  ================
  
  Instance retrieve metadata information
  
  Actual results
  ==============
  
  Instance is not able to retrieve metadata
  
  Environment configuration
  =========================
  
  OpenStack deployed with kolla
  Only source images from master fail, binary(rdo or ubuntu packages) works for now
  Affected CentOS, Ubuntu and OracleLinux distributions
+ Database and memcached works as expected, other services consuming them are not affected.
  
  Logs
  ====
  
  All logs can be found at kolla gates:
  
  Nova: http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-
  dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/nova/
  
  Neutron: http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-
  dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/neutron/
- 
  
  Related errors:
  
  Nova API:
  
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler [req-3daa8e91-93e5-4676-b77a-048ad3dd53d2 - - - - -] Failed to get metadata for instance id: 8cbd067f-8cd6-4365-b299-3ffc146d0790
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler Traceback (most recent call last):
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 285, in _get_meta_by_instance_id
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     remote_address)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 87, in get_metadata_by_instance_id
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     self._cache.set(cache_key, data)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return self.region.set(key, value)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/region.py", line 973, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     self.backend.set(key, self._value(value))
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     **self.set_arguments
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return getattr(client, __name)(*args, **kwargs)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 740, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return self._set("set", key, val, time, min_compress_len, noreply)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1060, in _set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return _unsafe_set()
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     store_info = self._val_to_store_info(val, min_compress_len)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     pickler.dump(val)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
- 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler 
+ 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler
  
  Neutron metadata:
  
  2017-05-31 09:09:24.985 23 DEBUG neutron.agent.metadata.agent [-] Request: GET /2009-04-04/meta-data/instance-id HTTP/1.0
  Accept: */*
  Connection: close
  Content-Type: text/plain
  Host: 169.254.169.254
  User-Agent: curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6
  X-Forwarded-For: 10.0.0.5
  X-Neutron-Network-Id: 50395d7c-fcc6-4aef-afdc-931c4573b0d1 __call__ /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/metadata/agent.py:86
  2017-05-31 09:09:25.269 23 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
  2017-05-31 09:09:25.269 23 INFO eventlet.wsgi.server [-] 10.0.0.5,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500  len: 361 time: 0.2839720
  2017-05-31 09:09:27.460 23 DEBUG eventlet.wsgi.server [-] (23) accepted '' server /var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py:883
  
- 
  Let me know if need more information.
  
  Regards, Eduardo

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1694666

Title:
  metadata service PicklingError: Can't pickle <class
  'sqlalchemy.orm.session.Session'>

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  Launching an instance causes a lot of error messages in nova-api logs.
  The instance is not able to retrieve metadata.

  How to reproduce
  ================

  Deploy nova master branch.
  Spawn an instance.
  Wait the instance to be active.
  In nova-api logs will see error messages.

  Expected results
  ================

  Instance retrieve metadata information

  Actual results
  ==============

  Instance is not able to retrieve metadata

  Environment configuration
  =========================

  OpenStack deployed with kolla
  Only source images from master fail, binary(rdo or ubuntu packages) works for now
  Affected CentOS, Ubuntu and OracleLinux distributions
  Database and memcached works as expected, other services consuming them are not affected.

  Logs
  ====

  All logs can be found at kolla gates:

  Nova: http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-
  dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/nova/

  Neutron: http://logs.openstack.org/73/469373/1/check/gate-kolla-
  ansible-dsvm-deploy-centos-source-
  centos-7-nv/8cecb36/logs/kolla/neutron/

  Related errors:

  Nova API:

  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler [req-3daa8e91-93e5-4676-b77a-048ad3dd53d2 - - - - -] Failed to get metadata for instance id: 8cbd067f-8cd6-4365-b299-3ffc146d0790
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler Traceback (most recent call last):
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 285, in _get_meta_by_instance_id
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     remote_address)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 87, in get_metadata_by_instance_id
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     self._cache.set(cache_key, data)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return self.region.set(key, value)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/region.py", line 973, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     self.backend.set(key, self._value(value))
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     **self.set_arguments
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return getattr(client, __name)(*args, **kwargs)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 740, in set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return self._set("set", key, val, time, min_compress_len, noreply)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1060, in _set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     return _unsafe_set()
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     store_info = self._val_to_store_info(val, min_compress_len)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler     pickler.dump(val)
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
  2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler

  Neutron metadata:

  2017-05-31 09:09:24.985 23 DEBUG neutron.agent.metadata.agent [-] Request: GET /2009-04-04/meta-data/instance-id HTTP/1.0
  Accept: */*
  Connection: close
  Content-Type: text/plain
  Host: 169.254.169.254
  User-Agent: curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6
  X-Forwarded-For: 10.0.0.5
  X-Neutron-Network-Id: 50395d7c-fcc6-4aef-afdc-931c4573b0d1 __call__ /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/metadata/agent.py:86
  2017-05-31 09:09:25.269 23 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error.
  2017-05-31 09:09:25.269 23 INFO eventlet.wsgi.server [-] 10.0.0.5,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500  len: 361 time: 0.2839720
  2017-05-31 09:09:27.460 23 DEBUG eventlet.wsgi.server [-] (23) accepted '' server /var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py:883

  Let me know if need more information.

  Regards, Eduardo

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1694666/+subscriptions
Follow ups

[Bug 1694666] Re: metadata service PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'> when using memcached
From: OpenStack Infra, 2017-06-30