yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1664607] [NEW] test_stamp_pattern failing cells v1 job

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Tue, 14 Feb 2017 15:20:20 -0000
Reply-to: Bug 1664607 <1664607@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

https://review.openstack.org/#/c/431800/ merged yesterday which unskips
test_stamp_pattern which uses cells v1. The test uses security groups
which cells v1 doesn't support, so it breaks the cells v1 job and blocks
nova's gate.

http://logs.openstack.org/57/433257/2/check/gate-tempest-dsvm-cells-
ubuntu-xenial/1732de2/logs/screen-n-cell-
child.txt.gz#_2017-02-13_22_55_18_192

2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler [req-d4aaa51d-7595-4881-9fa4-2e52425fec26 tempest-TestStampPattern-1688631461 tempest-TestStampPattern-1688631461] Couldn't communicate with cell 'child'
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler Traceback (most recent call last):
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 178, in _build_instances
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     image, security_groups, block_device_mapping)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 109, in _create_instances_here
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     num_instances, i)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/compute/api.py", line 1539, in create_db_entry_for_new_instance
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     instance.create()
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return fn(self, *args, **kwargs)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/objects/instance.py", line 550, in create
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     db_inst = db.instance_create(self._context, updates)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/api.py", line 715, in instance_create
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return IMPL.instance_create(context, values)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return f(*args, **kwargs)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     ectxt.value = e.inner_exc
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     self.force_reraise()
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     six.reraise(self.type_, self.value, self.tb)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return f(*args, **kwargs)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return f(context, *args, **kwargs)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1815, in instance_create
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     instance_ref.security_groups = _get_sec_group_models(security_groups)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1810, in _get_sec_group_models
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     context, security_groups))
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4364, in _security_group_get_by_names
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     project_id=context.project_id, security_group_id=group_name)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler SecurityGroupNotFoundForProject: Security group tempest-TestStampPattern-662835790 not found for project eda710d52d7140dc9bc1081995cb1315.
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler 
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler [req-d4aaa51d-7595-4881-9fa4-2e52425fec26 tempest-TestStampPattern-1688631461 tempest-TestStampPattern-1688631461] Couldn't communicate with any cells
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler [req-d4aaa51d-7595-4881-9fa4-2e52425fec26 tempest-TestStampPattern-1688631461 tempest-TestStampPattern-1688631461] Error scheduling instances ['8cd170c4-17bc-48e1-ab38-6195803ea270']
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler Traceback (most recent call last):
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 226, in _schedule_build_to_cells
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     method_kwargs)
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 195, in _build_instances
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     raise exception.NoCellsAvailable()
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler NoCellsAvailable: No cells available matching scheduling criteria.
2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler

** Affects: nova
     Importance: Critical
     Assignee: Matt Riedemann (mriedem)
         Status: In Progress

** Affects: nova/mitaka
     Importance: Undecided
         Status: New

** Affects: nova/newton
     Importance: Undecided
         Status: New

** Affects: nova/ocata
     Importance: Undecided
         Status: New


** Tags: cellsv1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1664607

Title:
  test_stamp_pattern failing cells v1 job

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) mitaka series:
  New
Status in OpenStack Compute (nova) newton series:
  New
Status in OpenStack Compute (nova) ocata series:
  New

Bug description:
  https://review.openstack.org/#/c/431800/ merged yesterday which
  unskips test_stamp_pattern which uses cells v1. The test uses security
  groups which cells v1 doesn't support, so it breaks the cells v1 job
  and blocks nova's gate.

  http://logs.openstack.org/57/433257/2/check/gate-tempest-dsvm-cells-
  ubuntu-xenial/1732de2/logs/screen-n-cell-
  child.txt.gz#_2017-02-13_22_55_18_192

  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler [req-d4aaa51d-7595-4881-9fa4-2e52425fec26 tempest-TestStampPattern-1688631461 tempest-TestStampPattern-1688631461] Couldn't communicate with cell 'child'
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler Traceback (most recent call last):
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 178, in _build_instances
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     image, security_groups, block_device_mapping)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 109, in _create_instances_here
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     num_instances, i)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/compute/api.py", line 1539, in create_db_entry_for_new_instance
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     instance.create()
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return fn(self, *args, **kwargs)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/objects/instance.py", line 550, in create
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     db_inst = db.instance_create(self._context, updates)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/api.py", line 715, in instance_create
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return IMPL.instance_create(context, values)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return f(*args, **kwargs)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     ectxt.value = e.inner_exc
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     self.force_reraise()
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     six.reraise(self.type_, self.value, self.tb)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return f(*args, **kwargs)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     return f(context, *args, **kwargs)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1815, in instance_create
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     instance_ref.security_groups = _get_sec_group_models(security_groups)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1810, in _get_sec_group_models
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     context, security_groups))
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4364, in _security_group_get_by_names
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     project_id=context.project_id, security_group_id=group_name)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler SecurityGroupNotFoundForProject: Security group tempest-TestStampPattern-662835790 not found for project eda710d52d7140dc9bc1081995cb1315.
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler 
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler [req-d4aaa51d-7595-4881-9fa4-2e52425fec26 tempest-TestStampPattern-1688631461 tempest-TestStampPattern-1688631461] Couldn't communicate with any cells
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler [req-d4aaa51d-7595-4881-9fa4-2e52425fec26 tempest-TestStampPattern-1688631461 tempest-TestStampPattern-1688631461] Error scheduling instances ['8cd170c4-17bc-48e1-ab38-6195803ea270']
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler Traceback (most recent call last):
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 226, in _schedule_build_to_cells
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     method_kwargs)
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler   File "/opt/stack/new/nova/nova/cells/scheduler.py", line 195, in _build_instances
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler     raise exception.NoCellsAvailable()
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler NoCellsAvailable: No cells available matching scheduling criteria.
  2017-02-13 22:55:18.192 25510 ERROR nova.cells.scheduler

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1664607/+subscriptions
Follow ups

[Bug 1664607] Re: test_stamp_pattern failing cells v1 job
From: Sean Dague, 2017-06-16
[Bug 1664607] Re: test_stamp_pattern failing cells v1 job
From: OpenStack Infra, 2017-02-15
[Bug 1664607] Re: test_stamp_pattern failing cells v1 job
From: Matt Riedemann, 2017-02-14