yahoo-eng-team team mailing list archive

[melanie witt <1824445@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Saw this issue downstream [1] and found a couple of similar issues
[2][3] where deployments were running the 'nova-manage cellv2
discover_hosts' command in parallel and experiencing tracebacks like
this:

"DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u\"Duplicate
entry 'compute-0.localdomain' for key 'uniq_host_mappings0host'\") [SQL:
u'INSERT INTO host_mappings (created_at, updated_at, cell_id, host)
VALUES (%(created_at)s, %(updated_at)s, %(cell_id)s, %(host)s)']
[parameters: {'host': u'compute-0.localdomain', 'cell_id': 5,
'created_at': datetime.datetime(2019, 4, 10, 15, 20, 50, 527925),
'updated_at': None}] (Background on this error at:
http://sqlalche.me/e/gkpj)",

After some discussion on IRC today [4], we concluded it would be best to
address the situation with improved command help and warnings when
duplicate host mappings are encountered.

While we could try-except to ignore DBDuplicateEntry, this is not a
situation we want to hide from users as it means they are likely
hammering their database with parallel updates that are mostly not going
to succeed. So we should stop and warn instead.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1698630
[2] https://bugs.launchpad.net/openstack-ansible/+bug/1752540
[3] https://github.com/bloomberg/chef-bcpc/issues/1378
[4] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-04-11.log.html#t2019-04-11T23:04:47

** Affects: nova
     Importance: Low
     Assignee: melanie witt (melwitt)
         Status: In Progress


** Tags: cells nova-manage

** Tags added: nova-manage

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1824445

Title:
  nova-manage cellv2 discover_hosts traces when run in parallel

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Saw this issue downstream [1] and found a couple of similar issues
  [2][3] where deployments were running the 'nova-manage cellv2
  discover_hosts' command in parallel and experiencing tracebacks like
  this:

  "DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u\"Duplicate
  entry 'compute-0.localdomain' for key 'uniq_host_mappings0host'\")
  [SQL: u'INSERT INTO host_mappings (created_at, updated_at, cell_id,
  host) VALUES (%(created_at)s, %(updated_at)s, %(cell_id)s, %(host)s)']
  [parameters: {'host': u'compute-0.localdomain', 'cell_id': 5,
  'created_at': datetime.datetime(2019, 4, 10, 15, 20, 50, 527925),
  'updated_at': None}] (Background on this error at:
  http://sqlalche.me/e/gkpj)",

  After some discussion on IRC today [4], we concluded it would be best
  to address the situation with improved command help and warnings when
  duplicate host mappings are encountered.

  While we could try-except to ignore DBDuplicateEntry, this is not a
  situation we want to hide from users as it means they are likely
  hammering their database with parallel updates that are mostly not
  going to succeed. So we should stop and warn instead.

  [1] https://bugzilla.redhat.com/show_bug.cgi?id=1698630
  [2] https://bugs.launchpad.net/openstack-ansible/+bug/1752540
  [3] https://github.com/bloomberg/chef-bcpc/issues/1378
  [4] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-04-11.log.html#t2019-04-11T23:04:47

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1824445/+subscriptions