yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1642483] Re: Deploy is failing after deregistering infoblox due to IPAM agent

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: John Belamaric <jbelamaric@xxxxxxxxxxxx>
Date: Mon, 16 Jan 2017 16:51:53 -0000
Reply-to: Bug 1642483 <1642483@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: networking-infoblox
       Status: Incomplete => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1642483

Title:
  Deploy is failing after deregistering infoblox due to IPAM agent

Status in networking-infoblox:
  Won't Fix
Status in neutron:
  Incomplete

Bug description:
  Deploy is failing when neutron network is created after registering
  infoblox and then infoblox is Unregistered before deploy of Virtual
  Machine.

  Error:
  Deploy of virtual machine xyz on host xxx failed with exception: Build of instance 8b9a1d54-9417-454b-bb4f-b94d0d89da5d was re-scheduled: Subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 could not be found. Neutron server returns request_ids: ['req-9bc7b6c9-bc73-4234-9854-f5d091a52616']

  Although

  The subnet is existing:
  [root@abc# neutron subnet-list
  +--------------------------------------+------+----------------+--------------------------------------------------+
  | id | name | cidr | allocation_pools |
  +--------------------------------------+------+----------------+--------------------------------------------------+
  | 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 | | 120.20.10.0/24 | {"start": "120.20.10.2", "end": "120.20.10.254"} |

  Steps:
  1.Registered Infoblox
  2.Created network net_test.

  [root@jupiter-vm951 powervc]# neutron net-show c51c8399-2b9e-4319-ab82-3dbbf46adfa8
  +---------------------------+--------------------------------------+
  | Field | Value |
  +---------------------------+--------------------------------------+
  | admin_state_up | True |
  | availability_zone_hints | |
  | availability_zones | |
  | created_at | 2016-11-11T10:15:59Z |
  | description | |
  | id | c51c8399-2b9e-4319-ab82-3dbbf46adfa8 |
  | ipv4_address_scope | |
  | ipv6_address_scope | |
  | mtu | 1500 |
  | name | net_test |
  | project_id | 38b6750413e742db97ffa854c2752848 |
  | provider:network_type | vlan |
  | provider:physical_network | default |
  | provider:segmentation_id | 1510 |
  | revision_number | 4 |
  | router:external | False |
  | shared | False |
  | status | ACTIVE |
  | subnets | 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 |
  | tags | |
  | tenant_id | 38b6750413e742db97ffa854c2752848 |
  | updated_at | 2016-11-11T10:16:01Z |
  +---------------------------+--------------------------------------+

  3.Unregistered infoblox and waited.
  4.Tried deploy of virtual machine xyz.
  Deploy of virtual machine xyz on host xxx failed with exception: Build of instance 8b9a1d54-9417-454b-bb4f-b94d0d89da5d was re-scheduled: Subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 could not be found. Neutron server returns request_ids: ['req-9bc7b6c9-bc73-4234-9854-f5d091a52616']

  Exception seen:

  2016-11-11 05:43:16.389 55297 WARNING nova.compute.manager [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] Instance failed to spawn
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] o.start_deploy_simple()
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] r = f(*args, **kwds)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] for d in self.network_info:
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] self.wait()
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] self[:] = self._gt.wait()
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] return self._exit_event.wait()
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] current.throw(*self._exc)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] result = function(*args, **kwargs)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] return func(*args, **kwargs)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] return self.post(self.ports_path, body=body)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] ret = obj(*args, **kwargs)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 365, in post
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] headers=headers, params=params)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 98, in wrapper
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] ret = obj(*args, **kwargs)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] self._handle_fault_response(status_code, replybody, resp)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 98, in wrapper
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] ret = obj(*args, **kwargs)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] exception_handler_v20(status_code, error_body)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] request_ids=request_ids)
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] NotFound: Subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 could not be found.
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] Neutron server returns request_ids: ['req-6a95977f-bb0a-4fb6-8fb4-e62ad8e957b8']
  2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d]
  2016-11-11 05:43:16.394 55297 INFO powervc_nova.compute.kvmhmc_manager [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] PC-FD7A7DA Terminating instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d
  2016-11-11 05:43:16.396 55297 DEBUG oslo_messaging._drivers.amqpdriver [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] CAST unique_id: 012da4800d8a4b079b81433e34b5f38a NOTIFY exchange 'nova' topic 'notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:432
  2016-11-11 05:43:16.436 55297 DEBUG powervc_nova.compute.kvmhmc_manager [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] block_device_info: {'swap': None, 'ephemerals': [], 'block_device_mapping': [], 'root_device_name': u'/dev/sda'} _shutdown_instance /usr/lib/python2.7/site-packages/powervc_nova/compute/kvmhmc_manager.py:1516

  In Infoblox ID following is the only log entry:
  2016-11-11 05:16:02.136 55565 INFO networking_infoblox.neutron.common.notification_handler [-] Created subnet: {u'service_types': [], u'description': u'', u'enable_dhcp': False, u'network_id': u'c51c8399-2b9e-4319-ab82-3dbbf46adfa8', u'tenant_id': u'38b6750413e742db97ffa854c2752848', u'created_at': u'2016-11-11T10:16:01Z', u'dns_nameservers': [], u'updated_at': u'2016-11-11T10:16:01Z', u'ipv6_ra_mode': None, u'allocation_pools': [{u'start': u'120.20.10.2', u'end': u'120.20.10.254'}], u'gateway_ip': u'120.20.10.1', u'revision_number': 2, u'ipv6_address_mode': None, u'ip_version': 4, u'host_routes': [], u'cidr': u'120.20.10.0/24', u'project_id': u'38b6750413e742db97ffa854c2752848', u'id': u'0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694', u'subnetpool_id': None, u'name': u''}

  Error seen in neutron log:
  2016-11-14 06:57:49.059 61224 DEBUG neutron.ipam.driver [req-2ab7f283-ebe0-4d12-82bb-af1211675b13 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] Loading ipam driver: internal get_instance /usr/lib/python2.7/site-packages/neutron/ipam/driver.py:51
  2016-11-14 06:57:49.067 61224 ERROR neutron.ipam.drivers.neutrondb_ipam.driver [req-2ab7f283-ebe0-4d12-82bb-af1211675b13 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] IPAM subnet referenced to Neutron subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 does not exist
  2016-11-14 06:57:49.087 61224 INFO neutron.api.v2.resource [req-2ab7f283-ebe0-4d12-82bb-af1211675b13 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] create failed (client error): The resource could not be found.

  According to Infoblox team:

  This won't work unless you migrate from Infoblox back to the built-in
  IPAM. There is no automatic migration, and currently we do not offer a
  script that goes from Infoblox -> Builtin, only the other way.

  It should be feasible to write a script or database migration that
  does what you want. There is enough data stored in the Neutron
  database to populate the default driver tables as needed. This would
  be part of Neutron though not the Infoblox driver, and should be
  entered as an RFE there.

To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-infoblox/+bug/1642483/+subscriptions
References

[Bug 1642483] [NEW] Deploy is failing after deregistering infoblox due to IPAM agent
From: Esha Seth, 2016-11-17